ID: 2111.14192

Zero-Shot Cross-Lingual Transfer in Legal Domain Using Transformer Models

November 28, 2021

View on ArXiv
Zein Shaheen, Gerhard Wohlgenannt, Dmitry Mouromtsev
Computer Science
Computation and Language
Artificial Intelligence

Zero-shot cross-lingual transfer is an important feature in modern NLP models and architectures to support low-resource languages. In this work, We study zero-shot cross-lingual transfer from English to French and German under Multi-Label Text Classification, where we train a classifier using English training set, and we test using French and German test sets. We extend EURLEX57K dataset, the English dataset for topic classification of legal documents, with French and German official translation. We investigate the effect of using some training techniques, namely Gradual Unfreezing and Language Model finetuning, on the quality of zero-shot cross-lingual transfer. We find that Language model finetuning of multi-lingual pre-trained model (M-DistilBERT, M-BERT) leads to 32.0-34.94%, 76.15-87.54% relative improvement on French and German test sets correspondingly. Also, Gradual unfreezing of pre-trained model's layers during training results in relative improvement of 38-45% for French and 58-70% for German. Compared to training a model in Joint Training scheme using English, French and German training sets, zero-shot BERT-based classification model reaches 86% of the performance achieved by jointly-trained BERT-based classification model.

Similar papers 1

MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer

September 2, 2021

95% Match
Ilias Chalkidis, Manos Fergadiotis, Ion Androutsopoulos
Computation and Language

We introduce MULTI-EURLEX, a new multilingual dataset for topic classification of legal documents. The dataset comprises 65k European Union (EU) laws, officially translated in 23 languages, annotated with multiple labels from the EUROVOC taxonomy. We highlight the effect of temporal concept drift and the importance of chronological, instead of random splits. We use the dataset as a testbed for zero-shot cross-lingual transfer, where we exploit annotated training documents in ...

Find SimilarView on arXiv

Realistic Zero-Shot Cross-Lingual Transfer in Legal Topic Classification

June 8, 2022

95% Match
Stratos Xenouleas, Alexia Tsoukara, Giannis Panagiotakis, ... , Androutsopoulos Ion
Computation and Language

We consider zero-shot cross-lingual transfer in legal topic classification using the recent MultiEURLEX dataset. Since the original dataset contains parallel documents, which is unrealistic for zero-shot cross-lingual transfer, we develop a new version of the dataset without parallel documents. We use it to show that translation-based methods vastly outperform cross-lingual fine-tuning of multilingually pre-trained models, the best previous zero-shot transfer method for Multi...

Find SimilarView on arXiv

Large Scale Legal Text Classification Using Transformer Models

October 24, 2020

93% Match
Zein Shaheen, Gerhard Wohlgenannt, Erwin Filtz
Computation and Language
Artificial Intelligence

Large multi-label text classification is a challenging Natural Language Processing (NLP) problem that is concerned with text classification for datasets with thousands of labels. We tackle this problem in the legal domain, where datasets, such as JRC-Acquis and EURLEX57K labeled with the EuroVoc vocabulary were created within the legal information systems of the European Union. The EuroVoc taxonomy includes around 7000 concepts. In this work, we study the performance of vario...

Find SimilarView on arXiv

The Right Model for the Job: An Evaluation of Legal Multi-Label Classification Baselines

January 22, 2024

92% Match
Martina Forster, Claudia Schulz, Prudhvi Nokku, Melicaalsadat Mirsafian, ... , Skylaki Stavroula
Computation and Language
Artificial Intelligence

Multi-Label Classification (MLC) is a common task in the legal domain, where more than one label may be assigned to a legal document. A wide range of methods can be applied, ranging from traditional ML approaches to the latest Transformer-based architectures. In this work, we perform an evaluation of different MLC methods using two public legal datasets, POSTURE50K and EURLEX57K. By varying the amount of training data and the number of labels, we explore the comparative advan...

Find SimilarView on arXiv

Large-Scale Multi-Label Text Classification on EU Legislation

June 5, 2019

91% Match
Ilias Chalkidis, Manos Fergadiotis, ... , Androutsopoulos Ion
Computation and Language

We consider Large-Scale Multi-Label Text Classification (LMTC) in the legal domain. We release a new dataset of 57k legislative documents from EURLEX, annotated with ~4.3k EUROVOC labels, which is suitable for LMTC, few- and zero-shot learning. Experimenting with several neural classifiers, we show that BIGRUs with label-wise attention perform better than other current state of the art methods. Domain-specific WORD2VEC and context-sensitive ELMO embeddings further improve per...

Find SimilarView on arXiv

An Empirical Study on Cross-X Transfer for Legal Judgment Prediction

September 25, 2022

91% Match
Joel Niklaus, Matthias Stürmer, Ilias Chalkidis
Computation and Language
Artificial Intelligence
Machine Learning

Cross-lingual transfer learning has proven useful in a variety of Natural Language Processing (NLP) tasks, but it is understudied in the context of legal NLP, and not at all in Legal Judgment Prediction (LJP). We explore transfer learning techniques on LJP using the trilingual Swiss-Judgment-Prediction dataset, including cases written in three languages. We find that cross-lingual transfer improves the overall results across languages, especially when we use adapter-based fin...

Find SimilarView on arXiv

Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer

June 30, 2021

90% Match
Iulia Turc, Kenton Lee, Jacob Eisenstein, ... , Toutanova Kristina
Computation and Language

Despite their success, large pre-trained multilingual models have not completely alleviated the need for labeled data, which is cumbersome to collect for all target languages. Zero-shot cross-lingual transfer is emerging as a practical solution: pre-trained models later fine-tuned on one transfer language exhibit surprising performance when tested on many target languages. English is the dominant source language for transfer, as reinforced by popular zero-shot benchmarks. How...

Find SimilarView on arXiv

Can Monolingual Pretrained Models Help Cross-Lingual Classification?

November 10, 2019

90% Match
Zewen Chi, Li Dong, Furu Wei, ... , Huang Heyan
Computation and Language

Multilingual pretrained language models (such as multilingual BERT) have achieved impressive results for cross-lingual transfer. However, due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. In this work, we present two approaches to improve zero-shot cross-lingual classification, by transferring the knowledge from monolingual pretrained models to multilingual ones. Experimental results on two cross-lingual classificat...

Find SimilarView on arXiv

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning

October 18, 2022

90% Match
Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, ... , Chen Yiren
Computation and Language

Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries, which are expensive and impractical for low-resource languages. To disengage from these dependencies, researchers have explored training multilingual models on English-only resources and transferring them to low-resource languages. However, its effect is limited by the gap between embedding clusters of different languages. To address this issue, we propose Embedding-Push, Att...

Find SimilarView on arXiv

T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification

June 8, 2023

90% Match
Inigo Jauregi Unanue, Gholamreza Haffari, Massimo Piccardi
Computation and Language

Cross-lingual text classification leverages text classifiers trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning (zero/few-shots cross-lingual transfer). Nowadays, cross-lingual text classifiers are typically built on large-scale, multilingual language models (LMs) pretrained on a variety of languages of interest. However, the performance of these models vary significantly across languages and classification tas...

Find SimilarView on arXiv