ID: 2106.16171

Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer

June 30, 2021

View on ArXiv

Similar papers 2

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning

October 18, 2022

92% Match
Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, ... , Chen Yiren
Computation and Language

Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries, which are expensive and impractical for low-resource languages. To disengage from these dependencies, researchers have explored training multilingual models on English-only resources and transferring them to low-resource languages. However, its effect is limited by the gap between embedding clusters of different languages. To address this issue, we propose Embedding-Push, Att...

Find SimilarView on arXiv

Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models

October 22, 2022

92% Match
Lifu Tu, Caiming Xiong, Yingbo Zhou
Computation and Language
Artificial Intelligence

Pre-trained multilingual language models show significant performance gains for zero-shot cross-lingual model transfer on a wide range of natural language understanding (NLU) tasks. Previously, for zero-shot cross-lingual evaluation, pre-trained models are only fine-tuned on English data and tested on a variety of target languages. In this paper, we do cross-lingual evaluation on various NLU tasks (sentence classification, sequence labeling, question answering) using prompt-t...

Find SimilarView on arXiv

Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer

December 4, 2022

92% Match
Benjamin Muller, Deepanshu Gupta, Siddharth Patwardhan, Jean-Philippe Fauconnier, ... , Agarwal Sachin
Computation and Language
Artificial Intelligence
Machine Learning

Multi-lingual language models (LM), such as mBERT, XLM-R, mT5, mBART, have been remarkably successful in enabling natural language tasks in low-resource languages through cross-lingual transfer from high-resource ones. In this work, we try to better understand how such models, specifically mT5, transfer *any* linguistic and semantic knowledge across languages, even though no explicit cross-lingual signals are provided during pre-training. Rather, only unannotated texts from e...

Find SimilarView on arXiv

Zero-shot Cross-lingual Transfer without Parallel Corpus

October 7, 2023

92% Match
Yuyang Zhang, Xiaofeng Han, Baojun Wang
Computation and Language

Recently, although pre-trained language models have achieved great success on multilingual NLP (Natural Language Processing) tasks, the lack of training data on many tasks in low-resource languages still limits their performance. One effective way of solving that problem is to transfer knowledge from rich-resource languages to low-resource languages. However, many previous works on cross-lingual transfer rely heavily on the parallel corpus or translation models, which are oft...

Find SimilarView on arXiv

Crosslingual Generalization through Multitask Finetuning

November 3, 2022

92% Match
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, ... , Raffel Colin
Computation and Language
Artificial Intelligence
Machine Learning

Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that ...

Find SimilarView on arXiv

Don't Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddings

April 30, 2020

92% Match
Phillip Keung, Yichao Lu, ... , Bhardwaj Vikas
Computation and Language
Machine Learning

Multilingual contextual embeddings have demonstrated state-of-the-art performance in zero-shot cross-lingual transfer learning, where multilingual BERT is fine-tuned on one source language and evaluated on a different target language. However, published results for mBERT zero-shot accuracy vary as much as 17 points on the MLDoc classification task across four papers. We show that the standard practice of using English dev accuracy for model selection in the zero-shot setting ...

Find SimilarView on arXiv

Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review

May 26, 2023

92% Match
Fred Philippy, Siwen Guo, Shohreh Haddadan
Computation and Language
Artificial Intelligence

In recent years, pre-trained Multilingual Language Models (MLLMs) have shown a strong ability to transfer knowledge across different languages. However, given that the aspiration for such an ability has not been explicitly incorporated in the design of the majority of MLLMs, it is challenging to obtain a unique and straightforward explanation for its emergence. In this review paper, we survey literature that investigates different factors contributing to the capacity of MLLMs...

Find SimilarView on arXiv

From English To Foreign Languages: Transferring Pre-trained Language Models

February 18, 2020

92% Match
Ke Tran
Computation and Language

Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre-trained models enables zero-shot transfer of NLP tasks from high resource languages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required a...

Find SimilarView on arXiv

Can Monolingual Pretrained Models Help Cross-Lingual Classification?

November 10, 2019

92% Match
Zewen Chi, Li Dong, Furu Wei, ... , Huang Heyan
Computation and Language

Multilingual pretrained language models (such as multilingual BERT) have achieved impressive results for cross-lingual transfer. However, due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. In this work, we present two approaches to improve zero-shot cross-lingual classification, by transferring the knowledge from monolingual pretrained models to multilingual ones. Experimental results on two cross-lingual classificat...

Find SimilarView on arXiv

Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learning

December 30, 2019

92% Match
Sean MacAvaney, Luca Soldaini, Nazli Goharian
Information Retrieval
Computation and Language
Machine Learning

While billions of non-English speaking users rely on search engines every day, the problem of ad-hoc information retrieval is rarely studied for non-English languages. This is primarily due to a lack of data set that are suitable to train ranking algorithms. In this paper, we tackle the lack of data by leveraging pre-trained multilingual language models to transfer a retrieval system trained on English collections to non-English queries and documents. Our model is evaluated i...

Find SimilarView on arXiv