ID: 2104.05064

Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modeling

April 11, 2021

View on ArXiv

Similar papers 2

Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations

February 9, 2022

90% Match
Yu Meng, Yunyi Zhang, Jiaxin Huang, ... , Han Jiawei
Computation and Language
Information Retrieval
Machine Learning

Topic models have been the prominent tools for automatic topic discovery from text corpora. Despite their effectiveness, topic models suffer from several limitations including the inability of modeling word ordering information in documents, the difficulty of incorporating external linguistic knowledge, and the lack of both accurate and efficient inference methods for approximating the intractable posterior. Recently, pretrained language models (PLMs) have brought astonishing...

Find SimilarView on arXiv

Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models

May 12, 2022

90% Match
Kabir Ahuja, Shanu Kumar, ... , Choudhury Monojit
Computation and Language

Massively Multilingual Transformer based Language Models have been observed to be surprisingly effective on zero-shot transfer across languages, though the performance varies from language to language depending on the pivot language(s) used for fine-tuning. In this work, we build upon some of the existing techniques for predicting the zero-shot performance on a task, by modeling it as a multi-task learning problem. We jointly train predictive models for different tasks which ...

Find SimilarView on arXiv

Bayesian multilingual topic model for zero-shot cross-lingual topic identification

July 2, 2020

90% Match
Santosh Kesiraju, Sangeet Sagar, Ondřej Glembek, ... , Gangashetty Suryakanth V
Computation and Language

This paper presents a Bayesian multilingual topic model for learning language-independent document embeddings. Our model learns to represent the documents in the form of Gaussian distributions, thereby encoding the uncertainty in its covariance. We propagate the learned uncertainties through linear classifiers for zero-shot cross-lingual topic identification. Our experiments on 5 language Europarl and Reuters (MLDoc) corpora show that the proposed model outperforms multi-ling...

Find SimilarView on arXiv

Multilingual Few-Shot Learning via Language Model Retrieval

June 19, 2023

90% Match
Genta Indra Winata, Liang-Kang Huang, ... , Chandarana Yash
Computation and Language

Transformer-based language models have achieved remarkable success in few-shot in-context learning and drawn a lot of research interest. However, these models' performance greatly depends on the choice of the example prompts and also has high variability depending on how samples are chosen. In this paper, we conduct a comprehensive study of retrieving semantically similar few-shot samples and using them as the context, as it helps the model decide the correct label without an...

Find SimilarView on arXiv

Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration

October 8, 2023

90% Match
Ercong Nie, Helmut Schmid, Hinrich Schütze
Computation and Language

Pretrained multilingual encoder models can directly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts. This is accomplished by predicting the probabilities of the label words at the masked token position, without requiring any updates to the model parameters. However, the performance of this method is limited by the model's bias toward predicting label words which frequently occurred during the pretraining....

Find SimilarView on arXiv

Robust Document Representations using Latent Topics and Metadata

October 23, 2020

90% Match
Natraj Raman, Armineh Nourbakhsh, ... , Veloso Manuela
Computation and Language
Artificial Intelligence
Machine Learning
Neural and Evolutionary Comp...

Task specific fine-tuning of a pre-trained neural language model using a custom softmax output layer is the de facto approach of late when dealing with document classification problems. This technique is not adequate when labeled examples are not available at training time and when the metadata artifacts in a document must be exploited. We address these challenges by generating document representations that capture both text and metadata artifacts in a task agnostic manner. I...

Find SimilarView on arXiv

MultiFiT: Efficient Multi-lingual Language Model Fine-tuning

September 10, 2019

90% Match
Julian Martin Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kardas, ... , Howard Jeremy
Computation and Language
Machine Learning

Pretrained language models are promising particularly for low-resource languages as they only require unlabelled data. However, training existing models requires huge amounts of compute, while pretrained cross-lingual models often underperform on low-resource languages. We propose Multi-lingual language model Fine-Tuning (MultiFiT) to enable practitioners to train and fine-tune language models efficiently in their own language. In addition, we propose a zero-shot method using...

Find SimilarView on arXiv

Revisiting Topic-Guided Language Models

December 4, 2023

90% Match
Carolina Zheng, Keyon Vafa, David M. Blei
Computation and Language
Machine Learning

A recent line of work in natural language processing has aimed to combine language models and topic models. These topic-guided language models augment neural language models with topic models, unsupervised learning methods that can discover document-level patterns of word use. This paper compares the effectiveness of these methods in a standardized setting. We study four topic-guided language models and two baselines, evaluating the held-out predictive performance of each mod...

Find SimilarView on arXiv

BERTTM: Leveraging Contextualized Word Embeddings from Pre-trained Language Models for Neural Topic Modeling

May 16, 2023

90% Match
Zheng Fang, Yulan He, Rob Procter
Computation and Language
Artificial Intelligence

With the development of neural topic models in recent years, topic modelling is playing an increasingly important role in natural language understanding. However, most existing topic models still rely on bag-of-words (BoW) information, either as training input or training target. This limits their ability to capture word order information in documents and causes them to suffer from the out-of-vocabulary (OOV) issue, i.e. they cannot handle unobserved words in new documents. C...

Find SimilarView on arXiv

Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluation

July 19, 2022

90% Match
Félix Gaschi, François Plesse, ... , Toussaint Yannick
Computation and Language

Some Transformer-based models can perform cross-lingual transfer learning: those models can be trained on a specific task in one language and give relatively good results on the same task in another language, despite having been pre-trained on monolingual tasks only. But, there is no consensus yet on whether those transformer-based models learn universal patterns across languages. We propose a word-level task-agnostic method to evaluate the alignment of contextualized represe...

Find SimilarView on arXiv