Please use this identifier to cite or link to this item: http://hdl.handle.net/10995/111587
Title: A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models
Authors: Mokrii, I.
Boytsov, L.
Braslavski, P.
Issue Date: 2021
Publisher: Association for Computing Machinery, Inc
ACM
Citation: Mokrii I. A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models / I. Mokrii, L. Boytsov, P. Braslavski // SIGIR 2021 - Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. — 2021. — Vol. — P. 2081-2085. — 3463093.
Abstract: Due to high annotation costs making the best use of existing human-created training data is an important research direction. We, therefore, carry out a systematic evaluation of transferability of BERT-based neural ranking models across five English datasets. Previous studies focused primarily on zero-shot and few-shot transfer from a large dataset to a dataset with a small number of queries. In contrast, each of our collections has a substantial number of queries, which enables a full-shot evaluation mode and improves reliability of our results. Furthermore, since source datasets licences often prohibit commercial use, we compare transfer learning to training on pseudo-labels generated by a BM25 scorer. We find that training on pseudo-labels - -possibly with subsequent fine-tuning using a modest number of annotated queries - -can produce a competitive or better model compared to transfer learning. Yet, it is necessary to improve the stability and/or effectiveness of the few-shot training, which, sometimes, can degrade performance of a pretrained model. © 2021 ACM.
Keywords: NEURAL INFORMATION RETRIEVAL
PSEUDO-LABELING
TRANSFER LEARNING
INFORMATION RETRIEVAL
LARGE DATASET
LEARNING SYSTEMS
TRANSFER LEARNING
EVALUATION MODES
FINE TUNING
RANKING MODEL
SYSTEMATIC EVALUATION
TRAINING DATA
LEARNING TO RANK
URI: http://hdl.handle.net/10995/111587
Access: info:eu-repo/semantics/openAccess
Conference name: 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2021
Conference date: 11 July 2021 through 15 July 2021
SCOPUS ID: 85111638459
PURE ID: 22990153
ISBN: 9781450380379
metadata.dc.description.sponsorship: Pavel Braslavski thanks the Ministry of Science and Higher Education of the Russian Federation (“Ural Mathematical Center” project).
Appears in Collections:Научные публикации, проиндексированные в SCOPUS и WoS CC

Files in This Item:
File Description SizeFormat 
2-s2.0-85111638459.pdf579,34 kBAdobe PDFView/Open


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.