11institutetext: NeuralMind, Brazil
UNICAMP, Brazil

# NeuralMind-UNICAMP at 2022 TREC NeuCLIR:
Large Boring Rerankers for Cross-lingual Retrieval

Vitor Jeronymo  Roberto Lotufo  Rodrigo Nogueira

###### Abstract

This paper reports on a study of cross-lingual information retrieval (CLIR)
using the mT5-XXL reranker on the NeuCLIR track of TREC 2022. Perhaps the
biggest contribution of this study is the finding that despite the mT5 model
being fine-tuned only on query-document pairs of the same language it proved
to be viable for CLIR tasks, where query-document pairs are in different
languages, even in the presence of suboptimal first-stage retrieval
performance. The results of the study show outstanding performance across all
tasks and languages, leading to a high number of winning positions. Finally,
this study provides valuable insights into the use of mT5 in CLIR tasks and
highlights its potential as a viable solution. For reproduction refer to
https://github.com/unicamp-dl/NeuCLIR22-mT5

###### Keywords:

CLIR reranker mT5 IR NeuCLIR.

##  1 Introduction

The increasing global interconnectedness of society and the growing need for
access to information in multiple languages have brought the importance of
cross-lingual information retrieval (CLIR). As businesses expand their
operations across borders and individuals become more multilingual, the
requirement for effective retrieval of information in multiple languages has
become increasingly compelling.

The TREC conference has recently launched the NeuCLIR dataset challenge, which
focuses on CLIR. The corpus includes 4.5 million Russian, 3 million Chinese,
and 2 million Persian documents obtained from the Common Crawl service between
August 1, 2016 and July 31, 2021. The dataset also features machine
translations of these documents into English, as well as 114 English queries,
along with their machine and human translations into the three languages.

The dataset consists of three tasks: Ad Hoc CLIR, Reranking CLIR, and
Monolingual Retrieval. Ad Hoc CLIR represents the standard retrieval pipeline
for IR datasets, while Reranking CLIR involves reranking the provided query-
document pairs. Monolingual Retrieval is similar to Ad Hoc, but uses human-
translated topics.

In order to address the challenges posed by the NeuCLIR dataset, we developed
a pipeline that utilizes machine translation to align the language of the
queries with that of the passages. This is followed by a multi-stage retrieval
pipeline that includes BM25 [28] as the first stage retriever and a large
multilingual T5 model as the second stage, proven to be an important component
in top-performing systems in various monolingual and multilingual IR
benchmarks [25, 26, 10, 9, 41, 12, 11, 3]. Our method takes advantage of the
observation that increasing the number of parameters in a model generally
leads to improved retrieval effectiveness [29].

##  2 Related Work

Information Retrieval (IR) has a long and robust history in research, with
numerous publicly accessible datasets developed to support its advancement.
Some of the most well-known and widely-used ones are MS-MARCO [2], TREC [35,
38, 5], Common Crawl [32], and ClueWeb22 [24]. In response to the challenges
of IR, various models and methods have been proposed, utilizing both classic
ones such as Vector Space Model (VSM) [31], Latent Semantic Indexing (LSI) [6]
and BM25, as well as more modern transformer-based models, such as RoBERTa
[19], BERT [7], and T5 [27]. The latter ones have shown effectiveness in
handling multiple tasks, such as text classification [33, 39, 16], named
entity recognition [37, 1, 20], and IR [18].

Regarding the branch of multilingual and CLIR, it is crucial to have access to
appropriate datasets that can be used for both development and evaluation of
models. In recent years, several datasets that support research in this area
have been made publicly available, such as Fire [22, 21], MLQA [15], NTCIR
[30], Mr. Tydi [40], and HC4 [13].

However, traditional lexical algorithms such as BM25 [28] face challenges in
providing efficient multilingual capabilities, as they rely on exact word
matches between the query and the passage for ranking relevance. To overcome
this, one solution is to use automatic translators, such as GoogleTranslate,
BingTranslate, Helsinki [34], or Facebook’s WMT19 [23], to transform either
the query or the passage into a common language.

As seen in monolingual, transformer-based models have also shown to be
effective in handling multilingual and cross-lingual tasks, and there are
various approaches to leverage their potential. One strategy is to fine-tune a
pre-trained model on a specific task and language, using either zero-shot or
few-shot learning techniques. Another approach is multilingual learning, where
a pre-trained model is fine-tuned on a multilingual corpus. A third strategy
is cross-lingual transfer learning, which involves fine-tuning a pre-trained
model on a specific task for multiple languages.

In addition to these fine-tuning approaches, there are also models that are
pre-trained on a massive multilingual corpus, such as mT5 [36], XLM-R [4], and
mBERT [7], which are specifically designed for multilingual scenarios.
Increasing the number of parameters in a model also tends to enhance retrieval
effectiveness. In light of the aforementioned considerations, this study
proposes the implementation of a reranking mechanism for the specified task,
and investigates the effectiveness of evaluating the mT5-XXL model (with 13B
parameters) in a zero-shot manner.

##  3 Methodology

In this section, we outline the models used for the first-stage retrieval and
reranking stages of our CLIR system. We explain how the reranker was trained
for the task, and how we arrived at the optimal configuration for our final
submission.

In order to perform the first stage retrieval, we utilized the runs provided
by the organizers of the competition, as well as the BM25 [28] retrieval
method implemented in the Pyserini framework [17] and the SPLADE [8]. To
accommodate for the cross-lingual nature of the corpus and queries, we
employed various automatic machine translation tools such as Google Translate,
Microsoft Bing Translator, Facebook, Huawei, Caiyun, and Youdao to translate
the English queries into the target languages of Persian, Chinese, and
Russian. These runs, which utilized automatic machine translation, are
classified as Ad Hoc Cross-Language Information Retrieval.

We utilized the multilingual variant of the T5 model, mT5-XXL to rerank the
runs provided by the first-stage retrievers. We fine-tuned it in the same
manner as Bonifacio et al. [3], on the mMARCO dataset utilizing a batch size
of 128 and a maximum sequence length of 512 tokens. The model was fine-tuned
for 100,000 iterations with a constant learning rate of 0.001, and all layers
incorporated a dropout of 0.1, taking around 114 hours on a TPU v3-8. It is
noteworthy that the standard training procedure in mMarco is multilingual in
nature, with query and document samples being in different languages, yet each
sample pair is in the same language.

The HC4 [14] dataset was employed as a validation set for the selection of the
most optimal translators and first-stage retrievers due to the shared language
coverage between it and NeuCLIR, as well as the existence of overlapping
annotated query-document pairs between the two datasets. The RRF and SPLADE
first-stage runs were provided by the NLE and h2loo teams, however, at the
time of submission, the NLE team did not have a SPLADE model available for
Chinese.

The selection of the optimal query translator for each language was performed
using the nDCG@20 and R@1000 metrics. The results presented in Table 1
indicate that the Bing translator was the most effective for Persian and
Russian, while the Youdao translator was deemed the best option for Chinese.

|  |  | NDCG@20 | MAP | RBP | R@100 | R@1000
---|---|---|---|---|---|---|---
fa | Bing | desc & title | 0.398 | 0.287 | 0.281 | 0.646 | 0.828
title | 0.345 | 0.261 | 0.255 | 0.577 | 0.772
Facebook | desc & title | 0.381 | 0.274 | 0.279 | 0.622 | 0.840
title | 0.297 | 0.224 | 0.232 | 0.541 | 0.739
Huawei | desc & title | 0.336 | 0.243 | 0.242 | 0.562 | 0.767
title | 0.212 | 0.155 | 0.154 | 0.399 | 0.621
Human translation | desc & title | 0.449 | 0.329 | 0.316 | 0.689 | 0.865
title | 0.385 | 0.288 | 0.274 | 0.636 | 0.822
ru | Bing | desc & title | 0.308 | 0.232 | 0.253 | 0.536 | 0.766
title | 0.315 | 0.231 | 0.266 | 0.492 | 0.712
Huawei | desc & title | 0.317 | 0.240 | 0.268 | 0.526 | 0.768
title | 0.290 | 0.204 | 0.249 | 0.475 | 0.728
Human translation | desc & title | 0.298 | 0.227 | 0.251 | 0.529 | 0.772
title | 0.295 | 0.219 | 0.248 | 0.487 | 0.718
zh | Bing | desc & title | 0.274 | 0.192 | 0.205 | 0.462 | 0.674
title | 0.241 | 0.179 | 0.186 | 0.466 | 0.659
Caiyun | desc & title | 0.276 | 0.195 | 0.198 | 0.500 | 0.687
title | 0.263 | 0.187 | 0.190 | 0.442 | 0.635
Huawei | desc & title | 0.273 | 0.185 | 0.198 | 0.468 | 0.663
title | 0.237 | 0.163 | 0.173 | 0.455 | 0.640
Human translation | desc & title | 0.260 | 0.191 | 0.187 | 0.484 | 0.710
title | 0.253 | 0.180 | 0.178 | 0.490 | 0.696
Youdao | desc & title | 0.268 | 0.197 | 0.197 | 0.473 | 0.673
title | 0.235 | 0.171 | 0.172 | 0.463 | 0.645
Table 1: Comparison of different machine translators on the test set of the
HC4 dataset.

After conducting multiple experiments, as illustrated in Table 2, we sought to
determine the optimal configuration for the reranker with regards to the
first-stage retriever, query translator, and type of query (title,
description, or both) to be input into the model. The most effective
combination was found to be:

  * •

Persian and Russian: Bing as query translator; SPLADE as a first-stage
retriever, followed by an mT5 reranker that uses the “description” field of
queries;

  * •

Chinese: Youdao as query translator; BM25 RRF as a first-stage retriever; mT5
as a reranker that uses the concatenation of fields “title” and “description”
as queries. For the reranker, we use the original query in English.

All first-stage retrievers use the concatenation of fields “title” and
“description” as queries and the corpora were in their respective original
languages.

Language | Run | NDCG@20 | MAP | RBP | R@100 | R@1000
---|---|---|---|---|---|---
fa | BM25 (Bing translated) | 0.3984 | 0.2873 | 0.2812 | 0.6460 | 0.8275
+description+title-mT5-en | 0.5558 | 0.4584 | 0.4143 | 0.7934 | 0.8275
+description+title-mT5-bing | 0.5639 | 0.4807 | 0.4332 | 0.7900 | 0.8275
+description-mT5-bing | 0.5813 | 0.4843 | 0.4353 | 0.7859 | 0.8275
RRF HT | 0.4104 | 0.3172 | 0.3066 | 0.6878 | 0.9044
+description+title-mT5-HT | 0.6000 | 0.5085 | 0.4469 | 0.8531 | 0.9044
ru | BM25 (Bing translated) | 0.3078 | 0.2317 | 0.2527 | 0.5357 | 0.7664
+description+title-mT5-en | 0.4369 | 0.3611 | 0.3670 | 0.6746 | 0.7664
+description-mT5-bing | 0.4513 | 0.3820 | 0.3829 | 0.7015 | 0.7664
RRF HT | 0.3262 | 0.2624 | 0.2842 | 0.5873 | 0.8206
+description+title-mT5-HT | 0.4535 | 0.3931 | 0.3783 | 0.7005 | 0.8206
zh | BM25 (Bing translated) | 0.2744 | 0.1916 | 0.2047 | 0.4623 | 0.6737
+description+title-mT5-en | 0.5539 | 0.4352 | 0.4154 | 0.6589 | 0.6737
BM25 (Youdao translated) | 0.2679 | 0.1972 | 0.1973 | 0.4734 | 0.6732
+description+title-mT5-youdao | 0.5516 | 0.4405 | 0.4188 | 0.6649 | 0.6732
+description+title-mT5-en | 0.5533 | 0.4382 | 0.4144 | 0.6610 | 0.6732
+description+title-mT5-mt | 0.4667 | 0.3689 | 0.3548 | 0.6330 | 0.6732
RRF HT | 0.2792 | 0.2127 | 0.2127 | 0.4940 | 0.7492
+description+title-mT5-HT | 0.5975 | 0.4784 | 0.4410 | 0.7356 | 0.7492
Table 2: HC4 results for first-stage retrieval (BM25, SPLADE or RFF) followed
by an mt5-XXL reranker. “HT” stands for human-translated queries.

##  4 Results

The NeuCLIR challenge submissions were constructed using SPLADE and BM25 RRF
runs provided by the NLE and h2loo teams, similar to the previous HC4
experiments. The best configuration found in the HC4 experiments was applied
to the NeuCLIR submissions, as illustrated in Table 3. However, contrary to
the expectations, the results showed that using SPLADE as the first-stage
retriever resulted in the worst performance. The best run for Persian was
achieved by reranking the organizer’s run, for Russian, the best run utilized
BM25 RRF, and for Chinese, the best nDCG@20 was achieved with RRF and human-
translated queries.

It is important to note that subsequent to conducting our experiments, we
discovered a bug in some of the BM25 RRF runs. This resulted in a significant
decrease in the performance of the Chinese BM25 RRF run, as evident from its
low R@1000 score of 0.4539. Given the impact of this bug, it is likely that
our runs using BM25 RRF as first-stage retrievers would have performed better
if this issue had not arisen. Unfortunately, at the current time, we are
unable to provide updated metrics as we do not have access to the NeuCLIR
annotated query-document relevance data (qrels).

1st-stage | fa | ru | zh
---|---|---|---
nDCG@20 | mAP | R@1000 | nDCG@20 | mAP | R@1000 | nDCG@20 | mAP | R@1000
SPLADE | 0.5356 | 0.3971 | 0.8069 | 0.5524 | 0.4167 | 0.7642 |  |  |
Organizer’s run | 0.5881 | 0.4350 | 0.8292 | 0.5483 | 0.4216 | 0.7744 | 0.5165 | 0.4039 | 0.7814
HT RRF | 0.5619 | 0.4215 | 0.8329 | 0.5673 | 0.4387 | 0.7612 | 0.4999 | 0.3837 | 0.7022
RRF | 0.5447 | 0.4042 | 0.7820 | 0.5630 | 0.4340 | 0.8036 | 0.3689 | 0.2177 | 0.4539
2nd best | 0.545 | 0.404 | 0.782 | 0.565 | 0.473 | 0.898 | 0.484 | 0.360 | 0.750
TREC Median | 0.3200 | 0.1983 | 0.8195 | 0.3725 | 0.2578 | 0.7590 | 0.2811 | 0.1854 | 0.7269
TREC Max | 0.7340 | 0.5963 | 0.9556 | 0.7053 | 0.5694 | 0.9445 | 0.6847 | 0.5600 | 0.9136
Table 3: Main results on NeuCLIR. Figure 1: Persian Leaderboard. The arrows
illustrated in black denote our submissions. Figure 2: Russian Leaderboard.
The arrows illustrated in black denote our submissions. Figure 3: Chinese
Leaderboard. The arrows illustrated in black denote our submissions.

##  5 Conclusion

Our evaluation results showed that most of our runs performed similarly
regardless of the first-stage retriever used. However, SPLADE performed
poorly, with a lower recall compared to even the faulty BM25 RRF runs.
Additionally, our mT5 reranker demonstrated its robustness even with the
faulty RRF runs as first-stage retrievers. The reranker was able to surface
relevant documents to the top of the list, making our submissions still
competitive in the competition, with some even surpassing the results of the
organizers’ runs. This highlights the robustness of the reranker, even in the
presence of suboptimal first-stage retrieval performance.

The evaluation of the model on a CLIR task marked a noteworthy achievement in
our research. Despite being fine-tuned on the mMarco dataset in a multilingual
context, rather than a cross-lingual one, mT5 demonstrated its viability as a
reranker for CLIR scenarios. This finding highlights the versatility of mT5 in
adapting to different tasks and languages.

Overall, our submissions exhibited exceptional performance across all
languages and achieved top ranking positions in the majority of tasks, as
demonstrated by a substantial margin of superiority when compared to other
teams’ submissions. This is illustrated by the black arrows in Figures 1, 2,
and 3, which pertain to the Persian, Russian, and Chinese languages,
respectively.

## References

  * [1] Arkhipov, M., Trofimova, M., Kuratov, Y., Sorokin, A.: Tuning multilingual transformers for language-specific named entity recognition. In: Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing. pp. 89–93 (2019)
  * [2] Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., Wang, T.: MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268v3 (2018)
  * [3] Bonifacio, L., Jeronymo, V., Abonizio, H.Q., Campiotti, I., Fadaee, M., Lotufo, R., Nogueira, R.: mmarco: A multilingual version of the ms marco passage ranking dataset. arXiv preprint arXiv:2108.13897 (2021)
  * [4] Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116 (2019)
  * [5] Craswell, N., Mitra, B., Yilmaz, E., Campos, D., Voorhees, E.M.: Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820 (2020)
  * [6] Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. Journal of the American society for information science 41(6), 391–407 (1990)
  * [7] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [8] Formal, T., Lassance, C., Piwowarski, B., Clinchant, S.: Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086 (2021)
  * [9] Gao, L., Callan, J.: Unsupervised corpus aware language model pre-training for dense passage retrieval. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2843–2853 (2022)
  * [10] Huang, Y., Huang, J.: York university at trec 2021: Deep learning track (2022)
  * [11] Jeronymo, V., Nascimento, M., Lotufo, R., Nogueira, R.: mrobust04: A multilingual version of the trec robust 2004 benchmark. arXiv preprint arXiv:2209.13738 (2022)
  * [12] Jeronymo, V., Rosa, G., Kallumadi, S., Lotufo, R., Nogueira, R.: A boring-yet-effective approach for the product ranking task of the amazon kdd cup 2022. arXiv preprint arXiv:2208.06264 (2022)
  * [13] Lawrie, D., Mayfield, J., Oard, D.W., Yang, E.: Hc4: A new suite of test collections for ad hoc clir. In: Hagen, M., Verberne, S., Macdonald, C., Seifert, C., Balog, K., Nørvåg, K., Setty, V. (eds.) Advances in Information Retrieval. pp. 351–366. Springer International Publishing, Cham (2022)
  * [14] Lawrie, D., Mayfield, J., Oard, D.W., Yang, E.: Hc4: a new suite of test collections for ad hoc clir. In: European Conference on Information Retrieval. pp. 351–366. Springer (2022)
  * [15] Lewis, P., Oğuz, B., Rinott, R., Riedel, S., Schwenk, H.: Mlqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475 (2019)
  * [16] Li, P., Zhong, P., Mao, K., Wang, D., Yang, X., Liu, Y., Yin, J., See, S.: Act: an attentive convolutional transformer for efficient text classification. In: Proceedings of the AAAI conference on artificial intelligence. vol. 35, pp. 13261–13269 (2021)
  * [17] Lin, J., Ma, X., Lin, S.C., Yang, J.H., Pradeep, R., Nogueira, R.: Pyserini: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations. arXiv preprint arXiv:2102.10073 (2021)
  * [18] Lin, J., Nogueira, R., Yates, A.: Pretrained transformers for text ranking: Bert and beyond. Synthesis Lectures on Human Language Technologies 14(4), 1–325 (2021)
  * [19] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [20] Lothritz, C., Allix, K., Veiber, L., Klein, J., Bissyande, T.F.D.A.: Evaluating pretrained transformer-based models on the task of fine-grained named entity recognition. In: Proceedings of the 28th International Conference on Computational Linguistics. pp. 3750–3760 (2020)
  * [21] Majumder, P., Pal, D., Bandyopadhyay, A., Mitra, M.: Overview of fire 2010. In: Multilingual Information Access in South Asian Languages, pp. 252–257. Springer (2013)
  * [22] Mitra, M.: Overview of fire 2008. In: Working Notes of Forum for Information Retrieval Evaluation (2008)
  * [23] Ng, N., Yee, K., Baevski, A., Ott, M., Auli, M., Edunov, S.: Facebook fair’s wmt19 news translation task submission. arXiv preprint arXiv:1907.06616 (2019)
  * [24] Overwijk, A., Xiong, C., Callan, J.: Clueweb22: 10 billion web documents with rich information. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 3360–3362 (2022)
  * [25] Qiao, Y., Chen, H., Cao, L., Chen, L., Li, P., Wang, J., Gao, P., Ni, Y., Xie, G.: Pash at trec 2020 deep learning track: Dense matching for nested ranking. In: TREC (2020)
  * [26] Qiao, Y., Chen, H., Lai, Y., Wang, J., Liu, T., Ye, X., Fang, R., Gao, P., Xie, W., Xie, G.T.: Pash at trec 2021 deep learning track: Generative enhanced model for multi-stage ranking. ArXiv abs/2205.11245 (2022)
  * [27] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(140), 1–67 (2020)
  * [28] Robertson, S.E., Walker, S., Jones, S., Hancock-Beaulieu, M., Gatford, M.: Okapi at trec-3. In: TREC (1994)
  * [29] Rosa, G.M., Bonifacio, L., Jeronymo, V., Abonizio, H., Fadaee, M., Lotufo, R., Nogueira, R.: No parameter left behind: How distillation and model size affect zero-shot retrieval. arXiv preprint arXiv:2206.02873 (2022)
  * [30] Sakai, T., Oard, D.W., Kando, N.: Evaluating Information Retrieval and Access Tasks: NTCIR’s Legacy of Research Impact. Springer Nature (2021)
  * [31] Salton, G., Wong, A., Yang, C.S.: A vector space model for automatic indexing. Commun. ACM 18(11), 613–620 (nov 1975). https://doi.org/10.1145/361219.361220, https://doi.org/10.1145/361219.361220
  * [32] Snæbjarnarson, V., Símonarson, H.B., Ragnarsson, P.O., Ingólfsdóttir, S.L., Jónsson, H.P., Þorsteinsson, V., Einarsson, H.: A warm start and a clean crawled corpus – A recipe for good language models (2022), https://arxiv.org/abs/2201.05601
  * [33] Tezgider, M., Yildiz, B., Aydin, G.: Text classification using improved bidirectional transformer. Concurrency and Computation: Practice and Experience 34(9), e6486 (2022)
  * [34] Tiedemann, J., Thottingal, S.: Opus-mt–building open translation services for the world. In: Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. European Association for Machine Translation (2020)
  * [35] Voorhees, E.M.: Overview of the TREC 2004 robust track. In: Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004). pp. 52–69. Gaithersburg, Maryland (2004)
  * [36] Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., Raffel, C.: mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934 (2020)
  * [37] Yan, H., Deng, B., Li, X., Qiu, X.: Tener: adapting transformer encoder for named entity recognition. arXiv preprint arXiv:1911.04474 (2019)
  * [38] Zhang, E., Gupta, N., Nogueira, R., Cho, K., Lin, J.: Rapidly deploying a neural search engine for the covid-19 open research dataset. In: Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020 (2020)
  * [39] Zhang, J., Chang, W.C., Yu, H.F., Dhillon, I.: Fast multi-resolution transformer fine-tuning for extreme multi-label text classification. Advances in Neural Information Processing Systems 34, 7267–7280 (2021)
  * [40] Zhang, X., Ma, X., Shi, P., Lin, J.: Mr. tydi: A multi-lingual benchmark for dense retrieval. In: Proceedings of the 1st Workshop on Multilingual Representation Learning. pp. 127–137 (2021)
  * [41] Zhang, Y., Long, D., Xu, G., Xie, P.: Hlatr: Enhance multi-stage text retrieval with hybrid list aware transformer reranking. arXiv preprint arXiv:2205.10569 (2022)