NAVER LABS Europe @ECIR 2023

NAVER LABS Europe is presenting 6 papers this year at ECIR.  We kick off the week with collaborative work with the Universities of Padova and Sorbonne on query predictors and IR in the User Aspects session. In the afternoon in ‘Systems and Foundations’ we share the results of an experiment where we show it’s better to pre-train from scratch on the retrieval collection first stage rankers! We round up Monday with recent work on our neural IR model SPLADE (check out models,  huggingface and ChatGPT plugins) called CoSPLADE as SPLADE for conversational IR. On Tuesday we show how FGSM has been overlooked for improving the robustness of neural retrieval models and on Wednesday we round up with the IR and Machine Learning morning session with MS-Shift where we propose explicit distribution shifts to better measure zero shot performance and benchmark several models . Our last presentation shows that, although adapters have been used with dense retrievers with mixed results, we made the work with SPLADE! Check out our program and papers below.  If you’re attending ECIR you can reach out to Stéphane Clinchant, Thibault Formal, Carlos Lassance and Simon Lupart.

Monday, April 3^rd

Session: User Aspects 1 (11:00 – 12:30 PM)

• Query performance prediction for Neural IR: are we there yet?
  Guglielmo Faggioli, Thibault Formal, Stefano Marchesin, Stéphane Clinchant, Nicola Ferro, Benjamin Piwowarski
  Abstract: Evaluation in Information Retrieval (IR) relies on post-hoc empirical procedures, which are time-consuming and expensive operations. To alleviate this, Query Performance Prediction (QPP) models have been developed to estimate the performance of a system without the need for human-made relevance judgements. Such models, usually relying on lexical features from queries and corpora, have been applied to traditional sparse IR methods – with various degrees of success. With the advent of neural IR and large Pre-trained Language Models, the retrieval paradigm has significantly shifted towards more semantic signals. In this work, we study and analyze to what extent current QPP models can predict the performance of such systems. Our experiments consider seven traditional bag-of-words and seven BERT-based IR approaches, as well as nineteen state-of-the-art QPPs evaluated on two collections, Deep Learning ’19 and Robust ’04. Our findings show that QPPs perform statistically significantly worse on neural IR systems. In settings where semantic signals are prominent (e.g., passage retrieval), their performance on neural models drops by as much as 10% compared to bag-of-words approaches. On top of that, in lexical-oriented scenarios, QPPs fail to predict performance for neural IR systems on those queries where they differ from traditional approaches the most.
  [arXiv]

Session: Systems and Foundations (1:30 – 3:00 PM)

• An experimental study on pretraining transformers from scratch for IR
  Carlos Lassance, Hervé Dejean and Stéphane Clinchant
  Abstract: Finetuning Pretrained Language Models (PLM) for IR has been de facto the standard practice since their breakthrough effectiveness few years ago. But, is this approach well understood and well founded?
  In this paper, we study the impact of the pretraining collection on the final IR effectiveness. In particular, we challenge the current hypothesis that PLM shall be trained on a large enough generic collection and we show that pretraining from scratch on the collection of interest is surprisingly competitive with the current approach. We benchmark first-stage ranking rankers and cross-encoders for reranking on the task of general passage retrieval on MSMARCO, Mr-Tydi for Arabic, Japanese and Russian, and TripClick for specific domain. Contrary to popular belief, we show that, for finetuning first-stage rankers, models pretrained solely on their collection have equivalent or better effectiveness compared to more general models. However, there is a slight effectiveness drop for rerankers pretrained only on the target collection. Overall, our study sheds a new light on the role of the pretraining collection and should make our community ponder on building specialized models by pretraining from scratch. Last but not least, doing so could enable better control of efficiency, data bias and replicability, which are key research questions for the IR community.
  [arXiv]

• CoSPLADE: contextualizing SPLADE for conversational information retrieval
  Nam Le Hai, Thomas Gerald, Thibault Formal, Jian-Yun Nie, Benjamin Piwowarski, Laure Soulier
  Abstract: Conversational search is a difficult task as it aims at retrieving documents based not only on the current user query but also on the full conversation history.
  Most of the previous methods have focused on a multi-stage ranking approach relying on query reformulation, a critical intermediate step that might lead to a sub-optimal retrieval. Other approaches have tried to use a fully neural IR first-stage, but are either zero-shot or rely on full learning-to-rank based on a dataset with pseudo-labels. In this work, leveraging the CANARD dataset, we propose an innovative lightweight learning technique to train a first-stage ranker based on SPLADE. By relying on SPLADE sparse representations, we show that, when combined with a second-stage ranker based on T5Mono, the results are competitive on the TREC CAsT 2020 and 2021 tracks.
  [arXiv] [Code]

Tuesday, April 4^th

Poster Session 2 (4:30 – 6:00 PM)

• A study on FGSM adversarial training for neural retrieval
  Simon Lupart, Stephane Clinchant
  Abstract: Neural retrieval models have acquired significant effectiveness gains over the last few years compared to term-based methods. Nevertheless, those models may be brittle when faced to typos, distribution shifts or vulnerable to malicious attacks. For instance, several recent papers demonstrated that such variations severely impacted models performances, and then tried to train more resilient models. Usual approaches include synonyms replacements or typos injections — as data-augmentation — and the use of more robust tokenizers (characterBERT, BPE-dropout). To further complement the literature, we investigate in this paper adversarial training as another possible solution to this robustness issue. Our comparison includes the two main families of BERT-based neural retrievers, i.e. dense and sparse, with and without distillation techniques. We then demonstrate that one of the most simple adversarial training techniques — the Fast Gradient Sign Method (FGSM) — can improve first stage rankers robustness and effectiveness. In particular, FGSM increases models performances on both in-domain and out-of-domain distributions, and also on queries with typos, for multiple neural retrievers.
  [arXiv]

Wednesday,  April 5^th 

Session: IR & Machine Learning (9:00 – 10:30 AM)

• MS-Shift: an analysis of MS MARCO distribution shifts on neural retrieval
  Simon Lupart, Stephane Clinchant, Thibault Formal
  Abstract: Pre-trained Language Models have recently emerged in Information Retrieval as providing the backbone of a new generation of neural systems that outperform traditional methods on a variety of tasks.
  However, it is still unclear to what extent such approaches generalize in zero-shot conditions. The recent BEIR benchmark provides partial answers to this question by comparing models on datasets and tasks that differ from the training conditions. We aim to address the same question by comparing models under more explicit distribution shifts. To this end, we build three query-based distribution shifts within MS MARCO (query-semantic, query-intent, query-length) which are used to evaluate the three main families of neural retrievers based on BERT: sparse, dense and late-interaction – as well as a mono BERT reranker. We further analyse the performance drops between the train and test query distributions. In particular, we identify two generalization indicators: the first one based on train/test query vocabulary overlap, the second based on each model retrieval score. Intuitively, those indicators verify that, the further away the test set is from the train one, the worse the drop in performance. We also show that models respond differently to the shifts – dense approaches being the most impacted. Overall, our study demonstrates that it is possible to design more controllable distribution shifts as a tool to better understand generalization of IR models. Finally, we release the MS MARCO query subsets, which provide an additional resource to benchmark zero-shot transfer in Information Retrieval.
  [arXiv]
• Parameter-efficient sparse retrievers and rerankers using adapters
  Vaishali Pal, Carlos Lassance, Hervé Dejean, Stéphane Clinchant
  Abstract: Parameter-Efficient transfer learning with Adapters have been studied in Natural Language Processing (NLP) as an alternative to full fine-tuning. Adapters are memory-efficient and scale well with downstream tasks by training small bottle-neck layers added between transformer layers while keeping the large pretrained language model (PLMs) frozen. In spite of showing promising results in NLP, these methods are under-explored in Information Retrieval. While previous studies have only experimented with dense retriever or in a cross lingual retrieval scenario, in this paper we aim to complete the picture on the use of adapters in IR. First, we study adapters for SPLADE, a sparse retriever, for which adapters not only retain the efficiency and effectiveness otherwise achieved by finetuning, but are memory-efficient and orders of magnitude lighter to train. We observe that Adapters-SPLADE not only optimizes just 2\% of training parameters, but outperforms fully fine-tuned counterpart and existing parameter-efficient dense IR models on IR benchmark datasets. Secondly, we address domain adaptation of neural retrieval thanks to adapters on cross-domain BEIR datasets and TripClick. Finally, we also consider knowledge sharing between rerankers and first stage rankers. Overall, our study complete the examination of adapters for neural IR.
  [arXiv]