Towards high quality multilingual NMT in production

Addressing many language pairs at once makes Multilingual Neural Machine Translation (MNMT) an appealing choice in production mode but the size of these MNMT models makes them expensive to train and slower at inference. We describe how it’s possible to have both high quality and efficient MNMT.

Introduction.

The common structure of a Neural Machine Translation (NMT) model generally follows the patterns whereby each token is represented by a fixed-size vector called an embedding; the sequence of input embeddings is processed by the encoder (composed of multiple layers) and the obtained representations are processed by the decoder through an attention mechanism. These models are generally auto-regressive, meaning that translations are generated sequentially, token by token, until an end-of-sequence (EOS) token is generated. This step-by-step computation cannot be done in parallel (e.g. on a GPU), which makes it very slow, especially with longer sequences. Inference costs also grow as the output vocabulary size grows. These encoder and decoder blocks often use the Transformer architecture (1) which has become a standard in natural language processing. The primary innovative component of this architecture is the self-attention mechanism in both the encoder and decoder layers that computes each token’s representation as a function of all other tokens in the sequence. As each token’s representation depends on all tokens of the sequence, self-attention complexity is quadratic with sequence length.

Such encoder-decoder approaches quickly lead to MNMT approaches whereby, instead of feeding a single parallel corpus (e.g., French as source language and English as target language), we feed multiple parallel corpora and train a single MNMT model for all language pairs. All source and target languages share the same vocabulary of subword units and the same embedding parameters. These models are often trained on English-centric parallel data which is more accessible. MNMT models have shown to have interesting properties such as zero-shot translation where the model can translate i.e. from French into Korean even though it hasn’t observed any parallel French-Korean sentences in training. MNMT is also very appealing in production because only one model has to be deployed to serve many languages. It also benefits from transfer learning whereby a single update of the model improves many language pairs at once and, in particular, those with small amounts of parallel data.

MNMT models do however have some important shortcomings and high resource language pairs may suffer from a drop in performance (compared to a bilingual baseline). This phenomenon is often referred to as the “curse of multilinguality”. Increasing the model size solves this problem but much larger models (with several billions of parameters) are more difficult to train and are slower at inference. The question is therefore how to obtain MNMT that performs well and which is efficient.

As inference speed and memory requirements are important in service environments, we conducted joint research with Papago Translate to develop (a) practical inference acceleration techniques without loss of translation quality and (b) parameter efficiency to adapt to new languages and domains at a lower cost.

Accelerating inference in MNMT models. 

Contrary to encoders which can process their entire input in parallel, decoders are auto-regressive meaning they take up most of the inference time. To reduce this time we began by decreasing the depth of the decoder e.g. from 6 layers to 2 and, to compensate for the loss in model capacity, added more layers in the encoder e.g. 12 layers instead of 6. We then implemented a hybrid architecture with a deep Transformer encoder and a shallow recurrent neural network (RNN) decoder. RNNs are faster at inference than Transformers as complexity is linear with sequence length whereas self-attention is quadratic. One difference between multilingual models and bilingual models is the size of the vocabulary which strongly impacts the time spent in the softmax (the function that normalizes the output of the model into a probability distribution over the full vocabulary) and in the beam search code (to search for high-probability translations). For this reason, our last step was to apply language-specific vocabulary filtering to decrease vocabulary size at inference e.g. 8k tokens per language.

We performed our experiments on the top 20 languages of ParaCrawl and reported results on the FLORES test set. Figure 1 shows the average BLEU scores in the 38 English-centric directions and in the 342 non-English directions. 

On the left of Figure 1 we plot the inference speed of each model, measured in words per second. Our baseline 6-6 Transformer (Big 6-6) performs better than the model reported in the FLORES dataset paper (2) which covers more language pairs. We observe that, in addition to being faster, our Big 12-2 model outperforms the 6-6 baseline in both English-centric and non-English directions. Our vocabulary filtering technique further improves speed without any significant decrease in BLEU. The hybrid model with the RNN decoder (Hybrid 12-2) is by far the fastest but it performs worse than Transformer 12-2. Its performance is still almost as good as the Transformer 6-6 baseline and it’s 3 times faster. We also experiment with a multi-decoder Transformer Big 12-2 (one decoder per language) which performs the best in both directions due to its higher modelling capacity. All of these findings were confirmed by additional analyses such as human evaluation and robustness to noise and to long sequences. The details are provided in our EMNLP21 long paper (3).

Towards continual learning of MNMT models.

As these MNMT models are large, we also want to avoid retraining them to adapt to a new domain or to cover a new language. In the remainder of this post, we briefly present what we’ve been doing in that direction. Key components for this are the adapter modules recently introduced for NMT (4). These lightweight blocks inserted between transformer layers, change the original model representations to specialise to a new language or domain. As the original model stays frozen, different adapter modules are activated depending on the input domain or language. In this WMT21 paper (5), we study how to compose language and domain adapters trained independently for NMT. While a naive stacking of these does not perform well, we study other ways to combine them and maximize cross-lingual transfer. With our best adapter combinations, we obtain improvements of 3-4 BLEU on average for source languages that do not have in-domain data. For target languages without in-domain data, we achieve a similar improvement by combining adapters with back-translation.

In another WMT21 paper (6), we efficiently add a new source or target language to an existing MT model without affecting its initial performance by replacing the source or target vocabulary by a new, smaller language-specific vocabulary and by training the new embeddings along with some adapter modules on the data of the new language. When adding new source languages, some zero-shot translation issues are sometimes observed. These can be solved with tiny amounts of back-translation. We also show that our approach can be used to translate between two new languages by simply combining their language-specific parameters at inference time.

We’ve seen how to add new languages to an existing multilingual NMT model without retraining it but how can we do that without using parallel data in the new language pairs? We define this task as unsupervised multilingual neural machine translation or ‘UMNMT’. In another EMNLP2021 long paper (7), our starting point is a multilingual sequence-to-sequence model (mBART) trained on 50 different languages using monolingual data. Such a model can be adapted to MT with parallel data by full fine-tuning or by updating only cross-attention parameters and adding task adapter parameters. However, if we apply these standard approaches to our UMNMT context, we face the issue illustrated in Figure 2. Even though Dutch (NL) was known by mBART i.e. it was trained on some Dutch (NL) monolingual data, when adapting it to MT without English-Dutch (EN-NL) parallel data, it is unable to translate from EN to NL (blue and green lines). It forgets about the initial mBART languages that do not have parallel data.

To overcome this issue, we introduce denoising adapters. These language-specific adapter modules are trained within mBART on monolingual data thanks to a self-supervised learning denoising objective. In a second step, we freeze them and fine-tune the cross-attention parameters of mBART to make it a proper MNMT model. During this stage, only the parallel data available for a subset of languages is used. At each update, the denoising adapters for the source and target languages of the current batch are activated. At inference time, to translate in a new language pair (e.g. from English to Dutch), we just plug in the English adapters in the encoder and the Dutch adapters in the decoder and translate. Figure 2 shows the translation performance of our denoising adapters in red and we can see that our model is now able to translate into Dutch even though it did not see any parallel data involving Dutch during training. Back-translation can be used on top of this to further improve performance. Our denoising adapters also work for languages that were not initially covered by mBART (see the paper (7) for details where extensive experiments on 17 unsupervised languages are described).

Conclusion and future work.

We propose several ways to improve the inference speed of large MNTM models and how to do fast and parameter-efficient domain and language adaptation with them. We also make some contributions related to the goal of continual learning in MNMT and show that it is possible to add and learn new languages efficiently without having to fully retrain a model and sometimes even without any parallel data! Our future work will pursue this goal of training a single model that could handle a large number of language pairs, topics and styles and that could be continuously adapted to new languages and domains. 

Full list of papers presented by NAVER at EMNLP 2021.

Other contributors to the work described in this blog: Stéphane Clinchant, Matthias Gallé, Kweonwoo Jung, Dain Lee, Asa Cooper Stickland and Ahmet Üstün.

References:

1: Attention is all you need. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser et al. 31st Conference on Neural Information Processing Systems (NIPS), Long Beach, California, USA, 4–9 December 2017.

2: The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, Angela Fan. arXiv 2106.03193, 2021.

3: Efficient Inference for Multilingual Neural Machine Translation. Alexandre Berard, Dain Lee, Stéphane Clinchant, Kweonwoo Jung and Vassilina Nikoulina. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online and Punta Cana, Dominican Republic, 7-11 November 2021. 

4: Simple, Scalable Adaptation for Neural Machine Translation. Ankur Bapna and Orhan Firat. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), Hong Kong, 3-7 November 2019.

5: Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters. Asa Cooper Stickland, Alexandre Bérard and Vassilina Nikoulina. Proceedings of the Sixth Conference on Machine Translation (WMT21), Online and Punta Cana, Dominican Republic, 10-11 November 2021.

6: Continual Learning in Multilingual NMT via Language-Specific Embeddings. Alexandre Berard. Proceedings of the Sixth Conference on Machine Translation (WMT21), Online and Punta Cana, Dominican Republic, 10-11 November 2021.

7: Multilingual Unsupervised Neural Machine Translation with Denoising Adapters. Ahmet Üstün, Alexandre Berard, Laurent Besacier, Matthias Gallé. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online and Punta Cana, Dominican Republic, 7-11 November 2021.