A scalable Transformer architecture for summarizing long documents

Our Global BERT-based Transformer architecture fuses global and local information at every layer, resulting in a reading comprehension model that achieves a deeper understanding for long documents and enables flexibility for downstream tasks.

For computers, reading comprehension is a challenging task. Recent advances in neural network architecture have greatly improved the ability of machines to process and understand passages of written text. However, most models still fall short when it comes to understanding the content of longer documents. Successful approaches could enable the creation of automatic summaries from any input text, such as scientific papers or restaurant reviews. Other contexts in which such models could prove useful include question answering on long documents, fact checking and document retrieval.

These days, machine-learning techniques for natural language processing (NLP) generally rely on pretrained language models, such as BERT (Bidirectional Encoder Representations from Transformers) (1). These models have become essential building blocks in the development of deep-learning models for reading comprehension and have improved on state-of-the-art performance for an extensive collection of NLP tasks, such as question answering and sentiment analysis (2).

Transformer layers and the attention mechanism

Most recent competitive architectures, including BERT (1, 3, 4), are based on stacked Transformer ‘encoder’ and ‘decoder’ layers (5). Transformers are a kind of deep-learning model that uses the mechanism of ‘attention’, where the model considers all words in the input text—paying particular attention to those that it deems important, which are assigned a greater ‘weight’—to create a context by which it can better understand the meaning of each subsequently processed word.

Although the attention mechanism of these Transformer layers is effective in terms of both performance and inference speed (2), it does bring with it some restrictions. First, Transformer layers are computationally expensive. The Transformer self-attention memory (i.e. the capacity required to process a given sentence, or sequence) increases quadratically with the number of words, or input tokens, in a document. This level of complexity makes it technically impossible to compute long documents. A second restriction is related to positional embedding (i.e. the weight that’s given to each word based on its position in the text). By design, the Transformer’s attention is independent of the word’s position in the input sequence, so most recent architectures use a trainable positional encoding layer. This means that, to ensure the positional information is encoded correctly, the maximum length of the input text to be processed must be defined at the pretraining stage, as it’s difficult to extend when the model is being fine-tuned on downstream tasks.

We aim to overcome these issues. Our objectives are to design a Transformer-based model that is capable of processing long documents; to make an architecture than can take full advantage of available pretrained language models; and to produce meaningful representations of tokens, contextualized in a full document, for downstream tasks.

Our Global BERT-based Transformer (GBT) architecture: fusing global and local knowledge

To achieve our aims, we propose a new hierarchical structure that leverages pretrained Transformers to encode long documents, as depicted in Figure 1. The novelty of our approach lies in its interweaving of Transformer layers (that encode relatively short local context, usually around 30 words) with propagation layers (that spread the knowledge throughout the document). Formally, the input document is first divided into a sequence of blocks, which in this case correspond to sentences. The first layer of the Transformer then independently processes each block to build local representations of the tokens. Each block is represented by its first token. Then, the propagation layer—a bidirectional gated recurrent unit (BiGRU) neural network—processes the sequence of sentence representations for the entire document to spread this local knowledge. At this stage, the first token of each block (the classification token, denoted [CLS]) is a local representation of the sentence that is enriched with global information from the entire context (the rest of the document). We repeat this process of local encoding and knowledge propagation for all layers of the pretrained architecture.

Local Transformer functions are initialized from available pretrained language models, such as BERT or RoBERTa (a Robustly Optimized BERT Pretraining Approach) (3). Because we construct and propagate document-level information between the layers, global and local information is fused at every layer of the architecture rather than just at the top layer (6). We use sentences as local blocks because the lengths are acceptable for the pretrained Transformers. The top, output layer computes the final representations, which can be adapted to any downstream task. The output layer, which acts as a sentence classifier that aims to label each sentence as ‘selected’ or ‘unselected’ for the summary, is a feedforward neural network that acts on each sentence representation.

Summarizing scientific papers from arXiv and PubMed

To evaluate the quality of our proposition, we use our model to extract a few sentences that best summarize the content of long documents from two summarization datasets (arXiv and PubMed, see Table 1 for details). This task—known as extractive summarization (7)—tests several skills that we consider necessary for reading comprehension models. We tackle it as a sentence-level classification problem, where each sentence is labelled according to whether it belongs to the summary.

To be effective, the model must be capable of two things. First, it should be able to understand long documents beyond the first few tokens. Second, the model needs to have a local understanding of each sentence, so that it’s capable of considering whether a sentence is meaningful or not, and a global understanding of the document, in order to produce a coherent summary.

As can be seen in Table 2, our model outperforms the other approaches on almost all metrics. This is particularly true for the arXiv dataset, where the documents tend to be longer than for PubMed. Sent-CLF, another hierarchical approach in the list of models tested, achieves the most comparable baseline to ours on the PubMed dataset, but performs comparatively poorly on the arXiv one. Overall, BertSumExt seems to be the approach closest to ours in terms of results and architecture. However, there remains a significant gap between the results from all models and the Oracle scores, which suggests that there is still room for improvement on this task. We provide a deeper analysis of all these results in the full version of our paper (6).

Figures 2 and 3 show two examples of extracted summaries produced by the models developed using our approach. Figure 2 shows an extracted summary for the paper ‘Attention Is All You Need’ (5) achieved with a model trained on arXiv papers. Additionally, we found that a model trained to summarize scientific papers is also able to produce relevant summaries in different contexts. In Figure 3, we present an example of a summary produced by a model trained on arXiv and tested on a collection of hotel reviews. We can see that, from a human perspective, the proposed summaries look coherent and provide meaningful information for the reader.

Summarizing GBT, and future work

Our novel Transformer-based model for long-document summarization is based on coupled Transformer layers and propagation layers that encode and spread information between multiple blocks of a document. This model preserves the architecture of commonly used pretrained language models, thus enabling it to take advantage of associated parameters. An evaluation of our BERT-based model in the context of an extractive summarization task further revealed its effectiveness in dealing with long documents compared to other adaptations of BERT and previously proposed models.

Although we focused our evaluations on extractive summarization, we’re interested in testing our architecture on other tasks. In the future, we plan to adapt our model to more tasks that require comprehension of long documents, such as question answering, document-scale machine translation and information retrieval. We’d also like to further investigate how these models, which are trained to summarize scientific papers, can transfer to different corpora, such as hotel/restaurant reviews or news articles.

This work was done in collaboration with the University Grenoble Alpes and the MIAI Grenoble Alpes Institute