Vital records and deep learning are helping us uncover the past from historical handwritten records

A new platform based on deep-learning approaches to handwritten-text recognition and information extraction enables data from century-old documents to be parsed and analysed, making it possible to explore epidemics and the evolution of populations over time.

A great deal of human history is detailed in handwritten documents that have yet to be analysed. Extracting information from such documents has, until recently, represented an extremely time-consuming and labour-intensive task. For this reason, there remains a treasure trove of untapped information that could help provide insight into, for example, the impact of industrialization on populations. Indeed, some of these records—such as those documenting epidemics—may provide useful context for our understanding of modern problems like the COVID-19 pandemic. Is there a way that we can glean and subsequently analyse data from these handwritten records such that we can quickly spot trends and draw conclusions?

Previous approaches to this problem have generally focused on extracting information from modern (printed or digital) documents. Automatic extraction from archival (handwritten) documents presents a much more nuanced challenge and has only been possible for the last 2–3 years (1). In 2019, we co-organized the first competition for table recognition (TR) with both archival and printed documents, with good results (2). Work has been carried out to extract geographical information from archival geometrical surveys (3), and work on machine-learning systems to digitize handwritten material by other groups is ongoing (4). However, as far as we know, we are the first to develop a platform based on this technology for public use.

Our demonstrator, called Vital Records, has been developed to parse and analyse large volumes of handwritten data. The project forms part of the EU Horizon 2020 project READ, which aims to revolutionize access to archival documents. The Vital Records demonstrator, based on a collection of 80,000 pages, brings to life records that were handwritten by more than 700 different priests from 200 parishes in Germany from 1848 to 1878. The demo illustrates how state-of-the-art deep-learning methods—handwritten text recognition (HTR), TR and information extraction—can be used to transform these records into a digital format that can be queried and visualized in different ways to enrich our knowledge from previously unexplored sources of information.

Using Vital Records to trace trends through history

Vital Records allows users to browse and visualize the data set extracted from these German records using spatio-temporal criteria, in addition to the usual search queries. From there on, some notions of German (and even old German) may be useful!

DISCLAIMER: The following examples are used to illustrate what’s possible. They have not been scientifically validated by social historians.

Figure 1: Visualization and data obtained from the Vital Records demonstrator, based on handwritten records from 200 parishes in Germany. This graph shows the evolution of the professions of weaver (blue), shoemaker (orange), and miller (yellow) between 1847 and 1877. Dotted lines represent interpolated values.

Death records provide a range of useful information. As well as the name, age, date and cause of death, the records include the profession of the deceased. With this information in hand, we are able to visualize trends over a given period. For example, we used Vital Records to trace the evolution of professions between 1847 and 1877. The resulting graph, shown in Figure 1, shows the number of deaths recorded for which the field of profession (Stand) contains weaver (Weber), shoemaker (Schumacher) and miller (Müller). The graph may be considered a proxy for estimating the evolution of these professions during this period. It’s possible, too, to spot locally relevant characteristics. For example, data extracted from these records shows a high number of glassmakers (Glasmacher) around Zwiesel, a region that remains well known for its production of glassware to this day.

In addition to the visualization of data using graphs, Vital Records makes possible the development of temporal animations for which the time step can be adjusted by year, month or day. By combining both spatial and temporal information in this way, we are able to visualize the evolution of a given query over space and time.
Our favourite query is the spread of scarlet fever in 1871 as illustrated in the video below:

It is also possible to track specific events, such as the opening of a hospital in a parish or the arrival of a train line. You may even query why, for example, Dr Kufner travelled from Osterhofen to Passau in June 1867; or what happened to railway workers Franzesk, Georg and Luigi in May 1876 in Regen. Answers to questions like these could quickly help solve longstanding puzzles in family trees. (Note: Searching for professions containing the string bahn will provide you with a map of three Bahnstrecke, or train lines: Passau–Regensburg, the Bavarian Forest Railway and München–Simbach.)

Deep-learning and recognition: the technology behind Vital Records

To develop this technology and create the Vital Records demo, NAVER LABS Europe and the Passau Diocesan Archives (ABP) began by transforming handwritten tables—like the one shown in Figure 2—into a digital format. The field of handwritten document processing has improved significantly over the past few years, thanks to the neural network paradigm (i.e. algorithms based on the human brain that are designed to recognize patterns). First, we used the automated recognition and transcription platform Transkribus (part of the READ project with over 30,000 users) to transcribe 1000 pages from the German records. We then used these transcriptions to train an HTR model that could automatically digitize the information from the Death, Birth and Wedding records of the archive. We found the character error rate to be around 10%, meaning that on average one letter out of every 10 is wrongly recognized.

Figure 2: An example from the German records in which deaths are recorded in table format. © ABP (Image ID: S_Uttlau_028_0026)

Next, we focused on converting an image of a table into a spreadsheet. Understanding a page means grasping the layout, the relations between textual elements within the page and so on. Although such ‘table understanding’ remains a challenge, it’s one that can now be addressed with deep neural network (DNN) technology. We developed a DNN that would learn how to organize a set of lines such that the information could be arranged into table rows. To achieve this, we enlisted the help of a graph convolutional network (4,5). Using the ABP collection (1000 pages containing tables that had been annotated to train the models), our system was able to recognize nine out of 10 table rows.

We didn’t know whether the 10% error rate of the HTR and the TR model was sufficient for the task. This uncertainty is what originally seeded the idea that led to the development of Vital Records. Building a user interface that enables experts to search the collection through space and time would provide an answer to the question, ‘Are the results sufficiently good to be useful to users of the archive or social historians?’

Figure 3: The original image and the corresponding extracted table.

Once the information in the table had been extracted (see Figure 3), we used a basic state-of-the-art named-entity recognition tool trained with synthetic data. As the textual data in the records is very regular (e.g. names, dates and family situation), we wrote a text generator that allowed us to generate a large quantity of training data and then train the named-entity recognition tool with it to recognize each word category (first name, last name, death date, etc).

Finally, we focused on spatio-temporal indexing. Associating each record to a geographical location and a temporal point enables navigation of the data through space and time. We achieved spatial indexing simply, through metadata at the parish level (since we know the parish associated with each book). We then refined this geographical data by indexing the location/address of each person. This required disambiguation, however, because some location names—even at the level of the diocese—are ambiguous, as they can refer to various locations/villages (e.g. the location name Oberndorf refers to more than 10 locations in Lower Bavaria and more than 40 in the whole of Bavaria). The second indexing step (i.e. temporal indexing) requires information to be extracted from each record. Even if there is a dedicated column for this information in every type of record, its extraction and normalization requires some processing. This is because the string extracted from the image by the HTR must be normalized and converted to a timestamp that a computer can handle (see Figure 4). Furthermore, although the month and day are usually written in the record itself, the year may be ‘factorized’ at the page level, might only exist in the first record of the page or could even occur a few pages before the one being processed (something a human would not have an issue inferring). Detecting this is still noisy, and we’re currently working on our model to improve its ability to appropriately extract this data.

The first time we showed colleagues the spatio-temporal result for scarlet fever (shown in the video), they said, ‘But this is just random data’! There were two reasons for this response: first, the data hadn’t been presented with a cumulative view; and second, they couldn’t see where the data had originated from. Their response made clear how important it is to provide cumulative information as well as the means to verify it. For this reason, we have implemented an easy way to go back to the original documents from which the information was extracted.

Vital Records, now and in the future

In summary, Vital Records uses information extracted (via HTR and TR machine-learning technology) from archival documents to enable users to visualize trends through history, and even to track specific events. The demonstrator that we have developed, based on data obtained from German parish records between 1848 and 1878, showcases the spatio-temporal capabilities of the demonstrator.

We have almost finished processing the Birth and Wedding records and will be adding them to the demo soon, along with the ability to search across the three types of records. We hope that one outcome of this upgrade will be the automatic generation of family trees. Additionally, a major milestone will be to infer demographic information from these data, such as the population size, and some demographic statistics (birth and death rate, for instance). This can only be computed/estimated when the three types of records are linked, enabling identification of the same person across the three records.

Check it out for yourself

The Vital Records demonstrator is now accessible online. Please try it out and let us know what you think.

If you’d like to explore the data that we used to create Vital Records and are a Transkribus user, just drop me a line and we’ll grant you access to all three collections (Marriages, Births, Deaths) stored in the Transkribus platform.

References

1. Transkribus – A Service Platform for Transcription, Recognition and Retrieval of Historical Documents. Philip Kahle, Sebastian Colutto, Günter Hackl and Günter Mühlberger. International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan, 9–15 November 2017. DOI: 10.1109/ICDAR.2017.307.
2. Competition on Table Detection and Recognition (cTDaR). Liangcai Gao, Yilun Huang, Hervé Déjean, Jean-Luc Meunier, Qinqin Yan, Yu Fang, Florian Kleber and Eva Lang. International Conference on Document Analysis and Recognition (ICDAR), Sydney, Australia, 20–25 September 2019. DOI: 10.1109/ICDAR.2019.00243.
3. I-Balsac: Completing Families with the Help of Automatic Text Recognition. Helene Vezina, Jean-Sebastien Bournival, Christopher Kermorvant and Marie-Laurence Bonhomme. Social History Science Association Annual Meeting, Chicago, IL, 21–13 November 2019.
4. Versatile Layout Understanding via Conjugate Graph. Animesh Prasad, Hervé Déjean and Jean-Luc Meunier, International Conference on Document Analysis and Recognition (ICDAR), Sydney, Australia, 20–25 September 2019.
5. Comparing Machine Learning Approaches for Table Recognition in Historical Register Books. Hervé Déjean, Stéphane Clinchant, Jean-Luc Meunier, Eva Maria Lang and Florian Kleber, 13th IAPR International Workshop on Document Analysis Systems, Vienna, Austria, 24–27 April 2018.

NAVER LABS Europe is a founding member of the European Cooperative Society READ-COOP and continues to work on document processing and information extraction tools. See https://readcoop.eu/ for more information about the READ-COOP.

About the author: Hervé Déjean is a senior research scientist and expert in document recognition and layout. He has created a number of tools that enrich digital documents with semantics and that transform historical handwritten documents into digital ones to help explore our cultural heritage.