CoNLL 2020 · 2020 · Noisy documents

Alleviating Digitization Errors in Named Entity Recognition for Historical Documents

This paper asks whether named entity recognition can be made more robust when historical documents arrive through damaged OCR: misspelled names, broken words, unstable spelling, and text that looks almost readable until a model has to classify it. The proposed answer is architectural: keep the BERT encoder, but stack extra Transformer blocks above it so the model has more room to handle noisy, abnormal, or historically unstable word forms before the CRF prediction layer.

Publication link → PDF →

Research question

Can a Transformer-based NER model better survive digitization errors in historical newspapers without relying on language-specific handcrafted features or a separate OCR correction step?

The paper focuses on the uncomfortable zone where OCR damage and historical language variation meet: entities are still there, but the surface forms are misspelled, fragmented, old-fashioned, or only partially readable.

Main result

Adding one or two Transformer blocks on top of BERT improves NER performance on French and German historical newspaper datasets, with statistically significant gains over a BiLSTM-CNN baseline and improvements over stand-alone BERT.

The extra Transformer layers also avoid damaging performance on a modern benchmark, which is useful: the model becomes more robust to historical noise without becoming weirdly bad on clean contemporary text.

Method / experiment

The model starts with a fine-tuned BERT encoder, adds n extra Transformer blocks on top, and uses a CRF layer for sequence prediction. The paper compares BERT alone, BERT+1×Transformer, and BERT+2×Transformer.

Experiments are run on two French/German historical newspaper benchmarks and on CoNLL-2003 as a contemporary control. The analysis then looks at OCR-related misspellings, long entities, partial predictions, and spurious entity predictions.

Figures / tables

Table of NER results comparing BiLSTM-CNN, BERT, and BERT plus stacked Transformer layers on French and German historical newspaper datasets. — **Table 2.** The main result table: BERT plus stacked Transformer layers improves entity-level precision, recall, and F1 on noisy historical newspaper data.

Examples of NER predictions comparing gold standard, BERT, and BERT plus Transformer layers on French and German historical text. — **Figure 2.** Qualitative examples showing how the stacked model recovers entities that stand-alone BERT misses or fragments.

Bar charts of correct predictions of misspelled entities by Levenshtein ratio for French and German historical data. — **Figure 3.** Correctly predicted misspelled entities, grouped by Levenshtein distance. This is the noise-behaviour figure.

Plot comparing spurious entity cases with micro and macro F1 scores. — **Figure 4.** Spurious entity predictions: BERT gets performance partly by overpredicting; the stacked model is more conservative.

Examples of long named entities where the stacked Transformer model predicts fuller entity spans than BERT. — **Table 4.** Long entity examples. The useful part is boring but important: full spans matter, especially when historical text is already damaged.

Why it matters

This was one of the early steps in my recurring obsession with noisy documents: OCR is not only a preprocessing inconvenience; it changes how semantic extraction behaves.

The paper shows that robustness can be pursued inside the NER architecture itself. Instead of only cleaning the input, the model can be given extra representational capacity to absorb spelling variation, OCR damage, and longer historical entity spans.

Reuse / links

Useful for work on historical NER, OCR robustness, entity span quality, long entity recognition, and model behaviour under digitization noise.

Reuse idea: compare a clean modern benchmark with noisy historical collections, then inspect not only F1 but also misspelled entities, long spans, and spurious predictions. The interesting failures usually hide there, as usual.

ACL Anthology → PDF →

One-line takeaway

Noisy historical text does not only need better OCR; it also needs models that do not panic when names arrive slightly broken.