ICADL 2024 · 2024 · Noisy documents

Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documents

This paper asks a very specific question: are some neurons inside large language models unusually sensitive to OCR noise, and can controlling them improve named entity recognition on historical documents? We look inside Llama2 and Mistral, compare how their layers react to clean and OCR-corrupted text, identify OCR-sensitive neurons, and test whether neutralising them makes NER more robust.

Publication link → arXiv → Code →

Research question

OCR noise is usually treated as something to clean before modelling. This paper asks a more internal question: where does OCR damage appear inside the model itself?

We test whether specific layers and neurons in Llama2 and Mistral react strongly to corrupted tokens, and whether reducing the influence of those neurons improves named entity recognition on historical documents.

Main result

OCR noise leaves measurable traces inside the models. Some MLP regions behave differently for clean and noisy inputs, and selected neurons consistently react to OCR-like distortions.

Neutralising OCR-sensitive neurons can improve NER performance. The strongest gains appear in specific middle and late layers, depending on the dataset and model.

Method / experiment

The study creates clean/noisy token pairs from corrected French historical newspaper text, then introduces synthetic OCR-like character errors at low, average, and high noise levels.

Those token pairs are passed through Llama2 and Mistral. We compare layer activations with CKA similarity, identify OCR-sensitive neurons from activation differences, and test neuron neutralisation during NER inference.

Why it matters

This is a way of studying OCR robustness from inside the model, not only from the input side. Instead of only asking whether noise hurts, the paper asks where the damage travels.

For historical NLP, that matters because archives rarely arrive clean. Robust extraction needs models that can handle damaged text without quietly turning small character errors into semantic errors.

Three figures tell the story quickly: where OCR noise appears inside the models, and where neuron neutralisation improves NER.

CKA layer similarity values and number of OCR-sensitive neurons for Llama2 and Mistral. — **Figure 1.** CKA values compare clean tokens with OCR-corrupted versions across model layers. The purple area shows the number of OCR-sensitive neurons. This is the best overview figure: it makes the internal sensitivity visible.

Heatmaps of F1-score improvements by layer and neuron bins on a French historical newspaper benchmark. — **Figure 2.** F1-score improvements on the historical newspaper benchmark. Warmer regions show where neutralising OCR-sensitive neurons helps most.

Heatmaps of F1-score improvements by layer and neuron bins on the AJMC classical commentary benchmark. — **Figure 3.** F1-score improvements on AJMC. Here, improvements concentrate around a smaller set of middle-to-late layers, especially around layers 20-24.

Reuse / links

Useful for work on OCR robustness, neuron analysis, historical NER, model interpretability, and controlled noise experiments.

Code / experiments → arXiv →

One-line takeaway

OCR errors are not just ugly characters at the input: they activate identifiable regions inside the model, and carefully reducing their influence can make historical NER less fragile.