Finetuning olmOCR to be a faithful OCR-Engine

Recently, the Allen Institute for Artificial Intelligence introduced olmOCR, a model that has demonstrated impressive results in converting PDFs into clean, linearized plain text while preserving structured content. However, olmOCR's primary use case is to provide more training data for Large Language Models and therefore ignores extraneous information like headers and footers in documents. While this is beneficial for generating training data, it is insufficient for real business applications where crucial information often resides in the header and footer part of the text (think of invoices, for example). This article describes how we fine-tuned the original olmOCR-7B-0225-preview to be a faithful OCR engine, enabling it to support a broader range of applications.

In this example, crucial information in the header and footer marked as red, is ignored by olmOCR.

Problem of pipeline-based OCR-Engines

Optical Character Recognition has broad applications in various business use cases. For a long time, the predominant paradigm for AI-based OCR engines have been pipeline-based systems. These are comprised of multiple machine-learning components, e.g., section segmentation, table parsing, character recognition, etc., chained together. However, one fundamental flaw of this approach is that the extracted results mostly do not flatten the context in a way that adheres to logical reading order, also known as linearization. This is particularly challenging for layout-rich documents with floating elements like multi-column documents with floating diagrams, headers, footers, etc. With the recent advent of Vision Language Models, a lot of effort has been put into leveraging them as alternative OCR systems to tackle this problem.

Starting Point: olmOCR-7B-0225-preview

For evaluation, we utilized a customized version of the olmOCR-mix-0225 eval-datasets, which include header and footer information, also acquired with Qwen2.5-VL-72B-Instruct.