Supercharge your OCR Pipelines with Open Models

We have added Chandra and OlmOCR-2 to this blog, as well as OlmOCR Scores of the models 🫡

TL;DR: The rise of powerful vision-language models has transformed document AI. Each model comes with unique strengths, making it tricky to choose the right one. Open-weight models offer better cost efficiency and privacy. To help you get started with them, we’ve put together this guide.

In this guide, you’ll learn:

• The landscape of current models and their capabilities
• When to fine-tune models vs. use models out-of-the-box
• Key factors to consider when selecting a model for your use case
• How to move beyond OCR with multimodal retrieval and document QA

By the end, you’ll know how to choose the right OCR model, start building with it, and gain deeper insights into document AI. Let’s go!

Table-of-Contents

• Supercharge your OCR Pipelines with Open Models
  • Brief Introduction to Modern OCR
    • Model Capabilities
      • Transcription
      • Handling complex components in documents
      • Output formats
      • Locality Awareness in OCR
      • Model Prompting
  • Cutting-edge Open OCR Models
    • Comparing Latest Models
    • Evaluating Models
      • Benchmarks
      • Cost-efficiency
      • Open OCR Datasets
  • Tools to Run Models
    • Locally
    • Remotely
  • Going Beyond OCR
    • Visual Document Retrievers
    • Using Vision Language Models for Document Question Answering
  • Wrapping up

Brief Introduction to Modern OCR

Optical Character Recognition (OCR) is one of the earliest and longest running challenges in computer vision. Many of AI’s first practical applications focused on turning printed text into digital form.

Model Capabilities

Transcription

Recent models transcribe texts into a machine-readable format.
The input can include:

• Handwritten text
• Various scripts like Latin, Arabic, and Japanese characters
• Mathematical expressions
• Chemical formulas
• Image/Layout/Page number tags

OCR models convert them into machine-readable text that comes in many different formats like HTML, Markdown and more.

Handling complex components in documents

• Images
• Charts
• Tables