Accelerating Document AI

Enterprises are full of documents containing knowledge that isn't accessible by digital workflows. These documents can vary from letters, invoices, forms, reports, to receipts. With the improvements in text, vision, and multimodal AI, it's now possible to unlock that information. This post shows you how your teams can use open-source models to build custom solutions for free!

Document AI includes many data science tasks from image classification, image to text, document question answering, table question answering, and visual question answering. This post starts with a taxonomy of use cases within Document AI and the best open-source models for those use cases. Next, the post focuses on licensing, data preparation, and modeling. Throughout this post, there are links to web demos, documentation, and models.

Use Cases

There are at least six general use cases for building document AI solutions. These use cases differ in the kind of document inputs and outputs. A combination of approaches is often necessary when solving enterprise Document AI problems.

What is Optical Character Recognition (OCR)?

Turning typed, handwritten, or printed text into machine-encoded text is known as Optical Character Recognition (OCR). It's a widely studied problem with many well-established open-source and commercial offerings. The figure shows an example of converting handwriting into text.

What is Document Image Classification?

Classifying documents into the appropriate category, such as forms, invoices, or letters, is known as document image classification. Classification may use either one or both of the document's image and text. The recent addition of multimodal models that use the visual structure and the underlying text has dramatically increased classifier performance.

What is Document layout analysis?

Document layout analysis with DiT.

What is Document parsing?

Donut model consisting of an encoder-decoder Transformer. Taken from the Donut paper.

What is Document question answering (DocVQA)?

What are Licensing Issues in Document AI? What are Data Prep Issues in Document AI? What are Modeling Issues in Document AI? What are Popular Open-Source Models for Document AI? What are Metrics and Datasets for Document AI?