Vision Language Models

Chapter 9. Document AI

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 9th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In this chapter, you will learn about solving many tasks that involve processing documents, and pick the right model and technique for the right document task.

Introduction to Document AI

Documents are one of the best examples of multimodal data: document has texts, images, charts, and structured data (tables). On top of that, document contents are placed with a custom layout on top, making every document unique. Due to this, you need modern solutions to extract information from documents.

You can extract information from documents in various ways:

• Directly asking questions to the model about documents without parsing them

• Parsing documents into markdown format, and then processing with language models (RAG etc.)

• Simpler tasks: document classification, form field extraction, layout analysis

When asking questions about the document, the model can generate responses, or can extract the exact word to the answer from it. The difference here is called ‘generative’ ...