Vision Language Models

Chapter 8. Document AI

Now that we have seen everything from fine-tuning to deployment, let’s see a real-world example: domain vision language models (VLMs) are the best choice. We can put what we’ve learned into practice: documents (like PDFs or scanned pages). Document processing is one of the core tasks for multimodal AI due to the multimodal nature of documents, and processing documents can save a lot of time. Let’s go!

Learning Objective

In this chapter, you will learn how to solve many tasks that involve processing documents, picking the right model and technique for the right document task, and customizing document models for the task at hand.

Introduction to Document AI

Documents are one of the best examples of multimodal data: they have text, images, charts, and structured data (tables). On top of that, document contents are placed with a custom layout on top, making every document unique. Due to this, you need modern solutions to extract information from documents.

You can extract information from documents in various ways:

• Directly asking questions to the model about documents without parsing them

• Parsing documents into markdown format, and then processing with language models (RAG, etc.)

• Doing simpler tasks: document classification, form field extraction, layout analysis

When asking questions about the document, the model can generate responses or can extract the exact answer from it. The difference here is called generative or extractive processing, respectively. ...