Chapter 7Multimodal Data and Information Fusion
If you have been doing something for a long time, you probably use some sort of multimodality to make your decisions. To motivate this discussion, let us imagine a financial institution deciding on whether to underwrite a large bond placement on the market. An internal group of analysts will study the financial situation of the company (structured data), read reports regarding the market (text data), maybe listen to the last investor call (audio data), watch the last interview by the CEO on the local business news channel (video data), talk to experts, and review a long list of data sources that will help them decide if underwriting the operation is a good idea. Among these data sources, there will most likely be some scores and ratings that come from models; maybe the analysts will use ChatGPT or other large language model to summarize reports, but the final decision will come from interpreting all of these data sources and joining them in some sort of mental or world model to arrive at a conclusion.
So, complex decision-making is multimodal; ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access