Deep Learning in Banking

Chapter 4Text Data and Transformers

Summary

After reading this chapter, and going through the exercises, you will be able to:

• Understand why text is one of the most ubiquitous sources of alternative data in banking.
• Understand the theory behind modern transformers for text processing.
• Learn to fine-tune and adjust pretrained language models and how to consider whether your data is sufficient to train a modern architecture from scratch.

Text is at the center of human interaction. From communicating via email, to text messages, to meeting transcripts and minutes, client communications, client forms, web logs, and numerous other sources, text is a part of most processes that involve any level of customer or coworker contact. This text is rich, vast, and full of context that had been a challenge to properly understand through automated means. This was true until the introduction of the Transformer architecture (section 4.5), which started in the public eye what has been the artificial intelligence (AI) revolution. As we have discussed in previous chapters, deep learning models have been an academic and corporate endeavor with very limited exposure to the general public. Because the transformer, and especially because they have been used as the building blocks of the modern large language models, deep learning models have entered our collective consciousness and lives as an everyday tool.

In banking, text fulfills the same role. It is even more important in this area, as it ...