Transformers for Natural Language Processing and Computer Vision - Third Edition

Appendix A: Revolutionizing AI: The Power of Optimized Time Complexity in Transformer Models

This appendix is a detailed explanation of the optimized time complexity of transformers introduced in the A brief history of how transformers were born section of Chapter 1, What Are Transformers?.

In this appendix, we will unveil the incredible power of the deceptively simple O(1) time complexity of a single operation in transformer models that changed everything.

To do this, we will look at O-1_and_Accelerators.ipynb, introduced in Chapter 1, in detail to see how transformers hijacked hardware accelerators.

We will then see how the token-to-token approach that transformers use enables them to generate text one token at a time, each influenced by the ...