10
Investigating the Role of Tokenizers in Shaping Transformer Models
When studying transformer models, we tend to focus on their architecture and the datasets provided to train them. This book covers the Original Transformer, BERT, RoBERTa, ChatGPT, GPT4, PaLM, LaMBDA, DALL-E, and more. In addition, the book reviews several benchmark tasks and datasets. We have fine-tuned a BERT-like model and trained a RoBERTa tokenizer, using tokenizers to encode data. In the previous Chapter 9, Shattering the Black Box with Interpretable Tools, we also opened the black box and analyzed the inner workings of a transformer model.
However, we did not explore the critical role tokenizers play and evaluate how they shape the models we build. AI is data-driven. ...
Get Transformers for Natural Language Processing and Computer Vision - Third Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
