Transformers: The Definitive Guide

Chapter 6. From Sound to Token and Back: Transformers in the Audio Domain

This chapter takes you one step further along the dimensional axis. You started with the original transformer—that is, language—then moved to time series. After that, you explored vision and video. Now, you’ll dive into the audio domain. And spoiler alert, once again, transformers prove to be surprisingly adaptable. As it turns out, it doesn’t matter whether you’re modeling discrete tokens, image patches, or spectrogram frames. The core logic remains intact. That is the elegance of the transformer. You change the axis but not the basic architecture. I’m sure, by now, you’re seeing it for yourself: transformers are not just models; they’re a framework for abstraction. This is why they aren’t a passing trend but a new way to generalize and unify modeling across domains.

What makes audio particularly interesting is that it sits right at the intersection of temporal structure and spectral representation. At first glance, audio might look like a classic time series, just a waveform over time, but its richness lies in the frequency domain. That’s why so many audio models convert raw waveforms into spectrograms or mel frequency features before tokenizing them.

A spectrogram shows how the energy of different frequency components in a signal changes over time, giving a time-frequency view of the sound. A mel frequency representation maps those frequencies onto a perceptual scale that better reflects how humans hear, ...