Transformers: The Definitive Guide

Chapter 3. Transformers for Vision Tasks

One of the great benefits of transformers in this domain is their few-shot and zero-shot learning capabilities. Labeling images is expensive, and often there are not enough labels available, as is the case with cancer detection. Having an image classification or segmentation model capable of performing well with few samples is a significant leap forward. This makes transformers particularly valuable for tasks where data scarcity is a critical issue, a challenge where traditional convolutional neural network (CNN)-based models often fall short.

Transformers have clearly revolutionized NLP. The next domain that transformers are conquering is vision tasks. The vision transformer (ViT)¹ is a landmark in demonstrating the full potential of transformers in computer vision.

In the sections to come, I’ll show you how you can leverage transformers for vision tasks such as classification, image segmentation, instance segmentation, and panoptic segmentation. I’ll explain the challenges you might encounter, how to mitigate them when working with images, and how to monitor the training of your model.

Additionally, I’ll discuss the impact of different loss functions on the quality of the results. For example, in medical imaging, compound loss functions are often the most robust, as they’re better suited for highly imbalanced segmentation tasks. This careful selection of loss functions can significantly enhance model performance, particularly in critical ...