Transformers: The Definitive Guide

Chapter 2. Transformers for Time Series

Transformers have gained popularity for time series modeling due to their ability to capture long-sequence interactions more effectively than recurrent neural network (RNN) models. The self-attention mechanism in transformers reduces the maximum length of network signal travel paths to the theoretical minimum of O(1) by allowing each element in the sequence to directly attend to every other element. This eliminates the need for recurrent structures and allows for more efficient information flow. As a result, transformers show great potential for long sequence time-series forecasting (LSTF) and other domains such as anomaly detection and spatiotemporal time series prediction.

Many real-world applications, such as weather forecasting, traffic prediction, industrial process controls, and electricity consumption planning, require the prediction of long sequence time series. LSTF demands a model with high prediction capacity, capable of efficiently capturing precise long-range dependencies between input and output.

In addition, probabilistic time series forecasting is an important task in many practical applications, from finance and weather forecasting to managing computer system performance. Accurate probabilistic forecasts are essential for making informed decisions in these areas. Probabilistic time series forecasting includes a range of probabilities for all possible future outcomes, rather than identifying a single specific outcome as “the ...