Chapter 5. The DataStream API (v1.7)
This chapter introduces the basics of Flinkâs DataStream API. We show the structure and components of a typical Flink streaming application, discuss Flinkâs type systems and the supported data types, and present data and partitioning transformations. Window operators, time-based transformations, stateful operators, and connectors are discussed in the next chapters. After reading this chapter, you will know how to implement a stream processing application with basic functionality. Our code examples use Scala for conciseness, but the Java API is mostly analogous (exceptions or special cases will be pointed out). We also provide complete example applications implemented in Java and Scala in our GitHub repositories.
Hello, Flink!
Letâs start with a simple example to get a first impression of what it is like to write streaming applications with the DataStream API. We will use this example to showcase the basic structure of a Flink program and introduce some important features of the DataStream API. Our example application ingests a stream of temperature measurements from multiple sensors.
First, letâs have a look at the data type we will be using to represent sensor readings:
case
class
SensorReading
(
Â
id
:
String
,
Â
timestamp
:
Long
,
Â
temperature
:
Double
)
The program in Example 5-1 converts the temperatures from Fahrenheit to Celsius and computes the average temperature every 5 seconds for each sensor.
Example 5-1. Compute the average ...
Get Stream Processing with Apache Flink now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.