Serving Machine Learning Models

Chapter 3. Implementing Model Scoring

As depicted in Figure 1-1, the overall architecture of our implementation is reading two input streams, the models stream and the data stream, and then joining them for producing the results stream.

There are two main options today for implementing such stream-based applications: stream-processing engines or stream-processing libraries:

• Modern stream-processing engines (SPEs) take advantage of cluster architectures. They organize computations into a set of operators, which enables execution parallelism; different operators can run on different threads or different machines. An engine manages operator distribution among the cluster’s machines. Additionally, SPEs typically implement checkpointing, which allows seamless restart execution in case of failures.

• A stream-processing library (SPL), on the other hand, is a library, and often domain-specific language (DSL), of constructs simplifying building streaming applications. Such libraries typically do not support distribution and/or clustering; this is typically left as an exercise for developer.

Although these are very different, because both have the word “stream” in them, they are often used interchangeably. In reality, as outlined in Jay Kreps’s blog, they are two very different approaches to building streaming applications, and choosing one of them is a trade-off between power and simplicity. A side-by-side comparison of Flink and Kafka Streams outlines the major differences between ...