Scala:Applied Machine Learning

Basic components of a data-driven system

In short, a data-driven architecture contains the following components—at least all the systems I've seen have them—or can be reduced to these components:

• Data ingest: We need to collect the data from systems and devices. Most of the systems have logs, or at least an option to write files into a local filesystem. Some can have capabilities to report information to network-based interfaces such as syslog, but the absence of persistence layer usually means potential data loss, if not absence of audit information.
• Data transformation layer: It was also historically called extract, transform, and load (ETL). Today the data transformation layer can also be used to have real-time processing, where the aggregates ...