Streaming Change Data Capture

Chapter 6. The Qlik Platform

Qlik Replicate, a modern data integration platform built on change data capture (CDC), is designed for the Systematic (Level 3) and Transformational (Level 4) maturity levels described in Chapter 5. It provides a highly automated and consistent platform for replicating incremental data updates while minimizing production workload impact. Qlik Replicate integrates with all major database, data warehouse, data lake, streaming, cloud, and mainframe end points.

With Qlik Replicate, you can address use cases that include the following:

• Data lake ingestion

• Zero-downtime cloud migrations and real-time cloud-based analytics

• Publication of database transactions to message streams

• Mainframe data offload to data lake and streaming platforms

• Real-time data warehousing

• Enabling SAP data analytics on cloud/streaming/data lake architectures

Qlik Replicate resides on an intermediate server that sits between one or more sources and one or more targets. With the exception of SAP sources, which have special native requirements, no agent software is required on either source or target. Its CDC mechanism captures data and metadata changes through the least-disruptive method possible for each specific source, which in nearly all cases is its log reader. Changes are sent in-memory to the target, with the ability to filter out rows or columns that do not align with the target schema or user-defined parameters. Qlik Replicate also can rename target tables or columns, change ...