Data Lake for Enterprises

Kafka message

As said in Apache Kafka documentation, the message contains a fixed header and an array of variable length key and variable length value. The header contains the following fields:

• CRC32 checksum to detect any corruption (4 bytes)
• A so-called magic identifier, having a value of either 0 or 1 (1 byte)
• Attributes identifier (1 byte):
  • Bit 0-2: Compression codec: contains the following values
    • 0: No compression
    • 1: GZIP gives a high compression ratio but low performance on compression and decompression with higher load on the CPU
    • 2: Snappy gives lower compression ratio with high performance in decompression and less burden on the CPU
    • 3: lz4 is a lossless compression algorithm (http://lz4.github.io/lz4/)
  • Bit 3: Timestamp type ...