Mastering Hadoop 3

Best practices

Apache Kafka has different components such as producer, consumer, and brokers. It is important to follow best practices at each component level, for example:

• Data validation: The main aspect in data validation while writing a producer system is to perform basic data validation tests on the data which is to be written on the Kafka cluster. The examples could be conformation to the schema and not null values for Key fields. By not doing data validation, you are risking breaking downstream consumer applications and affecting the load balancing of brokers as data may not be partitioned appropriately.
• Exception handling: It is the sole responsibility of producer and consumer programs to decide on program flows with respect to exceptions. ...