Mastering Apache Storm

Configuration of Logstash

Now, we are going to define the Logstash configuration to consume the Apache logs and store them into Kafka.

Create a logstash.conf file and add the following lines:

input {  file {    path => "PATH_TO_APACHE_LOG"    start_position => "beginning"  }}output {  kafka {    topic_id => "TOPIC_NAME"    bootstrap_servers => "KAFKA_IP:KAFKA_PORT"  }}

We should change the following parameters in the preceding configuration:

• TOPIC_NAME: Replace with the Kafka topic you want to use for storing the Apache log
• KAFKA_IP and KAFKA_PORT: Specify the comma separated list of all the Kafka nodes
• PATH_TO_APACHE_LOG: The location of Apache log file on the Logstash machine

Go to Logstash home directory and execute the following command to start the ...