Use case

Let's assume that we have an application deployed on an application server. That application is logging on to an access log. Then how can we analyze this access log using a dashboard? We would like to create a real-time visualization of the following info:

  • Number of various response codes
  • Total number of responses
  • List of IPs

Proposed technology stack:

  • Filebeat: To read access log and write to Kafka topic
  • Kafka: Message queues and o buffer message
  • Logstash: To pull messages from Kafka and write to Elasticsearch index
  • Elasticsearch: For indexing messages
  • Kibana: Dashboard visualization

In order to solve this problem, we install filebeat on Appserver. Filebeat will read each line from the access log and write to the kafka topic ...

Get Modern Big Data Processing with Hadoop now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.