Let's assume that we have an application deployed on an application server. That application is logging on to an access log. Then how can we analyze this access log using a dashboard? We would like to create a real-time visualization of the following info:
- Number of various response codes
- Total number of responses
- List of IPs
Proposed technology stack:
- Filebeat: To read access log and write to Kafka topic
- Kafka: Message queues and o buffer message
- Logstash: To pull messages from Kafka and write to Elasticsearch index
- Elasticsearch: For indexing messages
- Kibana: Dashboard visualization
In order to solve this problem, we install filebeat on Appserver. Filebeat will read each line from the access log and write to the kafka topic ...
