ELK is an open-source stack for capturing, indexing and visualizing data from various sources.
If you have any type of software applications running on in a production environment, you depend a lot on the log files generated by these applications to understand application behavior and debug issues. Most of the times such applications run on hundreds or even thousands of machines in production environment and they spit out millions (or even billions) of lines of log messages. There are various tools to aggregate these log messages and analyze them to get insights into usage patterns, errors, etc.
In this blog post I am going to talk about on of the ways to aggregate and analyze log files using various open source software. This tool is popularly known as ELK which stands for Elasticsearch, Logstash and Kibana. Althoug those are the main components of this tool, there will be additional tools and custom programs that you will have to write to efficiently collect and analyze your log messages.
High level diagram
Log Analysis using ELK