ELK for Data Analysis

Apps that mine data and provide insights are very popular as they help in planning the next course of action. Trend analysis, sentiment analysis, event prediction etc. are common applications of data mining. As they say, data is the next oil.

ELK stack comprising of Elasticsearch, Logstash and Kibana is a popular stack to analyze data. In a nutshell:

• Logstash gathers data from structured and unstructured sources
• Elasticsearch indexes and keeps it ready for searching
• Kibana uses the data for visualization 

After installation, a few quick tasks that one would do to get started with the stack are:

• Load data (log files/JSON/anything) using the Logstash conf file into Elasticsearch (default port:9200) 
• the conf file should include input, filter (blank is okay) and output
• the input block can be file, twitter, or any of the several sources that Logstash supports
• the Elasticsearch block in the output block should include the index name (this will match inside Kibana)
• In Kibana (default port:5601), create the index using the index name mentioned above
• Go to Discover and select new search, this will throw some results. If it does not, then change the time range until results are shown
• Next, go to Visualize - and create a visualization
• Inside the visualization, select the index that you created, select some fields from inside the indexed schema and you are done!

Some of the common commands used are:
To list indices  http://localhost:9200/_cat/indices?v
To delete an index curl -XDELETE localhost:9200/shakespeare
To check Elasticsearch version curl -XGET localhost:9200