Exploring data with Elasticsearch and Kibana

Patrick Puecher, Developer SFSCon, November 10th 2017

Elastic Stack (the ELK Stack)

Elasticsearch Kibana

BeatsLogstash

Elasticsearch- Distributed, RESTful search engine- Based on Lucene- Written in Java- Apache License- APIs

- Indexes APIs- Document APIs- Search APIs- …

Kibana- Visualize your data- Histograms, line graphs, pie charts, …- Time Series with Timelion

Logstash- Server-side data processing pipeline- How Logstash works

- Inputs- file, syslog, redis, beats, …

- Filters- split, mutate (convert, rename, add_field, remove_field), date, …

- Outputs- elasticsearch, file, email, exec, …

Beats- Send data from machines to Logstash and Elasticsearch- Beats family

- Filebeat- log files

- Metricbeat- system and service metrics

- Packetbeat- network data

- Winlogbeat- windows event logs

- Heartbeat (beta)- uptime monitoring

Demo time! 1Big Data 4 Tourism

- Input: CSV file

- Data processing: Java API

- Visualizing: Kibana

2Instagram Data

- Input: JSON files

- Data processing: Logstash & jq

- Visualizing: Kibana

./jq

Demo 1: Big Data 4 Tourism- Collect and visualize accommodation enquiries and bookings

○ Create Elasticsearch index○ Tourism Data Collector (https://github.com/idm-suedtirol/big-data-for-tourism)

- Upload and process CSV files- Written in Java- Open Source ツ

○ Kibana to visualize

- Big Data 4 Tourism working group by IDM Südtirol - Alto Adige○ Brandnamic, HGV, IDM Südtirol - Alto Adige, Internet Consulting, Limitis, LTS, Peer GmbH,

SiMedia …

PUT /tourism-data_2017{ "mappings" : { "enquiry" : { "properties" : { "arrival" : { "type" : "date", "format" : "epoch_millis||date" }, "departure" : { "type" : "date", "format" : "epoch_millis||date" }, "country.code" : { "type" : "keyword" }, "country.name" : { "type" : "keyword" }, "country.latlon" : { "type" : "geo_point" }, "adults" : { "type" : "byte" }, "children" : { "type" : "byte" }, "destination.code" : { "type" : "short" }, "destination.name" : { "type" : "keyword" }, "destination.latlon" : { "type" : "geo_point" }, "category.code" : { "type" : "byte" }, "category.name" : { "type" : "keyword" }, "booking" : { "type" : "boolean" }, "cancellation" : { "type" : "boolean" }, "submitted_on" : { "type" : "date", "format" : "epoch_millis||date||date_hour_minute_second" }, "length_of_stay" : { "type" : "short" } } } }}

Demo 1: Create Elasticsearch index"2015-01-01","2015-01-03","","2","0","21027","1","1","0","2015-01-01T01:59:00"

Demo 1: Tourism Data Collector

Demo 1: Visualize sample data (I)

1

2

Demo 1: Visualize sample data (II)

2

1

3

How to get Instagram postsfrom South Tyrol?

Demo 2: Instagram dataMission: Must-see places for route planner

Demo 2: Instagram data1. Get a shape file of South Tyrol (http://geoportal.buergernetz.bz.it/)

Demo 2: Instagram data1. Get a shape file of South Tyrol (http://geoportal.buergernetz.bz.it/)2. Use QGIS to create a regularly-spaced grid of points

Demo 2: Instagram data1. Get a shape file of South Tyrol (http://geoportal.buergernetz.bz.it/)2. Use QGIS to create a regularly-spaced grid of points3. Export points as latitude and longitude coordinates

Demo 2: Instagram rate limits & scopes- Global rate limits on the Instagram platform

(https://www.instagram.com/developer/limits/)- 5000 API calls / hour

- Scopes- public_content - to read any public profile info and media on a user’s behalf

(applications no longer accepted) :’(

Demo 2: Instagram search API

{ "data":[ { "id":"1614761577805643016_1157147895", "user":{ "id":"1157147895","full_name":"Marc Hochstaffl","profile_picture":"…","username":"marc_hochstaffl" }, "images":{ "thumbnail":{"width":150,"height":150,"url":"…"},"low_resolution":{…},"standard_resolution":{…} }, "created_time":"1506714602", "caption":{ … }, "user_has_liked":false, "likes":{"count":181}, "tags":["sam","karposfasttrail","autumn\ud83c\udf41","ahrntal","hundskehljoch"], "filter":"Normal", "comments":{"count":3}, "type":"image", "link":"https://www.instagram.com/p/BZoyRmCDf0I/", "location":{"latitude":47.05,"longitude":12.06667,"name":"Hundskehljoch","id":1033509208}, "attribution":null, "users_in_photo":[] } ], "meta":{"code":200}}

https://api.instagram.com/v1/media/search?lat=47.051124693028548&lng=12.039835734128651&access_token=key&distance=5000

PUT /_template/ instagram{ "template" : "instagram*" , "mappings" : { "_default_" : { "properties" : { "images" : { … }, "carousel_media" : { … }, "geoip" : { "type": "geo_point" }, "users_in_photo" : { … }, "link" : { … }, "created_time" : { "type" : "date", "format" : "strict_date_optional_time||epoch_second" }, "caption" : { … }, "type" : { "type": "keyword" }, "tags" : { "type": "keyword" }, "filter" : { "type": "keyword" }, "likes.count" : { "type" : "integer" }, "comments.count" : { "type" : "integer" }, "location" : { … }, "id" : { "type" : "keyword" }, "user" : { … } } } }}

Demo 2: Create Elasticsearch index

input { http_poller { urls => { insta1 => "/v1/media/search?lat=47.051124693028548&lng=12.039835734128651&access_token=key&distance=5000" insta2 => "/v1/media/search?lat=47.049359378811829&lng=12.105570031601609&access_token=key&distance=5000" … } keepalive => false cookies => false request_timeout => 30 schedule => { every => "10m" } codec => "json" }}

output { elasticsearch { hosts => ["127.0.0.1:9200"] index => "instagram-%{+YYYYMM}" document_id => "%{id}" }}

Demo 2: Grab and store posts using Logstash (I)

Demo 2: Grab and store posts using Logstash (II)

filter { split { field => "data" } if [data][id] { mutate { convert => { "[data][comments][count]" => "integer" "[data][likes][count]" => "integer" } rename => { "[data][created_time]" => "[created_time]" "[data][images]" => "[images]" "[data][comments][count]" => "[comments_count]" … "[data][id]" => "[id]" "[data][user]" => "[user]" "[data][likes][count]" => "[likes_count]" } add_field => [ "geoip", "%{[location][latitude]},%{[location][longitude]}" ] remove_field => ["data", "meta"] } date { match => ["[caption][created_time]" , "UNIX"] target => [ "[caption][created_time]" ] } date { match => ["[created_time]" , "UNIX"] remove_field => [ "[created_time]" ] } }}

Demo 2: Grab and store posts using Linux Shell

#!/bin/bash

insta=( 'https://api.instagram.com/v1/media/search?lat=47.051124693028548&lng=12.039835734128651&access_token=key&distance=5000' 'https://api.instagram.com/v1/media/search?lat=47.049359378811829&lng=12.105570031601609&access_token=key&distance=5000')

count=0while [ "x${insta[count]} " != "x" ]do MIN=`date -d '11 minutes ago' +"%s"` # reduce bandwidth URL="${insta[count]} &min_timestamp= $MIN" curl -s $URL | jq -c '.data[] | .geoip = ((.location.latitude | tostring) + "," + (.location.longitude | tostring)) | {'index': {'_index': ("instagram-" + (.created_time | ' tonumber' | gmtime | strftime("%Y%m"))), ' _type': "feed", ' _id': .id}}, .' | curl -s -XPOST localhost:9200/_bulk --data-binary @- & # start in background

if [ $((($count + 1) % 20)) = 0 ]; then # parallelize wait fi

count= $(( $count + 1 ))done

Use a cron job to run the shell script every 10 minutes!

Demo 2: Visualize posts by dateJuly - August

Demo 2: Daily rhythm (1 for monday … 7 for sunday)

Sunday… 2 pm - 7 pm

Demo 2: Top locations by number of posts (I)

Riva del Garda

Trento

BolzanoTre Cime

Merano

Demo 2: Top tags by number of postssnukiefulmartinisisters

giuliavalentina

valentinavignali valentinavignali

querly_official

igworld_globalmanueldietrichphotography

Demo 2: Top travellers

Demo 2: Influencer Trentino

Demo 2: Influencer South Tyrol

Demo 2:

Glassyhuman

Data-Driven Advertising