使用 Raspberry pi + fluentd + gcp cloud logging, big query 做iot 資料搜集與分析
-
Upload
simon-su -
Category
Technology
-
view
631 -
download
3
Transcript of 使用 Raspberry pi + fluentd + gcp cloud logging, big query 做iot 資料搜集與分析
Linker Networks Corp.http://www.linkernetworks.com
Use Raspberry Pi + Fluentd + GCP Cloud Logging, BigQuery
as IoT Data Collect & AnalysisSimon Su @ LinkerNetworks
var simon = {/** I am at GCPUG.TW **/};
simon.GDE = 'https://developers.google.com/experts/people/peihsin-su';
simon.nodejs = ‘http://opennodes.arecord.us';
simon.googleshare = 'http://gappsnews.blogspot.tw'
simon.nodejsblog = ‘http://nodejs-in-example.blogspot.tw';
simon.blog = ‘http://peihsinsu.blogspot.com';
simon.slideshare = ‘http://slideshare.net/peihsinsu/';
simon.email = ‘[email protected]’;
simon.say(‘Let’s go Cloud!');
BigQuery
● TB Level data analysis
● Fast mining tool
● SQL like interface
● Multi-dataset join support
● Cheap & Pay by Use
Google BigQuery - Big data with SQL like query feature, but fast...
Architecture
● BigQuery: Data Warehouse and
support query and analysis
● Cloud Logging: Log gateway
and easy to transfer to one or
multi destination with guarantee
● Fluentd: Device layer gateway
and support multi source for
user to extend http(s) input
tcp/udp input
forward / secure forward
other inputs
● Customize schema● Co-relation● Higher quota and
performance
● Web console● Export available for BQ,
GCS, Pub/Sub● Configure input source
curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
sudo bash install-logging-agent.sh
service google-fluentd start
Install Cloud Log AgentFrom: https://cloud.google.com/logging/docs/agent/installation
Configure # /etc/google-fluentd/google-fluentd.conf<source> type http port 9880</source><source> @type forward port 24224</source><filter *.**> @type stdout</filter># Do not collect fluentd's own logs to avoid infinite loops.<match fluent.**> type null</match># Configure all sources to output to Google Cloud Logging<match *.**> type google_cloud # Set the chunk limit conservatively to avoid exceeding the limit # of 10MB per write request. buffer_chunk_limit 2M flush_interval 5s # Never wait longer than 5 minutes between retries. max_retry_wait 300 # Disable the limit on the number of retries (retry forever). disable_retry_limit # Use multiple threads for processing. num_threads 8</match>
Set config to open fluentd forward and also screen output for debug..
Test run & watch log...
# service google-fluentd start
# tail -f /var/log/google-fluentd/google-fluentd.log
2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA:
{"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"}
2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA:
{"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"}
2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA:
{"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"}
….(skip)
Install Docker into Fluentd
http://blog.hypriot.com/downloads/
Getting start with fluentd… (https://peihsinsu.gitbooks.io/fluentd-in-action/content/)
Build your Raspberry Pi Fluentd - Dockerfile
# DockerfileFROM hypriot/rpi-alpine-scratchMAINTAINER linkeriotRUN adduser -D -g '' -u 1000 dockerENV FLUENTD_VERSION 0.14RUN apk --update add \ build-base \ ca-certificates \ ruby-dev \ && \ rm -rf /var/cache/apk/* && \ echo 'gem: --no-document' >> /etc/gemrc && \ gem install fluentd -v $FLUENTD_VERSION && \ apk del build-baseWORKDIR /fluentdRUN mkdir log etc plugins config.dCOPY fluent.conf /fluentd/etc/ONBUILD COPY fluent.conf /fluentd/etc/ONBUILD COPY plugins/* /fluentd/plugins/ENV FLUENTD_OPT=""ENV FLUENTD_CONF="fluent.conf"EXPOSE 24224EXPOSE 8888VOLUME ["/fluentd/log"]CMD fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT
Build your Raspberry Pi Fluentd - fluentd.conf
# fluentd.conf@include /fluentd/config.d/*.conf<label @mainstream> <filter **> @type record_transformer <record> linker_devid "#{ENV['TAG']}" </record> </filter> <match **> @type forward send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 16 hard_timeout 60s <server> name loggateway host "#{ENV['FDN']}" port "#{ENV['FPORT']}" weight 60 </server> <secondary> @type file path /var/log/fluent/forward-failed </secondary> </match></label>
Run from your images
docker run -d -p 24224:24224 --restart=always \ -e FDN=[remote-server-ip] \ -e FPORT=[remote-server-port] \ -e TAG=[your-tag] \ -v /var/log:/fluentd/log \ -v /data/cfg:/fluentd/config.d \ yout-image-name
Run using linker build images
docker run -d -p 24224:24224 --restart=always \ -e FDN=[remote-server-ip] \ -e FPORT=[remote-server-port] \ -e TAG=[your-tag] \ -v /var/log:/fluentd/log \ -v /data/cfg:/fluentd/config.d \ linkeriot/iot-log-agent