Puppalitics Insights Puppet landscape v1.0...
Transcript of Puppalitics Insights Puppet landscape v1.0...
![Page 1: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/1.jpg)
Use Big Data tooling to get insights in a Puppet landscapePuppalitics
24-09-15Version 1.0 by Kevin Schaul
![Page 2: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/2.jpg)
Who am I?� Kevin Schaul� Data engineer @ Itility� Analytics toolset
- Splunk- Python- R / Stata
24-09-15 www.itility.nl 2
![Page 3: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/3.jpg)
Big Data = Hot (or not?)
24-09-15 www.itility.nl 3
![Page 4: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/4.jpg)
Main Question of today
How to use a Big Data toolset on Puppet andother machine generated data
(and why should I care?)
24-09-15 www.itility.nl 4
![Page 5: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/5.jpg)
Agenda� Introduction to Splunk� Data Collection� Analytics approach� Dashboard demo� Hypothesis session� Questions
24-09-15 www.itility.nl 5
![Page 6: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/6.jpg)
Splunk
24-09-15 www.itility.nl 6
![Page 7: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/7.jpg)
Why Splunk� A Big Data toolset (for ingest, store, query, visualize)
� Connects any data, of any size, any location and any source- Splunk connects to IT infra through API’s, real-time data gathering- Combine, correlate and visualize machine-generated IT Data
Splunk provides a solution, not necessarily the solution, for the problem:
“How to manage infrastructure more efficiently than in the past”
24-09-15 www.itility.nl 7
![Page 8: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/8.jpg)
Data collection� Python scripts perform API calls to PuppetDB to obtain data
� 6 categories- Events- Facts- Resources- Catalogs- Reports- Nodes
� Additional source: Github, vCenter, ServiceNow24-09-15 www.itility.nl 8
![Page 9: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/9.jpg)
24-09-15 www.itility.nl 9
Example data collection
![Page 10: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/10.jpg)
Puppet API-endpoints� PuppetDB data can be queried with a REST API� Queries are processed immediately� Multiple endpoints
24-09-15 www.itility.nl 10
![Page 11: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/11.jpg)
Getting data in
Source: Patrick Maddox – Puppet labs
24-09-15 www.itility.nl 11
NODE
PUPPETMASTER
1 Facts
2 Catalog
4Report
3Report
SSLsecure
Report Collector
PuppetDB
![Page 12: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/12.jpg)
Getting data in – Puppet to Splunk
24-09-15 www.itility.nl 12
NODE
PUPPETMASTER
1 Facts
2 Catalog
4Report
3Report
SSLsecure
Report Collector
PuppetDB
ForwardingAgent
![Page 13: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/13.jpg)
Hypothesis session� Regular sessions to refine dashboard needs� Hypothesis approach for out-of-the-box thinking
24-09-15 www.itility.nl 13
viaPhysicalmodels
Observed data
HYPOTHESISExperiments
Scientific method
Simulated data
Simulation
Scientific method extended
viaMathematical
modelsRepresentations ofThe real system
![Page 14: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/14.jpg)
Hypothesis session
Hypothesis True False ResultA Puppet Master has < 70% CPU loadLess than 1% of a Puppet Master log contain errors95% of runs take less than 10 secondsPuppet runs never fail longer than 24 hours in a rowWindows R2 servers show less Puppet deviations(events) than non-R2 servers
24-09-15 www.itility.nl 14
![Page 15: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/15.jpg)
Demo
24-09-15 www.itility.nl 15
![Page 16: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/16.jpg)
Hypothesis brainstorm
Think of hypotheses that intrigue you:
24-09-15 www.itility.nl 16
PuppetDBEngineer/Module-
development
In combinationwith any other
datasource
Insights fromSplunk backto Puppet
![Page 17: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/17.jpg)
Way forward� Connect with multiple data sources
- vCenter- Netapp- Github- ServiceNow
� Which other data sources would you add?
� Other tools which provide similar or additional insights?
24-09-15 www.itility.nl 17
![Page 18: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/18.jpg)
End notes� Tutorial
- https://www.splunk.com/web_assets/v5/book/Exploring_Splunk.pdf
� Documentation- http://docs.splunk.com/Documentation/Splunk
� Best practices- http://wiki.splunk.com/Community:More_best_practices_and_processes
24-09-15 www.itility.nl 18
Tutorial Documentation Best practices
![Page 19: Puppalitics Insights Puppet landscape v1.0 240915files.meetup.com/7364292/Puppet_Meetup-Puppalitics_Insights_Pup… · Why Splunk A Big Data toolset (for ingest, store, query, visualize)](https://reader033.fdocuments.net/reader033/viewer/2022042221/5ec7fa634783a43a6215629d/html5/thumbnails/19.jpg)
Contact us at [email protected] of presentation