Sydney Python Presentation (October 2010) - Splunk
-
Upload
kelvin-nicholson -
Category
Technology
-
view
120 -
download
0
description
Transcript of Sydney Python Presentation (October 2010) - Splunk
Splunk and Python
Sydney Python October 2010
Kelvin Nicholson
What is Splunk?
“Splunk is the world’s leading software used to monitor, report and analyze live streaming IT data as well as terabytes of historical data – located on-premises or in the cloud.” -Splunk.com
“Splunk is like google for log files.” -Kelvin
Installing Splunk (on Ubuntu)
$ sudo dpkg -i splunk-4.1.5-85165-linux-2.6-intel.deb$ sudo splunk enable boot-start$ sudo /etc/init.d/splunk start
Splunk Welcome Screen
Configuring Splunk●Configure Splunk to allow syslog traffic●Configure devices to send syslog to Splunk
○ Linux (syslog-ng) destination loghost { udp("192.168.83.11" port (514)); }; log { source(s_all); destination(splunk); };
●Cisco IOS no logging console no logging monitor logging 192.168.83.11
● OSSEC <syslog_output> <server>192.168.83.11</server> <port>8514</port> </syslog_output>
Splunk Search Screen
Why I Like Splunk (Abridged)
●Dashboards of Search terms■ Security alerts “login failed for”■ STP network issues (“LEARNING AND FORWARDING”■ Duplex mismatches■ Wildcard searches, e.g. “-server2k3-”
●My “WTF” filter (easy filter building)●Beautiful trending (“cold start” AND “switch01”)
Splunk Simple Filtering
Extending Splunk with Python
●REST API. (Search only)●Custom search command. (iplocation)●Configuring scripted alerts. (tweet X alert)●Directly to backend using Splunk's built-in
modules. (Full module access)
Accessing Splunk Datastore>>> import splunk.auth, splunk.search>>> key = splunk.auth.getSessionKey('admin','changeme')>>> my_job = splunk.search.dispatch('search sypy', namespace='search')>>> event_list = []>>> for event in my_job.events:... event_list.append(event.fields)... >>> print event_list
kelvinn@splunk:/opt/splunk/bin$ ./splunk cmd python
[{'_si': splunk,main, 'index': main, 'sourcetype': syslog, 'source': udp:514, '_kv': 1, 'splunk_server': splunk, '_time': 2010-10-06T19:40:37+1100, 'host': 192.168.83.5, '_sourcetype': syslog, '_raw': Oct 6 19:40:37 192.168.83.5 Oct 6 19:40:38 mini kelvinn: hello SyPy, hope you are doing well., '_serial': 0, '_cd': 0:275}, {'_si': splunk,main, 'index': main, 'sourcetype': syslog, 'source': udp:514, '_kv': 1, 'splunk_server': splunk, '_time': 2010-10-06T19:39:33+1100, 'host': 192.168.83.5, '_sourcetype': syslog, '_raw': Oct 6 19:39:33 192.168.83.5 Oct 6 19:39:34 mini kelvinn: sypy, '_serial': 1, '_cd': 0:251}]
>>> event_list[0]['_raw']Oct 6 19:40:37 192.168.83.5 Oct 6 19:40:38 mini kelvinn: hello SyPy, hope you are doing well.
Splunk Architecture
CherryPy built-in, sweet. What can we do with that?
Built-in CherryPy Funkelvinn@splunk:/opt$ cat splunktest.py import cherrypyimport splunk.auth, splunk.search
def get_splunk_data():key = splunk.auth.getSessionKey('admin','changeme') # replace with your credentialsmy_job = splunk.search.dispatch('search sypy', namespace='search', earliest_time='-24h')
event_list = []for event in my_job.events:event_list.append(event.raw)return event_listclass HelloWorld:def index(self):splunk_list = get_splunk_data()return str(splunk_list)index.exposed = True
cherrypy.config.update({'server.socket_host': '0.0.0.0','server.socket_port': 9999,})cherrypy.quickstart(HelloWorld())kelvinn@splunk:/opt$ /opt/splunk/bin/splunk cmd python /opt/splunktest.pyP.S. I'm not a CherryPy expert, but it looks pretty fun.
View CherryPy Page
Resources + ThanksSplunk introduction:
http://www.splunk.com/base/Documentation/4.1.5/Installation/Splunksarchitectureandwhatgetsinstalled
Splunk REST Search (with Python httplib example):
http://www.splunk.com/base/Documentation/4.1.5/Developer/RESTCreateSearch
Custom search command (iplocation):
http://www.splunk.com/base/Documentation/latest/SearchReference/Customsearchiplocation
How to write custom alerts:
http://www.splunk.com/base/Documentation/4.1.5/Admin/Configurescriptedalerts
Using Splunk's built-in Python modules:
http://answers.splunk.com/questions/14/can-i-use-splunks-built-in-python-sdk-in-my-own-scripts
Some information about Splunk's Python SDK:
http://www.splunk.com/base/Documentation/4.1.5/Developer/PySDK
Thanks.