Putting Big D ata to Work
-
Upload
roary-holloway -
Category
Documents
-
view
44 -
download
0
description
Transcript of Putting Big D ata to Work
Putting Big Data to Work
AURIMS/ANZUIAG Conference 2014
University of Adelaide 2
Who Am I
• Mathew Benwell
• Information Security Specialist at the University of Adelaide
• Worked in Information Security for 8 years
University of Adelaide 3
First, a Disclaimer
• I work in a highly technical field, there will be a technology slant to this talk!
• However, the concepts in this talk translate to non technical fields
• My experiences are with a specific product called Splunk
University of Adelaide 4
About This Presentation
• What is Big Data?
• Big Data at the University of Adelaide
• Technology Use Cases
University of Adelaide 5
What is Big Data
• Big Data 3 V’s
VarietyVelocity
Volume
University of Adelaide 6
How is Big Data Useful?
• Analyse very large data sets quickly
• Add context using variety
• Can help spot unusual events
University of Adelaide 7
How is Big Data Useful?
• Analysis
– Arithmetic operations
– Trending
– Anomalous data
University of Adelaide 8
How is Big Data Useful?
• Visualisations
University of Adelaide 9
A Simple Big Data Analytics Process
What do you want to know?
• Be precise!
What dataset holds the answer?
• If required data is not logged, start logging now!
Collect data • Multiple sources
Analyse and get the answer!
• Get answers all the time, continuously
University of Adelaide 10
Big Data and Audit
• Why wait for the good old 90 day review??
• Why not have our Big Data system tell use when an interesting event occurs?
• Why not take it a step further and add context
• Advise system owner at the time it occurred
University of Adelaide 11
Big Data and Audit
• During an Audit we ask lots of questions
The Question: – Who maintains access to privileged information?
– More specifically, we aim to identify those with unauthorised access to privileged information
Data that could support an answer:– System logs of changes to user groups
– List of groups which maintain privileged access
– Change system records
University of Adelaide 12
Big Data and AuditQuestion: Is Domain Admins group restricted to authorised IT personnel?
Required Data: Current Members + Active Directory event log that fires when someone is added to the Domain Admins group
Active Directory
BIG DATA SYSTEM
John Doe added to Domain Admins
Alert
Could be any question:• Monitoring changes to bank transaction file• Monitoring anomolous pay runs• Overrides in requisition request• Mismatched invoices
University of Adelaide 13
Big Data and Compliance
• Assist with Compliance to standards
• Payment Card Industry – Digital Security Standard (PCI-DSS)
• ISO 27001
University of Adelaide 14
Big Data and Compliance
• PCI-DSS
• Many technical controls
• Identify credit card data– Known pattern
– On the network
– Emails
University of Adelaide 15
Big Data and Risk
• We could use Big Data to identify financial risks
• Help prioritise risk treatment
• Identify unusual events– Transaction without a purchase order
– Higher than normal transaction
– High volume or scheduled, low value transactions
University of Adelaide 16
Big Data and Risk
• Profiling financial transactions
• Say we see a regular payment that occurs routinely
• Imagine the transaction one day starts occurring more frequently, or the transaction value changes significantly?
• This would be worth investigation
University of Adelaide 17
About This Presentation
• What is Big Data?
• Big Data at the University of Adelaide
• Technology Use Cases
University of Adelaide 18
What is Splunk
• First the most asked question!
Where did the name come from?
• Derived from the word ‘Spelunk’ ‘to explore caves, especially as a hobby’
Our customers told us that finding their IT problems was like "digging through caves with headlamps and helmets, crawling through the muck"
University of Adelaide 19
What is Splunk
• Software that can be used to store, analyse and report on Big Data!
• Simple licence model, based on the total volume of data consumed daily
• Highly scalable. Performance is only limited by hardware resources
University of Adelaide 20
What Data Can Splunk Consume
• Machine data, any data generated by a computer
– System logs
– Text files
– Databases
– Output from systems
University of Adelaide 21
Getting Data into Splunk
• Getting data into Splunk• Syslog
• Splunk Forwarder• Tail/dump any local file• Windows registry• WMI• Script• Active Directory
• DB Connect – Oracle, MSSQL, MySql, PostGres
• API – Push data using Splunk API
University of Adelaide 22
Splunk at the University of Adelaide• Community driven collaboration
University of Adelaide 23
Splunk at the University of Adelaide• Initially purchased for the Security team to help
deal with the ‘Phishing’ problem
• Uses are expanding significantly
• Quick Statistics– 3 Primary Servers
– Total 19TB storage capacity
– 89 billion events, 30 event sources
University of Adelaide 24
Splunk at the University of Adelaide• Google for your data
University of Adelaide 25
Splunk at the University of Adelaide• More than Google for your data
University of Adelaide 26
Splunk at the University of Adelaide• Analysis
University of Adelaide 27
About This Presentation
• What is Big Data?
• Big Data at the University of Adelaide
• Technology Use Cases
University of Adelaide 28
Use Case – Vulnerability Data
• System vulnerability data (Nessus, Nexpose, Qualys, etc)
University of Adelaide 29
Use Case – Vulnerability Data
• Add context, this data becomes far more Useful!– Is the system accessible from the Internet (Firewall
policies)
– Is the system actively being attacked (Intrusion Detection System data)
– Is the system actually vulnerable
• Additional information leads to a more educated assessment of impact and likelihood of occurrence
University of Adelaide 30
Use Case – Internet Charges
• AARNet users pay subscription costs
• Most Australian Universities control using quota systems
• Beginning 2014, the University of Adelaide removed the quota system
University of Adelaide 31
Use Case – Internet Charges
• Potential Financial Risk– High volume of Internet usage
– Internet usage is not cheap when you account for ~25k students!
– We have a budget to stick to
• What are we doing to control the cost?
• Big Data!!
University of Adelaide 32
Use Case – Internet Charges
University of Adelaide 33
Use Case – Internet Charges
• Constantly analysing Internet traffic
• Comparing our traffic with a list of unmetered content
• Applying technical controls to limit impact of known high cost, non University related activities
University of Adelaide 34
Use Case – Internet Charges
University of Adelaide 35
Putting Big Data to Work
• In Summary:– Big Data systems are very powerful
– Big Data principles can be applied to many needs, just ask the question
– Big Data can help find needles in many haystacks
• I hope you enjoyed my presentation!
• Thank You