Data Driven Security, from Gartner Security Summit 2012
-
Upload
nick-galbreath -
Category
Technology
-
view
3.940 -
download
3
Transcript of Data Driven Security, from Gartner Security Summit 2012
Copyright © 2012 Splunk, Inc.
Data-Driven Security:Managing Risk at EtsyNick Galbreath @ngalbreathDirector of Engineering - EtsyGartner Security SummitNational Harbor, MDJune 12, 2012
Agenda
Who am I? Who is Etsy? Splunk at Etsy?
Web Application Security
Account Takeover
Payments and PCI
Credits, Data, Further Reading
2
3
@ngalbreath
http://slidesha.re/KPvHYu
Whois Nick GalbreathDirector of Engineering at Etsy covering:– Fraud– Security– Support Engineering– (and other stuff outside of this talk)
Software Development background inE-Commerce and Social MediaBooks, Patents, Oh My… http://client9.com/
4
2012-06-12
is my
two year
anniversary
at Etsy
2012-06-12
is my
two year
anniversary
at Etsy
5
$525,000,000 in community sales875,000 active sellers41MM unique visitors15MM registered members150 countries
What Could Possibly Go Wrong?• Marketplace Risk like Big Auction Site• Payment Risk like Payments Company• Social Risk like that Big Social Network
With a member base frequently:• New to Etsy• New to Running a Business• New to the Internet
6
Photo Credit: Rod Ramsey http://bit.ly/KnI8uB
To Make It More Interesting:Continuous Deployment
On average, there are 50+ production code changes per day. So when we have a problem:Is it an operations problem?Is it a development problem?Is it a product problem causing
complaints to come in?Or is it an attack?
7
Learn more http://bit.ly/KFYYlZ
Learn more http://bit.ly/KFYYlZ
Old Workflow: #notwinningLogging into production network (!)Finding the right fileUnzipping the right fileGrepping
• Writing very clever scripts to extract data• Writing more clever scripts to merge data• Making a report – in plain text• Alerting
8
34 minutes for one day’s log for nothing!
34 minutes for one day’s log for nothing!
Splunk installed at Etsy mid-2010
9
Serious New Technology FatigueWhy don’t we use a Real Database with SQL?Grep technology works&^#*&@^*#^%^ YAQL – Yet Another Query Language
"Hey. .. let's go try this NEW thing!" (door slamming shut)
"Sorry.... we're closed.”Steve Martin. Comedy is not pretty. 1979. Track 8 ~2:45
L’OutrageThen a colleague:• Didn’t know Etsy’s stack (new)• Remote and out of office• Didn’t have production access• Didn’t know any of my very clever scripts• Not experienced with Splunk• In about 30 minutes
whips up a real-time email alert for a velocity check on a particular URL
10
I only have one thing to say about this…..
OH, YEAAHH!400+GB indexed per day30+ TB total storage60+ data sources from“hundreds of servers” (via central syslog aggregation)
11
Data-Driven Security
12
Data-Driven By Mat Edelson. John Hopkins Engineering Magazine, Fall 2011http://eng.jhu.edu/wse/magazine-fall-11/item/data-driven/Illustration by Mark McGinnis No association, just a great article & illustration
Three examples of how we use data and Splunk to help make Etsy a safer place to conduct business.
•Web Application Security•Account Takeover•Payments and PCI
That said we are barely scratching the surface of Splunk!
WebApp Security
Make Security Visible
14
Your peers actually are interested in security. But are you letting them?
Turn security from a binary event intoa continuous event.
Detect the Steps
15
A journey of a thousand miles begins with a single step. Lao-tzu, China 600BC
A single breach begins with a journey of a thousand steps.Nick Galbreath, USA 2012AD
SQLi, XSS, CSRF
16
source=“info.log” log_name_space=“SECURITY” attacktype=“XSS”
That was easy
That was easy
SQLi, XSS, CRSF
17
source=“info.log” log_name_space=“SECURITY” attacktype=“SQL”| geoip ip
Paints a different picture
Paints a different picture
The Dumbest Check Possible for SQLiWe have some snazzy technology for detecting SQLi in Splunk, but you don’t need it to get started:
source=access.log
(uri="*UNION+ALL*" OR uri="*UNION%20ALL*”)
Will wildly undercount but also low false positive rateWill detect scans from various toolsWill get you started in making security visible
18
SQLi and Database Errors
19
• We use Splunk to alert on any database syntax errors too.• SQLi attacks and probes will likely trigger a bust of syntax errors
if code doesn’t properly sanitize data
That was
closeThat was
close
source="error.log" ( "syntax error" NOT "smarty" NOT "ClientLogger" ) | eval event=_raw | table event'
Do the same with server 500 errors,
core dumps
Investigating Rent-A-CPU Trafficsource=“access.log”| lookup datacenter-cidrs provider_cidr AS true_client_ip OUTPUTNEW provider_name | where isnotnull(provider_name) | top provider_name
20
Public DataSee Appendix
Public DataSee Appendix
SANS ISC 10K Sources
21
source=“access.log” | where isnotnull(true_client_ip) | lookup isc-bad-ips src_ip AS true_client_ip | where isnotnull(rank) | table true_client_ip, rank, reports, attacks, last_seen | stats count by true_client_ip,rank | sort rank
Public DataSee Appendix
Public DataSee Appendix
Attacker-Driven Testing
Attacker-driver testing augments Etsy’s proactive security measuresSplunk alerts us on potential attacks using a number of parametersWhat URLs are being targeted? Maybe they found something?Can it be reproduced? (sometimes completely automated validation)Fixes can be pushed out that day, if not within minutes.
22
“I thought I found something but then it stopped working…” Email to [email protected] from ethical hacker
Security Post-MortemsFor any security vulnerability, found either external or internal, exploited or not, we hold “blameless post-mortems”Use to teach about security issues
e.g. review OWASP Top 10 http://bit.ly/fXsJg6Can we make it so this mistake doesn’t happen again or can be automatically detected?A Key to post-mortem is know when something started and when it ended. Logs “at your fingertips” via Splunk helps greatly (and absolutely essential for actual incidents)
23
Account Takeover
Account Takeover
25
• Stolen credentials• Brute forcing of credentials• Using account takeover of email to
further takeover other accounts
Horrible for victim and really slow to clean up
Many Users Failing to Sign-in from One IP
26
'source=“info.log” log_namespace=“login” reason="wrong password” true_client_ip!=38.117.156.XXX| dedup etsy_username,true_client_ip | transaction true_client_ip | where eventcount > XXXX| table true_client_ip,etsy_username| geoip true_client_ip | table true_client_ip,true_client_ip_countryname,etsy_username'
Brute Forcing Passwords?source=”info.log”
log_namespace="login”
reason="wrong password"
true_client_ip!=38.117.156.XXX
| transaction etsy_username
| where eventcount > XXXX
| table etsy_username,true_client_ip,eventcount
| sort -eventcount
27
Frequency Buckets set in Splunk Dashboard
People will try 100 passwordsmanually
People will try 100 passwordsmanually
I Forgot My Password x1000
28
source=“/web/access.log” request_uri=/forgot_password.php http_method=POST | transaction true_client_ip | where eventcount > XXX| table true_client_ip,eventcount| sort –eventcount Hello from
Serbia!Hello from
Serbia!
Not just fraud… has disclosed problems in email transport and product problems with our reset flow
Apply the same analysis to other things that should not change much
– Payment cards– Email addresses– Passwords (successful change)– Regular physical addresses
29
CAPTCHA
30
Splunk 2x2 dashboard keeps us in-the-know on how often CAPTCHAs are being shown, to whom, and how often they pass.
reCAPTCHA http://www.google.com/recaptcha
Integrated into Support ToolsSplunk is glued into our internal tools used by General Support and MITS (Marketplace Integrity / Trust & Safety) teams.
31
Payments and PCI
Payments @ EtsyRamping up on our own payments platformFull PCI EnvironmentWith separate Splunk installation
33
This space intentionally left blank.
Alerting on Unusual Payment ActivityAll the WebApp security and account take- over rules apply, along with special checks for payment activityAbnormally large paymentsPayment velocityVery small payments (skimming?)The usual IP address checks.
34
Part of a largerpayment risk solution
Part of a largerpayment risk solution
Compliance and Reporting
Instead of building custom applications with fuzzy requirements“Log it, let Splunk figure it out later” Even the business guys can use it for ad-hoc queries. Unexpected side effect: removing and/or changing data is really hard. This is good. Compare to SQL. (Splunk also has a secure log system)Easy to make reportsPCI QSA so far says this meets PCI requirements.
35
Internal RiskAgain, instead of build out of new application (with fuzzy requirements) Log It, Splunk it later.Who, is what making what changesWho is looking at potentially sensitive dataAnd alert on it.Used in payments and main support applications
36
Etsy Support and MITS 2012100% Good Eggs
Team Etsy 2012
Credits
Acknowledgements
This presentation would not be possible without the hard work by:
Big thanks to everyone at Etsy in Engineering, Payments, Operations, Support and MITSAnd of course, the fine folks at Splunk!
38
Jerry SoungFraud and Risk Engineering
Zane LackeySecurity Engineering
Marcus BarczakOperations
Data and ReferencesDatacenter IP List:
https://github.com/client9/ipcat
ISC Top Troublemaker IPs:http://isc.sans.edu/ipsascii.htmlhttp://isc.sans.edu/sources.html
On Security and Continuous Deployment:http://bit.ly/KFYYlZ
Other presentations on Etsy and Security/Fraud/DevOps:http://slidesha.re/Kw5zdV http://slidesha.re/IMaavq http://slidesha.re/JGaU2s
39
Security Engineering and “Just Culture”Treating security mistakes as “accidents” (whether exploited or not)Based originally on health care initiativesPatient Safety and “Just Culture”, David Marx JD– http://psnet.ahrq.gov/resource.aspx?resourceID=1582– http://bit.ly/LhRHaT (presentation)
John Allspaw on Blameless Post-Mortems:http://codeascraft.etsy.com/2012/05/22/blameless-postmortems/
40
It’s time for questions!
Nick Galbreath@ngalbreath
www.etsy.com
http://slidesha.re/KPvHYu