Big problems with big data Hadoop interfaces · PDF fileBig problems with big data –...
-
Upload
vuongnguyet -
Category
Documents
-
view
257 -
download
7
Transcript of Big problems with big data Hadoop interfaces · PDF fileBig problems with big data –...
![Page 1: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/1.jpg)
Big problems with big data –Hadoop interfaces security
Jakub Kaluzny
![Page 2: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/2.jpg)
whoami
Sr. IT Security Consultant at SecuRing
• Consulting all phases of development
• penetration tests
• high-risk applications and systems
Researcher
• Hadoop, FOREX, MFP printers, proprietary network protocols
![Page 3: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/3.jpg)
Agenda
Big data nonsenses
Crash course on hacking Hadoop installations
Ways to protect big data environments
Expect some CVEs
![Page 4: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/4.jpg)
Results summary
no account
standard user
admin useradmin
privileges
data access
![Page 5: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/5.jpg)
WHAT IS HADOOP?Know your target
![Page 6: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/6.jpg)
Normal database
Users Roles Data Model
![Page 7: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/7.jpg)
Normal database architecture
http://hackaday.com/2014/04/04/sql-injection-fools-speed-traps-and-clears-your-record/
![Page 8: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/8.jpg)
Still normal database scenario
CWE-xxx: SQL Injection through license plate
http://hackaday.com/2014/04/04/sql-injection-fools-speed-traps-and-clears-your-record/
http://hococonnect.blogspot.com/2015/06/red-light-cameras-in-columbia.html
http://8z4.net/images/ocr-technology
![Page 9: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/9.jpg)
Normal database injection points
![Page 10: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/10.jpg)
Normal database
Users Roles Data Model
Clear rules
Clear target
![Page 11: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/11.jpg)
user db,
a lot of clients
criticalbanking data, one supplier
Anecdote
Only one common table
Q: Why don’t you split it into 2 dbs with a db link?
A: Too much effort and we want to have fast statistics
from all data.
![Page 12: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/12.jpg)
What is Hadoop?
https://www.flickr.com/photos/photonquantique/2596581870/
http://fiveprime.org/blackmagic.cgi?id=7007203773
![Page 13: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/13.jpg)
Hadoop architecture schema
![Page 14: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/14.jpg)
More on Hadoop
![Page 15: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/15.jpg)
Hadoop injection points
![Page 16: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/16.jpg)
Hadoop scenario
https://en.wikipedia.org/wiki/Moneygami
https://www.flickr.com/photos/mattimattila/8349565473
http://bigdataanalyticsnews.com/tag/hortonworks/
![Page 17: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/17.jpg)
• 21 PB of storage in a single HDFS cluster
• 2000 machines
• 12 TB per machine (a few machines have 24 TB each)
• 1200 machines with 8 cores each + 800 machines with 16 cores each
• 32 GB of RAM per machine
• 15 map-reduce tasks per machine
What is a lot of data?
![Page 18: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/18.jpg)
• Our latest assessment:
• 32 machines, 8 cores each
• 24TB per machine
• 64 GB of RAM per machine
• Almost 1 PB disk space and 2TB of RAM
What is a lot of data?
http://mrrobot.wikia.com/wiki/E_Corp
![Page 19: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/19.jpg)
Attacker perspective
https://plus.google.com/+Magiccardtrickszonetips
![Page 20: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/20.jpg)
RISK ANALYSISKnow your threats
![Page 21: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/21.jpg)
Who How What
Risk analysis
![Page 22: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/22.jpg)
• Business perspective: competitor, script-
kiddies, APT
• Technical perspective:
Who?
External attacker
• Anonymous
• Ex-employee
Insider
• Exployee (with some rights in Hadoop): user, admin
• Infected machine, APT
![Page 23: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/23.jpg)
Who How What
Risk analysis
![Page 24: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/24.jpg)
Full compromise
![Page 25: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/25.jpg)
Data safety vs. data security
![Page 26: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/26.jpg)
• Q: What will be stored? A: „We do not know what data will be stored!”
• Typical bank scenario
• Bigdata analytic says: „People who bought a dashcam are more likely to take a loan for a new car in the next month”
For what?
All transaction data
All sales data
All client datahttp://thewondrous.com/julia-gunthel-worlds-mosthttps://www.reddit.com/r/gifs/comments/37aara/calculations_intensify/
![Page 27: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/27.jpg)
For what? Data theft
![Page 28: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/28.jpg)
Other
Privilege escalation
• Authentication bypass
Abuse
• DoS
• Data tampering
![Page 29: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/29.jpg)
Who How What
Risk analysis
![Page 30: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/30.jpg)
How?
https://en.wikipedia.org/wiki/Dowsing#Rods
![Page 31: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/31.jpg)
WHAT HADOOP REALLY ISunder sales-magic-cloud-big-data cover
![Page 32: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/32.jpg)
Typical architecture
http://thebigdatablog.weebly.com/blog/the-hadoop-ecosystem-overview
![Page 33: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/33.jpg)
Apache Hue
http://techbusinessintelligence.blogspot.com/2014/11/tableau-software-cloudera-hadoop.html
![Page 34: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/34.jpg)
Hadoop injection points
Differs much amongst distros
![Page 35: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/35.jpg)
INTERFACES
![Page 36: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/36.jpg)
Hadoop
Distrosspecifics
Admin ifaces
external
ifaces
User ifaces
Interfaces
![Page 37: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/37.jpg)
OUR STORY WITH BIG DATA ASSESSMENT
a.k.a. crash course on hacking big data environments
![Page 38: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/38.jpg)
Hadoop
Distrosspecifics
Admin ifaces
external
ifaces
User ifaces
Interfaces
![Page 39: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/39.jpg)
USER INTERFACESfor employees and applications
![Page 40: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/40.jpg)
User interfaces
Hadoop
Distrosspecifics
Admin ifaces
external
ifaces
User ifaces
![Page 41: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/41.jpg)
User interfaces
Apache Hue
• Pig, Hive, Impala, Hbase, Zookeeper, Mahout, Oozie
Other
• Tez, Solr, Slider, Spark, Phoenix, Accummulo, Storm
H
D
A
E
U
![Page 42: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/42.jpg)
Is Hue an internal interface?
H
D
A
E
U
http://9gag.com/gag/awrwVL1/hue-hue-hue
![Page 43: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/43.jpg)
Apache Hue overview
H
D
A
E
U
http://gethue.com/
![Page 44: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/44.jpg)
Apache Hue DOM XSS
• var _anchor = $("a[name='" + decodeURIComponent(window.location.hash.substring(1)) + "']").last();
• Payload: URL/help/#<img src="x" onerror="alert(1)"> H
D
A
E
U
![Page 45: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/45.jpg)
Target old Hadoop installation (with Hue 2.6.1, Django 1.2.3)
Target a user with access to Hue
Send him XSS
Get access to all Hadoop data designated for the user
Apache Hue attack scenario
![Page 46: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/46.jpg)
Default configurations sucks
X-Frame-Options:ALLOWALL
![Page 47: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/47.jpg)
ADMIN INTERFACESfor admins and maintenance
![Page 48: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/48.jpg)
Hadoop
Distrosspecifics
Admin ifaces
external
ifaces
User ifaces
Admin interfaces
![Page 49: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/49.jpg)
Admin interfaces
Apache Ambari
• Provisioning, monitoring
Apache Ranger
• Security: authorization, authentication, auditing, data encryption, administration
Other
• Knox, Cloudbreak, Zookeeper, Falcon, Atlas, Sqoop, Flume, Kafka
H
D
A
E
U
![Page 50: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/50.jpg)
Apache Ambari
• About Ambari
http://www.slideshare.net/hortonworks/ambari-using-a-local-repository?next_slideshow=1
H
D
A
E
U
![Page 51: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/51.jpg)
Apache Ambari
http://www.slideshare.net/hortonworks/ambari-using-a-local-repository?next_slideshow=1
H
D
A
E
U
![Page 52: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/52.jpg)
Is Ambari an internal interface?
H
D
A
E
U
http://knowyourmeme.com/memes/facepalm
![Page 53: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/53.jpg)
Apache Ambari
• Standard users can sign into Ambari (WHY?)
• Low hanging fruits: directory listing by default, no cookie flags, no CSRF protection
• Interesting proxy script ->
H
D
A
E
U
![Page 54: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/54.jpg)
Apache Ambari REST API proxy
Standard request:
/proxy?url=http://XXXXXXXXX:8188/ws/v1/timeline/HIVE_QUERY_ID?limit=1&secondaryFilter=tez:true&_=1424180016625
H
D
A
E
U
Tampered request (logs accessible only from DMZ):/proxy?url=http://google.com/proxy?url=http://XXXXXXX:8088/logs/proxy?url=http://XXXXXXX:8088/logs/yarn-yarn-resourcemanager-XXXXXXX.log
![Page 55: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/55.jpg)
Apache Ambari Server Side Request Forgery
H
D
A
E
UCVE-2015-1775
![Page 56: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/56.jpg)
Apache Ambari attack scenario
Target old Hadoop installation with Ambari 1.5.0 to 2.0.2
Hijack standard account (or use Hue XSS to perform CSRF)
Log into Ambari, use CVE-2015-1775
Get access to local network (DMZ) –HTTP only
Download logs, exploit other Hadoopservers in DMZ
H
D
A
E
U
![Page 57: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/57.jpg)
Hadoop
Distrosspecifics
Admin ifaces
external
ifaces
User ifaces
Admin interfaces
![Page 58: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/58.jpg)
Apache Ranger overview
• Previously: Apache Argus, XA-Secure
• Provides central administration for policies, users/groups, analytics and audit data.
H
D
A
E
Uhttp://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Sys_Admin_Guides/content/ref-746ce51a-9bdc-4fef-85a6-69564089a8a6.1.html
![Page 59: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/59.jpg)
Apache Ranger overview
H
D
A
E
Uhttp://hortonworks.com/blog/best-practices-for-hive-authorization-using-apache-ranger-in-hdp-2-2/
![Page 60: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/60.jpg)
• Low hanging fruits: no HTTP hardening, SlowHTTP DoS
• Standard users can log into Ranger but have no permissions
• Interesting function level access control ->
Apache Ranger
![Page 61: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/61.jpg)
Apache Ranger vulnerabilities
H
D
A
E
U
![Page 62: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/62.jpg)
Missing function level access control
H
D
A
E
U
CVE-2015-0266
![Page 63: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/63.jpg)
Apache Ranger attack scenario
Target an old Hadoop installation (Apache Ranger 0.4 or XA-Secure v. 3.5.001 )
Hijack standard Hadoop account
Log into Ranger (with low permissions)
Use CVE-2015-0266 to escalate privileges
Edit accounts, authorization rules, accesspolicies
H
D
A
E
U
![Page 64: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/64.jpg)
Apache Ranger vulnerabilities
H
D
A
E
U
https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5+-+User+Guide
![Page 65: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/65.jpg)
• User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) <script>alert(1);</script>
Apache Ranger XSS through UserAgent
H
D
A
E
UCVE-2015-0265
![Page 66: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/66.jpg)
H
D
A
E
U
Apache Ranger attack scenario
Target an old Hadoop installation (Apache Ranger0.4 or XA-Secure v. 3.5.001 )
Network access to Apache Ranger is necessary(either from the internet or local network)
Log in with any user and password using XSS in UserAgent
You don’t need to escalate privileges, you’realready an admin (after admin opens sessiontab)
Deploy BEEF or whatsoever (CSRF script) to create users and change policies
![Page 67: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/67.jpg)
• Affected version: Apache Ranger v 0.4.0, XA Secure v. 3.5.001
• Both vulnerabilities patched in Ranger v 0.5.0
• For a while developers did a self-full-disclosure ->
Apache Ranger patched
![Page 68: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/68.jpg)
RANGER-284 in public Jira now
H
D
A
E
U
![Page 69: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/69.jpg)
RANGER-284 shortly after vendor contact
H
D
A
E
U
![Page 70: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/70.jpg)
DISTRIBUTIONS SPECIFICSnot in every environment
![Page 71: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/71.jpg)
Hadoop
Distrosspecifics
Admin ifaces
external
ifaces
User ifaces
Distribution specifics
![Page 72: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/72.jpg)
Distros
H
D
A
E
U
http://blog.cloudera.com/blog/2012/07/the-hadoop-ecosystem-visualized-in-datameer/
![Page 73: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/73.jpg)
cloudbased
hostedlocally
Basic distinction
![Page 74: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/74.jpg)
Distros
How long does it take to create a new distro version?
How many components are outdated at that time?
How long does it take to deploy a new distro at a company?
How many components are outdated at that time?
H
D
A
E
U
Most cases:• MAJOR – ca. 1 year• MINOR – ca. 3 months• PATCH – ca. 1-2 months (differs much)
![Page 75: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/75.jpg)
Hortonworks HDP components by version
http://hortonworks.com/hdp/whats-new/
![Page 76: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/76.jpg)
Distros
Old components with known issues
• Old OS components (java, php, ruby, etc.)
• Old OS components (e.g. old tomcat used by Oozie and HDFS)
• Old Hadoop components (e.g. old Hue, Ambari, Ranger)
Default passwords
Default configuration
H
D
A
E
U
![Page 77: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/77.jpg)
vuln found(e.g. Ambari)
Hadooppached
distro update deployment
Vulnerability timeline
ResponsibleDisclosure?
H
D
A
E
U
vuln found(e.g. jQuery)
jQuerypatched
Djangopatched
Hue updatedistro
updatedeployment
Responsibledisclosure?
Full disclosure?
Full disclosure?
![Page 78: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/78.jpg)
Distros
Old components with known issues
Default passwords
• SSH keys configured but default passwords still work
• Default mysql passwords, NO mysql passwords
Default configuration
H
D
A
E
U
![Page 79: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/79.jpg)
Distros
Old components with known issues
Default passwords
Default configuration
• No network level hardening
• No HTTP hardening (clickjacking, session mgmt, errors)
• Hue uses Django with DEBUG turned on by default
• „Hacking virtual appliances” by Jeremy Brown
H
D
A
E
U
![Page 80: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/80.jpg)
H
D
A
E
U
Default configurations sucks
X-Frame-Options:ALLOWALL
![Page 81: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/81.jpg)
EXTERNAL INTERFACESFor clients or whatsoever
![Page 82: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/82.jpg)
Hadoop
Distrosspecifics
Admin ifaces
external
ifaces
User ifaces
External interfaces
![Page 83: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/83.jpg)
External
• More than 25 internal Apache apps/modules• Vendor/distro specific apps/interfaces• Popular monitoring: Ganglia, Splunk• Auth providers: LDAP, Kerberos, OAuth• Many apps, many targets
H
D
A
E
U
![Page 84: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/84.jpg)
Hadoop
Hadoop
Distrosspecifics
Admin ifaces
external
ifaces
User ifaces
![Page 85: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/85.jpg)
SUMMARYways to protect your big data environment
![Page 86: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/86.jpg)
Ways to protect your Hadoop environment
Excessive network access
• Keep it super tight!
Excessive user pesmissions
Typical web vulnerabilities
Obsolete software
Distros dependent vulnerabilities
External system connections
![Page 87: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/87.jpg)
Ways to protect your Hadoop environment
Excessive network access
Excessive user permissions
• Map business roles to permissions
Typical web vulnerabilities
Obsolete software
Distros dependent vulnerabilities
External system connections
![Page 88: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/88.jpg)
Ways to protect your Hadoop environment
Excessive network access
Excessive user permissions
Typical web vulnerabilities
• Pentest it! Introduce application independent security countermeasures
Obsolete software
Distros dependent vulnerabilities
External system connections
![Page 89: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/89.jpg)
Ways to protect your Hadoop environment
Excessive network access
Excessive user permissions
Typical web vulnerabilities
Obsolete software
• Make a list of all components. Monitor bugtracks and CVEs.
Distros dependent vulnerabilities
External system connections
![Page 90: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/90.jpg)
Ways to protect your Hadoop environment
Excessive network access
Excessive user permissions
Typical web vulnerabilities
Obsolete software
Distros dependent vulnerabilities
• A pentest after integration is a must. Demand security from software suppliers.
External system connections
![Page 91: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/91.jpg)
Ways to protect your Hadoop environment
Excessive network access
Excessive user permissions
Typical web vulnerabilities
Obsolete software
Distros dependent vulnerabilities
External system connections
• Make a list of all external system connections. Do a threat modeling and pentest corresponding systems.
![Page 92: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/92.jpg)
Current status
![Page 93: Big problems with big data Hadoop interfaces · PDF fileBig problems with big data – Hadoop interfaces security Jakub Kaluzny](https://reader034.fdocuments.net/reader034/viewer/2022050916/5a78e2f67f8b9a21538ece2c/html5/thumbnails/93.jpg)
THANK YOU!
谢谢
Jakub Kaluzny