OSMOSIS - Hackito Ergo Sum2014.hackitoergosum.org/slides/day2_OSMOSIS_HES2014.pdf · Web based...
Transcript of OSMOSIS - Hackito Ergo Sum2014.hackitoergosum.org/slides/day2_OSMOSIS_HES2014.pdf · Web based...
OSMOSIS Open Source Monitoring Security Issues HACKITO ERGO SUM 2014 / April 2014 / Paris
AGENDA
§ Who are we?
§ Open Source Monitoring Software
§ Results
§ Demonstration
§ Responses
§ Mitigations and conclusion
4/25/14 2 – Public – Deutsche Telekom AG / OSMOSIS
DEUTSCHE TELEKOM PROFILE
4/25/14 3 – Public – Deutsche Telekom AG / OSMOSIS
COSTUMERS & MARKETS FACTS & FIGURES
Telekom in figures § Revenue € 58.7 bn
§ Adjusted Ebitda € 18.7 bn
§ Free cash flow € 6.4 bn
§ Among the top100 companies worldwide (#75 in 2012 Fortune500 list)
Employees & responsibility § Employees worldwide:
235, 000
§ 9 ,000 trainees und cooperative degree students in Germany
§ Pioneer of social issues (pomotion of woman, data privacy, climate protection etc.)
Customers
§ >141 m mobile customers
§ >32 m fixed-line customers/ >17 m broadband customers
§ rd. 3 m (IP) TV customers
§ About 2 m workstation systems marketed
Markets § Presence in 50 countries
§ Deutschland, Europa, USA: using our own infrastructur
§ T-Systems: globale presence & alliances via partners
Source: DT annual report to shareholders 2012/TMUS annual report to shareholders 2012
DEUTSCHE TELEKOM GROUP INFORMATION SECURITY
4/25/14 4 – Public – Deutsche Telekom AG / OSMOSIS
Intelligente Netzlösungen
§ Security requirements
§ Privacy & Security Assessment (PSA)
§ Deutsche Telekom Cyber Emergency Response Team (CERT)
§ Implementation of measures
§ Technology
§ Testing
§ Abuse-Handling
Security levels
Security strategies
Standards
Incident management
Consulting
Innovation
Security requirements
OPEN SOURCE MONITORING SOFTWARE OVERVIEW
SUMMARY § Critical function in a corporate network § Lets you know how well the network is running § End-to-end monitoring for services up to detailed hardware view
JOINT FUNCTIONS IN THIS CASE § Web based solution § Agent based
OUT OF SCOPE § No IDS / IPS § No commercial solutions § No security monitoring
4/25/14 5 – Public – Deutsche Telekom AG / OSMOSIS
OPEN SOURCE MONITORING SOFTWARE THREATS
§ Ubiquitous component in network environments
§ Centralized access to multiple networks
§ Usually position deep in the internal network (as in: semi-trusted network) § Used in nearly each environment (from small business, over mid range up to enterprises)
§ MTAACA (machine that acts as client attack) and CTAMTAACA (clients that access machines that act as clients attack)
4/25/14 6 – Public – Deutsche Telekom AG / OSMOSIS
OPEN SOURCE MONITORING SOFTWARE RISKS
§ A more valuable target than perimetric systems
§ Input data parsing (logfiles, SNMP, traps, ...)
§ Web GUIs (OWASP Top 10 anyone?)
§ Some have home-brew agents – on EVERY system
§ Potential access to a lot of components in the perimeter and internal network
4/25/14 7 – Public – Deutsche Telekom AG / OSMOSIS
OPEN SOURCE MONITORING SOFTWARE HOW IS IT IMPLEMENTED TYPICALLY?
4/25/14 8 – Public – Deutsche Telekom AG / OSMOSIS
SNMP
OWN CHECKS
OPEN SOURCE MONITORING SOFTWARE WHAT WE COVERED
§ This is not an academic talk - we are talking about actual experience
§ Open Source tools are easy to audit (kinda)
§ Everyone has the chance to audit their own solution
§ Focus on market leading / industry standard software
4/25/14 9 – Public – Deutsche Telekom AG / OSMOSIS
OPEN SOURCE MONITORING SOFTWARE WHAT WE DID NOT COVER
§ No commercial / closed source solutions
§ Architectural software flaws
§ Critical “features” which should be disabled anyways
e.g. nrpe.cfg dont_blame_nrpe
§ No additional plugins, features , add-ons
§ Not the (home brewed) agents itself
4/25/14 10 – Public – Deutsche Telekom AG / OSMOSIS
OPEN SOURCE MONITORING SOFTWARE TOOLS WE COVERED
§ CACTI “… network graphing solution …”; “… frontend is completely PHP driven …” src: http://www.cacti.net
§ NAGIOS “Nagios Is The Industry Standard In IT Infrastructure Monitoring” src: http://www.nagios.org/
§ CHECK_MK (NAGIOS ADD-ON) “Check_MK is a comprehensive add-on for the famous Open Source monitoring software Nagios …” src: https://mathias-kettner.com/check_mk_introduction.html
§ ICINGA “Icinga is an enterprise grade open source monitoring system …” src: https://www.icinga.org/
4/25/14 11 – Public – Deutsche Telekom AG / OSMOSIS
OPEN SOURCE MONITORING SOFTWARE PUBLICLY KNOWN INCIDENTS
4/25/14 12 – Public – Deutsche Telekom AG / OSMOSIS
CVE2012-096 – Remote Buffer Overflow Nagios Hetzner (06/2013)
OPEN SOURCE MONITORING SOFTWARE OTHER INTERESTING INFORMATION
4/25/14 13 – Public – Deutsche Telekom AG / OSMOSIS
Public Buffer Overflow in CACTI (since 10/2013) NRPE - Remote command exec (04/2014)
RESULTS OVERALL
§ Critical issues were found in ALL audited solutions … § Memory corruption – Buffer/Heap Overflows § Off-by-one’s § CSRF § XSS § eval-processing untrusted input § Remote Code Execution § Arbitrary file access
§ Many web based bugs, as all the solutions use web GUIs
4/25/14 14 – Public – Deutsche Telekom AG / OSMOSIS
(Cacti)
Version 3.5.0b 1.9.1b 1.2.2p2 0.8.8a
Number of Findings 1 2 7 3
CVSS 2 Score (highest finding) 4.9 AV:N/AC:M/Au::S/C:P/I:N/A:P
8.5 AV:N/AC:M/Au:S/C:C/I:C/A:C
8.5 AV:N/AC:M/Au:S/C:C/I:C/A:C
8.5 AV:N/AC:M/Au:S/C:C/I:C/A:C
Criticality medium high high high
Number of open findings 1* 0 1** 3
Announcement to vendor / developer 5th Dec. 2013 2nd Dec. 2013 8th Oct. 2013 15th Oct. 2013
Bug Fix Release 3.5.x*, 4.0.3 1.10.2, 1.9.4, 1.8.5 or latest
release
1.2.4p1, 1.2.5i2 or
latest release n/a
Public DTAG CERT Advisory DTC-A-20140324-004
DTC-A-20140324-003
DTC-A-20140324-002
DTC-A-20140324-001
Remarks
* Bug fixes in the source code only available. No updates release available.
** exec of python code within WATO
RESULTS DETAILED VIEW
03.04.2014 15 – Confidential – Christian Sielaff / OSMOSIS
DEMONSTRATION
CAN WE GET A SHELL?
4/25/14 16 – Public – Deutsche Telekom AG / OSMOSIS
DEMONSTRATION NETWORK OVERVIEW
03.04.2014 17 – Confidential – Christian Sielaff / OSMOSIS
Hacker Terminal Server Cacti / Check_MK Administrator
DEMONSTRATION CACTI
03.04.2014 18 – Confidential – Christian Sielaff / OSMOSIS
Hacker Cacti Administrator
Bugs:
§ cross site request forgery
§ command like exec
DEMONSTRATION CACTI
03.04.2014 19 – Confidential – Christian Sielaff / OSMOSIS
Hacker Cacti Administrator
Bugs:
§ cross site request forgery
§ command like exec Get executed on Cacti server if: § Administrator clicks on a link or § Visit a malicious web site
DEMONSTRATION CACTI
03.04.2014 20 – Confidential – Christian Sielaff / OSMOSIS
Hacker Cacti Administrator
Bugs:
§ cross site request forgery
§ command like exec Get executed on Cacti server if: § Administrator clicks on a link, or § Visit a malicious web site
Pro:
§ Get a shell Con: § Need to know the Cacti URL § Admins needs to access link or site with link to
trigger exploit § Outgoing connections my be restricted § Admins needs to logged in
DEMONSTRATION CACTI
03.04.2014 21 – Confidential – Christian Sielaff / OSMOSIS
Hacker Cacti Administrator
Bugs:
§ cross site request forgery
§ command like exec Get executed on Cacti server if: § Administrator clicks on a link, or § Visit a malicious web site
Pro:
§ Get a shell Con: § Need to know the Cacti URL § Admins needs to access link or site with link to
trigger exploit § Outgoing connections my be restricted § Admins needs to logged in … not really
let’s brute force the Admin account J
DEMONSTRATION CHECK_MK
03.04.2014 22 – Confidential – Christian Sielaff / OSMOSIS
Hacker Terminal Server Check_MK Administrator
DEMONSTRATION CHECK_MK
03.04.2014 23 – Confidential – Christian Sielaff / OSMOSIS
Hacker Terminal Server Check_MK Administrator
Bugs:
§ cross site request forgery
§ command like exec
§ cross site scripting
DEMONSTRATION CHECK_MK
03.04.2014 24 – Confidential – Christian Sielaff / OSMOSIS
Hacker Terminal Server Check_MK Administrator
Bugs:
§ cross site request forgery
§ command like exec
§ cross site scripting What is the problem: § Exploits a feature in WATO § Uploads and exec a snapshot § Snapshot contains plain python code
DEMONSTRATION CHECK_MK
03.04.2014 25 – Confidential – Christian Sielaff / OSMOSIS
Hacker Terminal Server Check_MK Administrator
Bugs:
§ cross site request forgery
§ command like exec
§ cross site scripting What is the problem: § Exploits a feature in WATO § Uploads and exec a snapshot § Snapshot contains plain python code
Pro:
§ Get a shell Con: § Need to know the Check_MK URL § Admins needs to access link or site with link to
trigger exploit § Outgoing connections my be restricted § Admins needs to logged in
DEMONSTRATION CHECK_MK
03.04.2014 26 – Confidential – Christian Sielaff / OSMOSIS
Hacker Check_MK Administrator
Bugs:
§ cross site request forgery
§ command like exec
§ cross site scripting What can we do better? § Use the agent on a system § Re-use existing connections
Terminal Server
DEMONSTRATION CHECK_MK
03.04.2014 27 – Confidential – Christian Sielaff / OSMOSIS
Hacker Check_MK Administrator
Bugs:
§ cross site request forgery
§ command like exec
§ cross site scripting What can we do better? § Use the agent on a system § Re-use existing connections
Terminal Server
DEMONSTRATION CHECK_MK
03.04.2014 28 – Confidential – Christian Sielaff / OSMOSIS
Hacker Check_MK Administrator
Bugs:
§ cross site request forgery
§ command like exec
§ cross site scripting What can we do better? § Use the agent on a system § Re-use existing connections
Terminal Server
Pro:
§ Get a shell
§ URL is no longer needed § Administrator not need a link to click § Triggers when the Administrator logs in § Using existing connections
Con: § Need (privileged) access to a monitored system
DEMONSTRATION CHECK_MK
03.04.2014 29 – Confidential – Christian Sielaff / OSMOSIS
Hacker Check_MK Administrator
Bugs:
§ cross site request forgery
§ command like exec
§ cross site scripting What can we do also? § Just a simple SSH login?
Terminal Server
DEMONSTRATION CHECK_MK
03.04.2014 30 – Confidential – Christian Sielaff / OSMOSIS
Hacker Check_MK Administrator
Bugs:
§ cross site request forgery
§ command like exec
§ cross site scripting What can we do also? § Just a simple SSH login? § A XSS triggers a CSRF triggers
an …
Terminal Server
DEMONSTRATION CHECK_MK
03.04.2014 31 – Confidential – Christian Sielaff / OSMOSIS
Hacker Check_MK Administrator
Bugs:
§ cross site request forgery
§ command like exec
§ cross site scripting What can we do also? § Just a simple SSH login? § A XSS triggers a CSRF triggers
an upload that triggers a shell J
Terminal Server
DEMONSTRATION CHECK_MK
03.04.2014 32 – Confidential – Christian Sielaff / OSMOSIS
Hacker Check_MK Administrator
Bugs:
§ cross site request forgery
§ command like exec
§ cross site scripting What can we do also? § Just a simple SSH login? § A XSS triggers a CSRF triggers
an upload that triggers a shell J
Terminal Server
Pro:
§ Get a shell
§ URL is no longer needed § Administrator not need a link to click § Triggers when the Administrator logs in
Con: § Logwatch feature (default installation is fair) § Outgoing connections my be restricted
DEMONSTRATION
CAN WE GET A SHELL?
… YES J
4/25/14 33 – Public – Deutsche Telekom AG / OSMOSIS
RESPONSES CONTACT AND TIMELINES
CONTACTING § some developer without a contact option (expect a public mailing list – is this a good idea in such case?) § usually an Email contact is possible – also with a privacy option § Only Icinga provides an option for a private information sharing
http://www.icinga.org/faq/how-to-report-a-bug/#securityissue
TIMELINE § approximately six days from first response to a bug fix release – well done! § up to 85 days to a bug fix release § up to nothing until now L
ADVISORIES § post flaws to Bugtraq on 24th of March § get first responses regarding open findings 28th / 31st of March
4/25/14 34 – Public – Deutsche Telekom AG / OSMOSIS
RESPONSES FEEDBACK
§ „WHAT IS OWASP?"
It’s 2014, guys!
§ „THIS IS A FEATURE"
Yes, and a backdoor!
§ „WHAT TOOLS DID YOU USE FOR SCANNING?“
Hint: None, we had the source code – Duh!
§ „WHY SHOULD WE FIX WHAT YOU SEE AS A SECURITY PROBLEM? WE NEVER ASKED FOR THIS AUDIT!“
Approximately Right. Remember it’s open source? Open as in: I audit this code as much as I want to?
§ „-„
As in: No response at all after issues were committed to developer.
4/25/14 35 – Public – Deutsche Telekom AG / OSMOSIS
RESPONSES DISCLOSURE
SECURITY FIXES § Change log or Release notes _never_ mentions security fixes explicitly § No hints or information on the developer Web sites! § CVE _Common_ – never heard about that
CREDITS § What’s that?
BUT THERE ARE SOME PROFESSIONALS § The Icinga Team has published bug fix releases (incl. back ports), ordered CVE numbers and assign the issues as
Security issues. MANY THANKS AND WELL DONE!
4/25/14 36 – Public – Deutsche Telekom AG / OSMOSIS
MITIGATIONS BEST PRACTICES
BEST PRACTICES § Consider Icinga and Nagios Security Guidelines
e.g. http://docs.icinga.org/latest/en/security.html § Nothing similar available for Cacti and Check_MK
GENERAL BASICS § Patching and regular updates § OS and middleware hardening § Minimal rights on application level, but also on operating system level § Remove critical features (e.g. WATO in Check_MK) § Passwords
4/25/14 37 – Public – Deutsche Telekom AG / OSMOSIS
MITIGATIONS SEGREGATION
ON NETWORK LEVEL § Do not place such systems flat in your corporate network § Consider segregation based on functions, e.g. own monitoring systems for dedicated services § No internet for the admin workstations and monitoring system (incl. ICMP, DNS, NTP, … )
ON APPLICATION LEVEL § Segregate user and roles
4/25/14 38 – Public – Deutsche Telekom AG / OSMOSIS
MITIGATIONS ARCHITECTURE
AGENT BASED MONITORING § Needs privileged rights to get all information and listen to the network (often unauthenticated) § Security of agents should be discussed separately
e.g. http://www.securityfocus.com/archive/1/531063/30/0/threaded
CHECK VIA SSH § Must be secured carefully via SSHd configuration – otherwise direct shell login
SOLUTION § Change the communication direction § Based on Check_MK’s agent, it’s just a configuration – no additional software needed
4/25/14 39 – Public – Deutsche Telekom AG / OSMOSIS
MITIGATIONS ARCHITECTURE
HOW IT WORKS § Run Check_MK agent locally and pipe output to a file
4/25/14 40 – Public – Deutsche Telekom AG / OSMOSIS
MITIGATIONS ARCHITECTURE
HOW IT WORKS § Run Check_MK agent locally and pipe output to a file § Secure transfer, e.g. via SCP/SFTP
4/25/14 41 – Public – Deutsche Telekom AG / OSMOSIS
MITIGATIONS ARCHITECTURE
HOW IT WORKS § Run Check_MK agent locally and pipe output to a file § Secure transfer, e.g. via SCP/SFTP § Configure Check_MK Configuration & Check Engine to get information from a local file
4/25/14 42 – Public – Deutsche Telekom AG / OSMOSIS
MITIGATIONS ARCHITECTURE
4/25/14 43 – Public – Deutsche Telekom AG / OSMOSIS
OWN CHECKS
CONCLUSION
§ Take care about your used solutions incl. additional features, add-ons, plug ins, self written checks and architecture.
§ When it named Open Source, it does not mean it is secure itself!
§ In general Open Source Monitoring solutions are not more or less secure than commercial ones.
§ Strong isolation of administrator workstations and your monitoring system as well.
§ @Developer: Check OWASP regularly!
4/25/14 44 – Public – Deutsche Telekom AG / OSMOSIS