Using Splunk$$ toProtect PaentPrivacy$$ … Splunk*searched*on*User*ID*/*tabletname*atdrop*?mes.*...
53
Copyright © 2014 Splunk Inc. Ant Lefebvre [email protected] Senior Systems Engineer Middlesex Hospital Using Splunk to Protect Pa=ent Privacy and Achieve Meaningful Use
Transcript of Using Splunk$$ toProtect PaentPrivacy$$ … Splunk*searched*on*User*ID*/*tabletname*atdrop*?mes.*...
Copyright © 2014 Splunk Inc.
Ant Lefebvre [email protected] Senior Systems Engineer Middlesex Hospital
Using Splunk to Protect
Pa=ent Privacy and Achieve
Meaningful Use
About Middlesex Hospital • Complete range of medical services • Some of Connec?cut’s highest quality and pa?ent sa?sfac?on ra?ngs – 30 Networked Offsite Loca?ons – 10 Primary Care Offices – 3 Emergency Departments
• Recipient of the CIO 100 award for our use of Splunk soNware
• 100 Top Hospitals list for two years running
• HealthCare’s Most Wired List 2012-‐2014
whoami? • Systems Engineer • Network Engineer • Security / Compliance • Wireless/Wired • IT Director • IT Consultant • Splunk .conf 2013 Revolu?on Award Winner!
Hospital Network Opera=ons
Challenges in Healthcare
Event Log Correla?on
Virtualiza?on Management
Global View of Environment
Applica?on Performance
Hospital’s Visibility Gap • Not easy to navigate Windows event viewer • Log by log review for troubleshoo?ng • Manual event correla?on spanning mul?ple systems • No log access when host down or off network • Hours/days to find root cause(s) for end user device issues
Wasted (me and effort to track down issues
Splunk Solves Visibility Gap Steps to success: 1. Downloaded free demo 2. Globally installed Splunk Universal Forwarders on Windows
server and client opera?ng systems 3. Indexed Windows event log data 4. Instantly gained visibility into Windows environment like never
before
Troubleshoo(ng (me now a frac(on of what it used to be
Splunk in Produc=on Finding new use cases EVERY DAY!!
• Audit consolida?on – One tool to monitor all systems • Event correla?on – Is the issue happening everywhere? When? • Recognize anomalous ac?vi?es – Something strange going on? • Add new log sources – See what shakes out…
No need to purchase addi(onal products. Index the data in Splunk.
Success Stories Mystery “wireless disconnects” persisted for years.
Using Splunk searched on User ID / tablet name at drop ?mes. Discovered crashing process on Citrix server at dropping event ?me!
“Wireless disconnects” reported HERE
Root cause was back end service crashing
in datacenter
What computer am I connected to? Mystery name resolu?on issues.
Connec?ng to wrong worksta?ons when using hostname.
*error* search found DNS record scavenging was accidentally off aNer AD/DNS server migra?on.
Want to connect to PC A
But get connected to PC B
Helpdesk
Finding a Botnet
• Index firewall traffic logs using Splunk and Google Maps
• Discovered a health library machine connected to an interna?onal botnet
• No business need to communicate with Peru
Boot Times Table
Found File Dele=on Incident • User files “vanish with
no insight from file audit tool
• Search for user id AND delete finds over 300 events in an hour over the weekend
• User accidentally deleted one too many folders
Blocking streaming HDTV through Firewall
Program Intelligence into Apps/Dashboards • Created useful dashboards for opera?ons/helpdesk team • No need to know Splunk search commands to use • Help less knowledgeable staff troubleshoot environment issues • Each new dashboard is created in-‐house. No need for addi?onal purchase. No need to ask for product enhancement or feature from vendors.
• Single point of reference for mul?ple uses
The Splunk Admin can create point and click knowledge
Citrix User Login Finder
Find Server Behind Load Balancer
Where has this user logged in?
Most Numerous Cisco Syslog Messages
Web Traffic!
Power Dashboard
Windows NPS RADIUS Dashboard
Print Server Log Dashboard
Print User to IP Correla=on
Print logs do not contain where user prints from. Windows Event logs show where user last logged in.
Viral Spread of Splunk
Word of Splunk’s capability to audit systems and solve mysteries trickled through other IT staffers. Addi?onal systems I didn’t even know we had were added to Splunk.
IT Director’s Challenge
• A system to audit our Electronic Health Record access. • A single solu?on to audit mul?ple systems. • Easy to manage. • Cost is always a factor. • We have two op?ons. Which one is beher?
• The answer: Op?on 3 – Splunk!
Pa=ent Privacy &
Meaningful Use
Healthcare Jargon • EMR/EHR -‐ Electronic Pa?ent Records • PHI/ePHI – (electronic) Protected Health Informa?on • HIPAA -‐ The Health Insurance Portability and Accountability Act of 1996 • HITECH Act -‐ Health Informa?on Technology for Economic and Clinical Health Act • Meaningful Use – Goal is to not just adopt an EHR, but to leverage it to achieve significant
improvements in care • Cerner -‐ Middlesex Hospital’s Primary EHR • Results – Middlesex Hospital’s home grown EHR lookup applica?on • eClinicalWorks – Middlesex Hospital’s Primary Care / Family Prac?ce / Mul?specialty EHR • McKesson Homecare – Middlesex Hospital Home care EHR
• GE Flowcast – Pa?ent registra?on/demographics • Lawson – Employee Database
Electronic Health Record Audi=ng • Federal reimbursement for having cer?fied technologies to audit Electronic Health Record (EHR) access, Meaningful Use Requirement
• Splunk v6.0 is currently v1.0 Cer?fied (for both Ambulatory and Inpa?ent) §170.314(d)(3) -‐ Audit report(s)
• EHR provider offers specialized (and expensive) point solu?on • Other EHR vendors couldn’t correlate between systems/databases • Other vendor solu?ons specific to their product. Can’t build intelligence.
Splunk for MU2 EHR Module, 2014 Edi=on means EHR technology that is cer?fied to at least one of the 2014 Edi?on EHR cer?fica?on criteria for either the ambulatory or inpa?ent prac?ce sepng. An EHR Module could provide a single capability required by one cer?fica?on criterion or it could provide all capabili?es but one, required by the cer?fica?on criteria for a Complete EHR. Splunk is 1 of 20 modules required to meet Base EHR defini?on for 2014 Edi?on EHR cer?fica?on. 170.314(d)(3) Audit reports Required 170.314(g)(4) Quality Management System Needed for all modules Splunk will not fulfill your EHR product cer?fica?on alone, but will check the (d)(3) Audit Report(s) box on the cer?fied health IT product list: hhp://oncchpl.force.com/ehrcert/ehrproductsearch or hhp://goo.gl/5PsHd
Primary vendor solu=on
EHR
Vendor Audit Repository
Data Inputs
• Similar in ability to Splunk
• Much more expensive to implement
• Very lihle if any community support
• New inputs require vendor services to implement
• Data elements have to be pre-‐programed into repository
Other vendor solu=ons • Each system has its own
audi?ng capabili?es (maybe)
• No way to centrally look into all system access.
• Log into each app to run access reports
• Advance inves?ga?ve dashboards unavailable, limited, or costly to implement
Taking a stab at an EHR audit App
• Newbie Splunk user’s first App • Cerner audit data only • PoC rolled into preliminary App • Much development needed • Worked well enough to sa?sfy
audi?ng requirements
Challenges in building the App • First of it’s kind in Splunk
• I am not a compliance officer
• I am not a developer • Limits on my ?me • Only IT staffer with end game in focus
Raw EHR formats?
Splunk to indexes ALL!
XML with checksum to prevent tampering
mySQL
SQL
Comma Separated Value
Splunk
Human Readable Columns Key Value Pairs
Under the Hood
Cerner Listener / Splunk Universal
Forwarder
Cerner Audit Outbound Server
Splunk Indexer
Inges?ng Cerner EHR (XML format) audit data into Splunk By far the most comprehensive audi?ng
Real-‐(me Audit Events
Under the Hood Part 2 Inges?ng CSV exports into Splunk
FTP server / Splunk
Universal Forwarder
Results Flowcast Lawson
Yesterday’s Audit Events
Splunk Indexer
Under the Hood Part 3 Inges?ng database EHR audit data into Splunk
DB Connect/ Splunk Heavy Forwarder
Engage your EHR vendor EARLY!
ECW – mySQL McKesson Homecare – SQL
Near Real-‐(me Audit Events
Splunk Indexer
Healthcare App fields? EHR A: 35 fields
Pa?ent Registra?on App
Employee Database
EHR C: 5 fields
EHR B: 15 fields Homegrown EHR
Splunk
Healthcare common informa(on model?
HIPAA Privacy and Security Scout™ Healthcare Compliance Splunk App
HIPAA Privacy and Security Scout™ and HIPAA Scout™ are protected by U.S. and interna?onal copyright and intellectual property laws.
Middlesex is able to ensure that staff is compliant with State and Federal privacy regula?ons. The hospital has the ability to monitor user level access to several EHR
systems from single interface using Splunk Healthcare CIM. App is available from Splunk Partner Conducive Consul?ng -‐ hhp://www.conducivesi.com
What HIPAA Scout Provides • Get right to the facts • Compliance isn’t prehy • Auditors are going to love it! • Meaningful Use of EHR logs • HIPAA viola?on inves?ga?on made easy
• Common Informa?on Model • Universal EHR Audi?ng App
HIPAA Privacy and Security Scout™ • Auditor Home Page
– Quick links to most used reports
• Applica=on Report Categories
• New reports are only limited by the logs and the imagina=on – Every hospital is different. Requirements and problems vary.
– Ac?vity Audit – Admin Audit – Disclosure Report – Employee Info – Login Report
– Inves?ga?ons – Suspicious Ac?vity – User Account Sharing – VIP Pa?ent Access
HIPAA Privacy and Security Scout™
Most Useful Dashboards • Record Access Inves?ga?on • Coworker Record Access • Same Last Name • Wrong Unit • Employee Admission Report • Same Street
Example Fields Available for Inves=ga=ons • User Name • User ID • Pa?ent Name • Medical Record Number • Account Number • Hospital Unit Number
Example Dash: Same Last Name
44
Example Dash: Wrong Unit
45
Example Dash: Record Access Inves=ga=on
46
Example Dash: Coworker Record Access
47
Example Dash: Record Print by Pa=ent
48
Splunk & Compliance
49
" Re-‐draN our policies on regarding what a HIPAA viola?on actually is. " Create policies regarding how we will move forward with Splunk & HIPAA
Privacy and Security Scout app. " Will we survive an audit? We have the power. Use it! " Educate the masses. Goal is for Splunk to find nothing.
Barriers to Progress
50
" Beher at finding poten?al viola?ons. Takes more ?me to inves?gate. Splunk is too good!
" EHR vendors don’t supply enough audit info to automate more.
" Finding the informa?on with DB Connect takes lots of ?me. Hope the schema doesn’t change!
" Vendors unable/unwilling to co-‐operate.
Vision into Our Future NOW Compliance Officers, Auditors, Applica?on Staff, Opera?ons Team, Infrastructure Team
Splunk indexing mul?ple diverse, but related systems
Splunk search heads with TAs (Technology Add-‐ons) and a Common Healthcare App
EHR, Finance, Infrastructure, Clients, Servers, Systems,
the list goes on….
Lessons Learned
• Budget for servers/storage. • Don’t roll PoC into produc?on system. Start fresh. • Sync ?mes before indexing (where is that s?nking real ?me data?). • Expect to frequent answers.splunk.com if you want to be successful. • When inges?ng data, it helps to have friends on the inside.
“If I had known then what I know now…”
