Copyright  ©  2014  Splunk  Inc.  

Ant  Lefebvre  ant@midhosp.org  Senior  Systems  Engineer  Middlesex  Hospital  

Using  Splunk    to  Protect    

Pa=ent  Privacy    and  Achieve  

Meaningful  Use    

About  Middlesex  Hospital  •  Complete  range  of  medical  services  •  Some  of  Connec?cut’s  highest  quality  and  pa?ent  sa?sfac?on  ra?ngs  –  30  Networked  Offsite  Loca?ons  –  10  Primary  Care  Offices  –  3  Emergency  Departments  

•  Recipient  of  the  CIO  100  award  for  our  use  of  Splunk  soNware  

•  100  Top  Hospitals  list  for  two  years  running  

•  HealthCare’s  Most  Wired  List  2012-­‐2014  

whoami?  •  Systems  Engineer  •  Network  Engineer  •  Security  /  Compliance  •  Wireless/Wired  •  IT  Director  •  IT  Consultant  •  Splunk  .conf  2013  Revolu?on          Award  Winner!  

Hospital  Network  Opera=ons  

Challenges  in  Healthcare  

   

Event  Log  Correla?on  

Virtualiza?on  Management  

Global  View    of  Environment    

Applica?on  Performance  

Hospital’s  Visibility  Gap  •  Not  easy  to  navigate  Windows  event  viewer    •  Log  by  log  review  for  troubleshoo?ng  •  Manual  event  correla?on  spanning  mul?ple  systems    •  No  log  access  when  host  down  or  off  network    •  Hours/days  to  find  root  cause(s)  for  end  user  device  issues  

Wasted  (me  and  effort  to  track  down  issues  

Splunk  Solves  Visibility  Gap  Steps  to  success:  1.  Downloaded  free  demo  2.  Globally  installed  Splunk  Universal  Forwarders  on  Windows  

server  and  client  opera?ng  systems  3.  Indexed  Windows  event  log  data  4.  Instantly  gained  visibility  into  Windows  environment  like  never  

before  

Troubleshoo(ng  (me  now  a  frac(on  of  what  it  used  to  be  

Splunk  in  Produc=on  Finding  new  use  cases  EVERY  DAY!!  

•  Audit  consolida?on  –  One  tool  to  monitor  all  systems  •  Event  correla?on  –  Is  the  issue  happening  everywhere?  When?  •  Recognize  anomalous  ac?vi?es  –  Something  strange  going  on?  •  Add  new  log  sources  –  See  what  shakes  out…    

No  need  to  purchase  addi(onal  products.  Index  the  data  in  Splunk.  

Success  Stories  Mystery  “wireless  disconnects”  persisted  for  years.    

Using  Splunk  searched  on  User  ID  /  tablet  name  at  drop  ?mes.    Discovered  crashing  process  on  Citrix  server  at  dropping  event  ?me!  

 

“Wireless  disconnects”    reported  HERE  

Root  cause  was  back  end  service  crashing  

in  datacenter    

What  computer  am  I  connected  to?  Mystery  name  resolu?on  issues.    

Connec?ng  to  wrong  worksta?ons  when  using  hostname.    

*error*  search  found  DNS  record  scavenging  was  accidentally  off  aNer  AD/DNS  server  migra?on.  

  Want  to  connect  to  PC  A  

But  get  connected  to  PC  B  

Helpdesk  

Finding  a  Botnet  

•  Index  firewall  traffic  logs  using  Splunk  and  Google  Maps  

•  Discovered  a  health  library  machine  connected  to  an  interna?onal  botnet    

•  No  business  need  to  communicate  with  Peru  

Boot  Times  Table  

Found  File  Dele=on  Incident  •  User  files  “vanish  with  

no  insight  from  file  audit  tool  

•  Search  for  user  id  AND  delete  finds  over  300  events  in  an  hour  over  the  weekend  

•  User  accidentally  deleted  one  too  many  folders  

Blocking  streaming  HDTV  through  Firewall  

Program  Intelligence  into  Apps/Dashboards  •  Created  useful  dashboards  for  opera?ons/helpdesk  team  •  No  need  to  know  Splunk  search  commands  to  use  •  Help  less  knowledgeable  staff  troubleshoot  environment  issues  •  Each  new  dashboard  is  created  in-­‐house.  No  need  for  addi?onal  purchase.  No  need  to  ask  for  product  enhancement  or  feature  from  vendors.  

•  Single  point  of  reference  for  mul?ple  uses  

The  Splunk  Admin  can  create  point  and  click  knowledge  

Citrix  User  Login  Finder  

Find  Server  Behind  Load  Balancer  

Where  has  this  user  logged  in?  

Most  Numerous  Cisco  Syslog  Messages    

Web  Traffic!  

Power  Dashboard  

Windows  NPS  RADIUS  Dashboard  

Print  Server  Log  Dashboard  

Print  User  to  IP  Correla=on  

Print  logs  do  not  contain  where  user  prints  from.    Windows  Event  logs  show  where  user  last  logged  in.  

Viral  Spread  of  Splunk  

Word  of  Splunk’s  capability  to  audit  systems  and  solve  mysteries  trickled  through  other  IT  staffers.      Addi?onal  systems  I  didn’t  even  know  we  had  were  added  to  Splunk.  

IT  Director’s  Challenge  

•  A  system  to  audit  our  Electronic  Health  Record  access.  •  A  single  solu?on  to  audit  mul?ple  systems.  •  Easy  to  manage.    •  Cost  is  always  a  factor.  •  We  have  two  op?ons.  Which  one  is  beher?  

•  The  answer:  Op?on  3  –  Splunk!  

Pa=ent  Privacy  &  

Meaningful  Use  

Healthcare  Jargon  •  EMR/EHR  -­‐  Electronic  Pa?ent  Records  •  PHI/ePHI  –  (electronic)  Protected  Health  Informa?on    •  HIPAA  -­‐  The  Health  Insurance  Portability  and  Accountability  Act  of  1996  •  HITECH  Act  -­‐  Health  Informa?on  Technology  for  Economic  and  Clinical  Health  Act  •  Meaningful  Use  –  Goal  is  to  not  just  adopt  an  EHR,  but  to  leverage  it  to  achieve  significant  

improvements  in  care  •  Cerner  -­‐  Middlesex  Hospital’s  Primary  EHR    •  Results  –  Middlesex  Hospital’s  home  grown  EHR  lookup  applica?on  •  eClinicalWorks  –  Middlesex  Hospital’s  Primary  Care  /  Family  Prac?ce  /  Mul?specialty  EHR  •  McKesson  Homecare  –  Middlesex  Hospital  Home  care  EHR  

•  GE  Flowcast  –  Pa?ent  registra?on/demographics  •  Lawson  –  Employee  Database  

Electronic  Health  Record  Audi=ng  •  Federal  reimbursement  for  having  cer?fied  technologies  to  audit  Electronic  Health  Record  (EHR)  access,  Meaningful  Use  Requirement  

•  Splunk  v6.0  is  currently  v1.0  Cer?fied  (for  both  Ambulatory  and  Inpa?ent)  §170.314(d)(3)  -­‐  Audit  report(s)  

•  EHR  provider  offers  specialized  (and  expensive)  point  solu?on  •  Other  EHR  vendors  couldn’t  correlate  between  systems/databases  •  Other  vendor  solu?ons  specific  to  their  product.  Can’t  build  intelligence.  

Splunk  for  MU2  EHR  Module,  2014  Edi=on  means  EHR  technology  that  is  cer?fied  to  at  least  one  of  the  2014  Edi?on  EHR  cer?fica?on  criteria  for  either  the  ambulatory  or  inpa?ent  prac?ce  sepng.  An  EHR  Module  could  provide  a  single  capability  required  by  one  cer?fica?on  criterion  or  it  could  provide  all  capabili?es  but  one,  required  by  the  cer?fica?on  criteria  for  a  Complete  EHR.    Splunk  is  1  of  20  modules  required  to  meet  Base  EHR  defini?on  for  2014  Edi?on  EHR  cer?fica?on.    170.314(d)(3)  Audit  reports  Required    170.314(g)(4)  Quality  Management  System    Needed  for  all  modules    Splunk  will  not  fulfill  your  EHR  product  cer?fica?on  alone,  but  will  check  the  (d)(3)  Audit  Report(s)  box  on  the  cer?fied  health  IT  product  list:  hhp://oncchpl.force.com/ehrcert/ehrproductsearch  or  hhp://goo.gl/5PsHd  

Primary  vendor  solu=on  

EHR  

Vendor  Audit  Repository  

Data  Inputs  

•  Similar  in  ability  to  Splunk  

•  Much  more  expensive  to  implement  

•  Very  lihle  if  any  community  support  

•  New  inputs  require  vendor  services  to  implement  

•  Data  elements  have  to  be  pre-­‐programed  into  repository  

Other  vendor  solu=ons  •  Each  system  has  its  own  

audi?ng  capabili?es  (maybe)  

•  No  way  to  centrally  look  into  all  system  access.  

•  Log  into  each  app  to  run  access  reports  

•  Advance  inves?ga?ve  dashboards  unavailable,  limited,  or  costly  to  implement    

Taking  a  stab  at  an  EHR  audit  App  

•  Newbie  Splunk  user’s  first  App  •  Cerner  audit  data  only  •  PoC  rolled  into  preliminary  App  •  Much  development  needed  •  Worked  well  enough  to  sa?sfy  

audi?ng  requirements  

Challenges  in  building  the  App  •  First  of  it’s  kind  in  Splunk  

•  I  am  not  a  compliance  officer  

•  I  am  not  a  developer  •  Limits  on  my  ?me  •  Only  IT  staffer  with  end  game  in  focus  

 

Raw  EHR  formats?  

Splunk  to  indexes  ALL!  

XML  with  checksum  to    prevent  tampering  

mySQL  

SQL  

Comma  Separated  Value  

Splunk  

Human  Readable  Columns   Key  Value  Pairs  

Under  the  Hood  

Cerner  Listener  /  Splunk  Universal  

Forwarder  

Cerner  Audit  Outbound  Server  

Splunk  Indexer  

Inges?ng  Cerner  EHR  (XML  format)    audit  data  into  Splunk  By  far  the  most  comprehensive  audi?ng  

Real-­‐(me  Audit  Events  

Under  the  Hood  Part  2  Inges?ng  CSV  exports  into  Splunk  

FTP  server  /  Splunk  

Universal  Forwarder  

Results  Flowcast  Lawson  

Yesterday’s  Audit  Events  

Splunk  Indexer  

Under  the  Hood  Part  3  Inges?ng    database  EHR  audit  data  into  Splunk  

DB  Connect/  Splunk  Heavy  Forwarder  

Engage  your  EHR  vendor  EARLY!    

ECW  –  mySQL    McKesson  Homecare  –  SQL    

Near  Real-­‐(me  Audit  Events  

Splunk  Indexer  

Healthcare  App  fields?  EHR  A:  35  fields  

Pa?ent  Registra?on  App  

Employee  Database  

EHR  C:  5  fields  

EHR  B:  15  fields   Homegrown  EHR  

Splunk  

Healthcare  common  informa(on  model?  

HIPAA  Privacy  and  Security  Scout™    Healthcare  Compliance  Splunk  App  

HIPAA  Privacy  and  Security  Scout™  and    HIPAA  Scout™  are  protected  by  U.S.  and  interna?onal  copyright  and  intellectual  property  laws.  

Middlesex  is  able  to  ensure  that  staff  is  compliant  with  State  and  Federal  privacy  regula?ons.  The  hospital  has  the  ability  to  monitor  user  level  access  to  several  EHR    

systems  from  single  interface  using  Splunk  Healthcare  CIM.  App  is  available  from  Splunk  Partner  Conducive  Consul?ng  -­‐  hhp://www.conducivesi.com  

What  HIPAA  Scout  Provides  •  Get  right  to  the  facts  •  Compliance  isn’t  prehy  •  Auditors  are  going  to  love  it!  •  Meaningful  Use  of  EHR  logs  •  HIPAA  viola?on  inves?ga?on  made  easy  

•  Common  Informa?on  Model  •  Universal  EHR  Audi?ng  App    

HIPAA  Privacy  and  Security  Scout™  •  Auditor  Home  Page  

–  Quick  links  to  most  used  reports  

•  Applica=on  Report  Categories    

 

•  New  reports  are  only  limited  by  the  logs  and  the  imagina=on  –  Every  hospital  is  different.  Requirements  and  problems  vary.  

–  Ac?vity  Audit  –  Admin  Audit  –  Disclosure  Report  –  Employee  Info  –  Login  Report  

–  Inves?ga?ons  –  Suspicious  Ac?vity  –  User  Account  Sharing  –  VIP  Pa?ent  Access  

HIPAA  Privacy  and  Security  Scout™  

   

Most  Useful  Dashboards  •  Record  Access  Inves?ga?on  •  Coworker  Record  Access  •  Same  Last  Name  •  Wrong  Unit  •  Employee  Admission  Report  •  Same  Street  

Example  Fields  Available  for  Inves=ga=ons  •  User  Name  •  User  ID  •  Pa?ent  Name  •  Medical  Record  Number  •  Account  Number  •  Hospital  Unit  Number  

Example  Dash:  Same  Last  Name  

44  

Example  Dash:  Wrong  Unit  

45  

Example  Dash:  Record  Access  Inves=ga=on    

46  

Example  Dash:  Coworker  Record  Access  

47  

Example  Dash:  Record  Print  by  Pa=ent  

48  

Splunk  &  Compliance  

49  

"   Re-­‐draN  our  policies  on  regarding  what  a  HIPAA  viola?on  actually  is.  "   Create  policies  regarding  how  we  will  move  forward  with  Splunk  &  HIPAA  

Privacy  and  Security  Scout  app.  "   Will  we  survive  an  audit?  We  have  the  power.  Use  it!  "   Educate  the  masses.    Goal  is  for  Splunk  to  find  nothing.  

Barriers  to  Progress  

50  

"   Beher  at  finding  poten?al  viola?ons.  Takes  more  ?me  to  inves?gate.  Splunk  is  too  good!  

"   EHR  vendors  don’t  supply  enough  audit  info  to  automate  more.  

"   Finding  the  informa?on  with  DB  Connect  takes  lots  of  ?me.  Hope  the  schema  doesn’t  change!  

"   Vendors  unable/unwilling  to  co-­‐operate.  

 

Vision  into  Our  Future  NOW  Compliance  Officers,  Auditors,  Applica?on  Staff,  Opera?ons  Team,  Infrastructure  Team  

Splunk  indexing  mul?ple  diverse,  but  related  systems  

Splunk  search  heads  with  TAs  (Technology  Add-­‐ons)    and  a  Common  Healthcare  App  

EHR,  Finance,  Infrastructure,  Clients,  Servers,  Systems,  

the  list  goes  on….  

Lessons  Learned  

•  Budget  for  servers/storage.  •  Don’t  roll  PoC  into  produc?on  system.  Start  fresh.  •  Sync  ?mes  before  indexing  (where  is  that  s?nking  real  ?me  data?).  •  Expect  to  frequent  answers.splunk.com  if  you  want  to  be  successful.  •  When  inges?ng  data,  it  helps  to  have  friends  on  the  inside.  

“If  I  had  known  then  what  I  know  now…”  

THANK  YOU!    

Ant  Lefebvre  ant@midhosp.org  

Senior  Systems  Engineer  Middlesex  Hospital