Like what you hear? Tweet it using: # - Review Hadoop Security Benchmark project at the Center For...

download Like what you hear? Tweet it using: # - Review Hadoop Security Benchmark project at the Center For Internet

of 27

  • date post

    20-May-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of Like what you hear? Tweet it using: # - Review Hadoop Security Benchmark project at the Center For...

  • Like what you hear? Tweet it using: #Sec360

  • HADOOP SECURITY

    Like what you hear? Tweet it using: #Sec360

  • HADOOP SECURITY About Robert:

    School: UW Madison, U St. Thomas

    Programming: 15 years, C, C++, Java

    Security Work: §  Surescripts, Minneapolis (present) §  Big Retail Company, Minneapolis §  Big Healthcare Company, Minnetonka OWASP Local Volunteer

    CISSP, CISM, CISA, CHPS Email: bob@confidentialsoftware.com

    Twitter: @msp_sullivan

  • HADOOP SECURITY History

    What is new?

    Common Applications

    Threats

    Security Architecture

    Secure Baseline and Testing

    Policy Impact

  • HADOOP HISTORY •  2002 : Doug Cutting & Mike Cafarella: Nutch

    •  Crawl and index hundreds of millions of pages •  2003: Google File System paper released •  2004: Google MapReduce paper released •  2006: Yahoo formed Hadoop 5 to 20 nodes •  2008: Yahoo, Hadoop “behind every click” •  2008: Google spun off Cloudera 2,000 Hadoop nodes •  2008: Facebook open sourced Hive for Hadoop •  2011: Yahoo spins out Hortonworks •  Hortonworks Hadoop 42,000 nodes, hundreds of petabytes

    Derrick Harris “The History of Hadoop from 4 nodes to the future of data”, gigamon.com

  • HADOOP IS The Apache Hadoop software library is a framework that allows for the

    distributed processing of large …

    -  Software Framework -  Distributed Processing -  Large Data Sets -  Clusters of Computers -  High Availability -  Scale to Thousands of Machines Link:

    https://developer.yahoo.com/hadoop/tutorial

  • MAPREDUCE IS NEW

    REDUCE

    MAP

  • HADOOP COMMON APPLICATIONS

    1. Web Search 2. Advertising & recommendations 3. Security Threat Identification 4. Fraud Detection 5. Patient Record Search

  • Source: Yahoo: https://developer.yahoo.com/blogs/ydn/hadoop-yahoo-more-ever-54421.html

  • PATIENT MATCHING AT SURESCRIPTS

    -  Surescripts provides a Patient Matching service -  230 Million Patients -  Over 1 Billion matches last year -  Requirements:

    -  Reliability and performance -  Data Protection at rest is required -  Data Protection in transit is required -  Comprehensive security logging is needed -  ISO 27001 & EHNAC Audit Accreditation status must be

    maintained

  • NOW WHAT?

    SECURE THE BEES

  • HADOOP THREAT MODEL 1)   Unauthorized data access (protected health information access) 2)   Unauthorized data change 3)   Unauthorized job submission, delete or change 4)   Task may access other tasks or access local data 5)   Rogue DataNode, NameNode or Job Tracker 6)   User spoofing to submit workflow as another user From:

    “Adding Security to Apache Hadoop”, Das, O’Malley, Rhadia, Zhang, 2011, http://hortonworks.com/wp-content/uploads/2011/10/security- design_withCover-1.pdf

  • HADOOP SECURITY -  Network Security -  Authentication -  Authorization -  Auditing -  Data Protection

    Admins

    Data Nodes Management Nodes

    Applications

    Enterprise Identity, Logging, Encryption, Key Management

    Application Users

  • DATA PROTECTION -  Network Security -  Authentication -  Authorization -  Auditing -  Data Protection

    -  Encryption at rest; -  Volume, file

    -  Encryption in transit: -  HTTPS

    Admins

    Data Nodes Management Nodes

    Applications

    Enterprise Identity, Logging, Encryption, Key Management

    Application Users

    HTTPS HTTPS

  • SECURITY AUDITING -  Network Security -  Authentication -  Authorization -  Auditing

    -  Failed/Successful Authn. -  System changes -  Access to PHI -  Application logs: HDFS,

    YARN, MapReduce…

    -  Data Protection

    Admins

    Data Nodes Management Nodes

    Applications

    Enterprise Identity, Logging, Encryption, Key Management

    Application Users

  • AUTHORIZATION -  Network Security -  Authentication -  Authorization

    -  Limit user access to function

    -  Limit user access to objects -  Manage delegation of

    access

    -  Auditing -  Data Protection

    Admins

    Data Nodes Management Nodes

    Applications

    Enterprise Identity, Logging, Encryption, Key Management

    Application Users

  • AUTHENTICATION -  Network Security -  Authentication

    -  All users, all applications, all access paths

    -  Apache Knox Gateway -  Authorization -  Auditing -  Data Protection

    Admins

    Data Nodes Management Nodes

    Applications

    Enterprise Identity, Logging, Encryption, Key Management

    Application Users

    HTTPS

  • NETWORK SECURITY -  Network Security -  Authentication -  Authorization -  Auditing -  Data Protection

    Admins

    Data Nodes Management Nodes

    Applications

    Enterprise Identity, Logging, Encryption, Key Management

    Application Users

  • HADOOP SECURE MODE Apache Hadoop Secure Mode: 2.6.0 (March 14’)

    -  Authentication -  Covers HDFS, YARN, MapReduce & Web Console -  Uses central LDAP Server or Active Directory -  Requires Kerberos keytabs for each application

    -  Authorization -  Each Hadoop service has a list of users and groups -  Group permissions on HDFS filesystem components

    -  Audit -  Hadoop log, YARN log, other logs

    -  Data Protection -  Encryption in transit between Hadoop services & clients -  Encryption in transit between DataNodes -  Encryption in transit between web console & clients (HTTPS) -  Encryption at rest for HDFS columns

  • HADOOP SECURE MODE Apache Hadoop Secure Mode: 2.6.0 (March 14’)

    Data Access

    Data Change

    Job Submission

    Task Access

    Rogue Node

    User Spoofing

    Network Security

    Authentication

    Authorization

    Audit

    Data Protection

  • APACHE KNOX The Apache Knox Gateway is a REST API Gateway for interacting with

    Hadoop clusters. The Knox Gateway provides a single access point for all REST interactions with Hadoop clusters.

    Knox can provide:

    •  Authentication (LDAP and Active Directory Authentication Provider) •  Federation/SSO (HTTP Header Based Identity Federation) •  Authorization (Service Level Authorization) •  Auditing Integrations:

    - WebHDFS (HDFS), Templeton (Hcatalog), Stargate (Hbase), Oozie, Hive/ JDBC

    Status: Incubating

  • APACHE RANGER A centralized security framework to manage fine grained access control.

    Status: Incubating

    Authentication

    •  Kerberos in native Apache Hadoop

    •  Secured by the Apache Knox Gateway via the HTTP/REST API

    Authorization •  on the folder and file level, via HDFS •  on the database, table and column level, via Hive •  on the table, column family and column level, via HBase

    Audit

    User access auditing in HDFS, Hive and HBase at IP address, Resource/resource type, Timestamp, Access granted or denied

    Data Protection

    •  Wire, volume and file/column encryotion

    •  HDFS Transparent Encryption (TDE)

    •  Third-Party Partners (Hortonworks)

    Administration

    •  Policy management, administration and delegation

    http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.0/Ranger_U_Guide_v22/index.html#Item1.1

  • HADOOP SECURITY POLICY Authentication of processes:

    -  May go into existing application security policy Security Logging requirements:

    -  Which applications must be logged? -  Add node identifier to standard log records De-anonymization Issues

    -  Sparse data can be de-anonymized through matching to public sources -  Could 200 days of tweets be matched to any of my de-identified data? Key Management & Business Continuity

  • BUILD A SECURITY BASELINE -  Start with your Vendor’s distribution -  Add your company’s sauce -  Review Hadoop Security Benchmark project at the Center For Internet

    Security: -  Apache Hadoop 2.6.0 Benchmark -  Community Discussion -  Editors and members get free access to validation tools -  Everyone gets free access to baselines -  Registration is moderated. That means human registrants are approved and

    receive a welcome email. -  Link:

    -  http://tinyurl.com/HadoopSecurityBenchmark

  • HADOOP SECURITY REVIEW 1.  Start with the threats 2.  Choose your diagram 3.  Ask the standard security questions:

    u Network Security u Authentication u Authorization u Security Audit u Data Protection

    4.  Update your policy 5.  Build a Security Baseline

  • HADOOP SECURITY RESOURCES 1.  Apache “Hadoop in Secure Mode

    http://tinyurl.com/hadoopSecureMode 2.  Yaho