Hortonworks Cybersecurity Platform Deep Divedataplatform.jp/program/files/A-7.pdf · Hortonworks...

Hortonworks Cybersecurity PlatformDeep DivePowered by Apache Metron

Dave Russell

Global SME Cybersecurity

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

0 1 2 3 4 5 6 7 8 9

BREACH DURATION VS DATA RETENTION

Average silo retention Average breach unnoticed


0 5 10 15 20 25 30 35 40

Average breach unnoticed

Average silo retention

Yahoo (US)

PoliceOne

BREACH DURATION VS DATA RETENTION

Duration in months


“Sometime in the next few years we're going to have our first

category-one cyber-incident; one that will need a national response.”

Ian LevyTechnical Director

National Cyber Security Centre


Andhra Pradesh Police, IndiaAristotle University of Thessaloniki, GreeceAutomobile Dacia, RomaniaCambrian College, CanadaChinese public security bureauCJ CGVDalian Maritime UniversityDeutsche BahnDharmais Hospital, IndonesiaFaculty Hospital, Nitra, SlovakiaFedExGarena Blade and SoulGuilin University Of Aerospace TechnologyGuilin University Of Electronic TechnologyHarapan Kita Hospital[disambiguation needed], IndonesiaHezhou University

SandvikSão Paulo Court of JusticeSaudi Telecom CompanySberbankShandong UniversityState Governments of India Government of GujaratGovernment of KeralaGovernment of MaharashtraGovernment of West BengalSuzhou Vehicle AdministrationSun Yat-sen University, ChinaTelefónicaTelenor Hungary, HungaryTelkom (South Africa)Timrå Municipality, SwedenUniversitas Jember, IndonesiaUniversity of Milano-Bicocca, ItalyUniversity of Montreal, CanadaVivo, Brazil

HitachiHondaInstituto Nacional de Salud, ColombiaLakeridge HealthLAKSLATAM Airlines GroupMegaFonMinistry of Internal Affairs of the Russian FederationMinistry of Foreign Affairs (Romania)National Health Service (England)NHS ScotlandNissan Motor Manufacturing UKO2, GermanyPetrobrásPetroChinaPortugal TelecomPulse FMQ-ParkRenaultRussian Railways


https://pixabay.com/p-906036/


http://www.thebluediamondgallery.com/wooden-tile/images/failure.jpg


Problem Posed For Security Analysts

Too many disparate tools

Too many alerts to process

Too much noise

Too slow

Unnecessary plumbing work


Timeline

Sept 2014

June2015

Dec 2015

April 2016

Feb 2017

Jun 2017

Sept2017

OpenSOC

Beta

OpenSOC

Community Edition

Metron enters Apache Incubator

Apache Metron 0.1 Apache Metron 0.3

HCP 1.1

Apache Metron 0.4

HCP 1.2

Secure Cluster

Apache Metron0.4.1

HCP 1.3

Alerts UI

April2017

Apache Metronexits Apache

Incubator


Hortonworks Cybersecurity Platform – Powered By Apache Metron

Real-Time ingestion of application and system logs

Real-Time cyber security dashboard and cyber workbench

Real-Time ingestion, correlation and enrichment of PCAP and NetFlow telemetries

Real-Time integration of Cyber security feeds

Advanced statistical and machine learning models to detect cyber security attacks

Integration with existing SIEMs and enterprise assets

ApacheMetron

Cyber Security Data Ingestion

Package

Cyber Security Analytics


Data Se

rvices an

d In

tegratio

n Laye

r

ModulesReal-time ProcessingCyber Security Engine

TelemetryParsers

Apache Metron: Overview

Tele

metry In

gest B

uffe

r

TelemetryData Collectors

Real-timeEnrich / ThreatIntel Streams

PerformanceNetwork

IngestProbes

/ OtherMachine Generated Logs(AD, App / Web Server,

firewall, VPN, etc.)

Security Endpoint Devices (Fireye, Palo Alto,

BlueCoat, etc.)

Network Data(PCAP, Netflow, Bro, etc.)

IDS(Suricata, Snort, etc.)

Threat Intelligence Feeds(Soltra, OpenTaxi,third-party feeds)

TelemetryData Sources

Data Vault

Real-Time Search

Evidentiary Store

Threat Intelligence Platform

Model as a Service

Community Models

Data Science Workbench

PCAP Forensics

Threat IntelligenceEnrichment

Indexers and WriterProfiler Alert Triage

Cyber SecurityStream Processing Pipeline


Profiler: User and Entity Behavior analytics

HBaseProfiler Bolt

• HyperLogLogPlus

• T-Digest

• Bloom filter

• MAD outlier

Cardinality

Statistics

Presence

Outliers

How many servers connected?

Average over different periods

Finding small needles in big haystacks

Detecting unusual events in streams

Triage Scoring Model features Aggregations over Time

Fast Cache


YARN

Model as a Service

Historical Data Store

Model ServiceREST interface

Model Store

ZookeeperStorm Enrichment Bolt Service Discovery

HDFS

Trai

n /

Up

dat

e

HBase

Metron JSON Object

Metron JSON Object with added score, confidence

etc. from model

• Real-time scoring• Versioned deployment of

models in any tech• Service discovery


A deeper look at Hortonworks Cybersecurity Platform…


“stop blaming the users, and make the systems usable.”

Ian LevyTechnical Director

National Cyber Security Centre


Questions?


FAQ

Combined or separate cybersecurity datalake?

Cloud or on premise?

Streaming engine, why not spark/flink/any other hotness?

What ML in Model as a Service?

Where can I get more information?– https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.3.0/index.html

– http://metron.apache.org/

– https://cwiki.apache.org/confluence/display/METRON/Community+Resources

– https://github.com/apache/metron

Hot? Warm? Cold?

What is supported out of the box?

https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.3.0/index.html

http://metron.apache.org/

https://cwiki.apache.org/confluence/display/METRON/Community+Resources

https://github.com/apache/metron


Apache Metron Datasheet Ingest:

– Apache NiFi: syslog, socket, file, web services, SQL, RDBMS, Windows Event Log, FTP, MQ, JMS

– High-performance DPDK Packet Capture

Parsers:

– Cisco ASA

– Bluecoat

– Fireeye

– Palo Alto

– SourceFire

– WebSphere

– Snort

– Bro

– YAF (Netflow, IPFIX)

– Grok (Custom)

– Java (Custom)

– JSON

– Applications: DHCPD, AD

Enrichments and threat feeds:

– Geo

– Whois

– HBase

– JDBC

– Stellar

– CSV

– Stix, Taxii threat intel feeds

Analytics features:

– Profiler

– Model Services

– Threat Triage

Indexing and search:

– Elasticsearch, Kibana

– Solr

– HDFS

– Kafka

Data science features:

– Spark Machine Learning

– Zeppelin notebooks and reporting

– Wide partner eco-system

Forensic features:

– PCAP inspector

– PCAP query

– Long term data store


Thankyou

Hortonworks Cybersecurity Platform Deep Divedataplatform.jp/program/files/A-7.pdf · Hortonworks...

Documents

Transcript of Hortonworks Cybersecurity Platform Deep Divedataplatform.jp/program/files/A-7.pdf · Hortonworks...