Scott Miao, Trend Micro s [email protected] @ takeshi.miao

2013 Trend Micro 25th Anniversary

Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem

Scott Miao, Trend [email protected]

@takeshi.miao


Who am I

• RD, SPN, Trend Micro• 3+ years for Hadoop eco system• Expertise in HDFS/MR/HBase• @takeshi.miao


Agenda• Threat intelligence problem• Challenges and Solutions• Summary


THREAT INTELLIGENCE PROBLEM

“I want to quickly get an overview of the incident, including its scope, timeline, and impact.”

2013 Trend Micro 25th Anniversary 7


Threat Connect

• A Web Service for Threat Information Report– RESTful Interface to access– Integrated with TM Deep Discovery products

• Relevant and Actionable Intelligence


IP, domain, URL, filename, process, file hash, Virus detection, registry key, etc.

Product 1 Product 2 Product 3 …

Threat Conne

ct

Sand-box File

Detection

Threat

Web

Web Reputatio

nFamil

y Write-up

TE

Virus DB

APT KB

Most relevant threat report with actionable

intelligenceon a single portal

Process and correlates different data sources


CHALLENGES AND SOLUTIONS


StoringReal Time AccessPick Your right tool

Big DataMoving

Process & CorrelateGraph Problem


MOVING


Hadoop

Event Logs

FBSFBS

FBS

Feed Back log ServiceDear users/services

Accumulate small files


STORING


Cost

Easy Process

Archive

HDFS


PROCESS & CORRELATE


Pig/MR

• UDFs• MRs for special

cases

Store

• HDFS• Hbase• Solr• RDB

Time

• Batch• Performance


REAL TIME ACCESS


Real Time

Access

Free form

search

Random Access

Solr Cloud

HBase

• EX. Sandbox Reports

EX. Threat Detection DBs


GRAPH MODEL


Massive scalable ?

Active community ?

Analyzable ?


• We use HBase as a Graph Storage– Google BigTable and PageRank– HBaseCon2012

http://www.slideshare.net/cloudera/3-storing-and-manipulating-graphs-in-h-base-updated-last-minute?from_search=1


HGraph

Schema Design

Blueprints API

Graph Analysis MRs

https://github.com/tinkerpop/blueprints/wiki


PICK RIGHT TOOL


Pick right tool for right usecases

• Silver bullet ?• No one project fits all• One problem may has several choices

http://www.neevtech.com/blog/2013/03/18/hadoop-ecosystem-at-a-glance/


SUMMARY


Small files

• Namenode fsimage would explore the memory

• Too many map tasks to run for a job

FBSFBSFBS


Store your data anyway

• Store all the raw data on the HDFS– Break invisible isolation from different data

sources• Archive your data with deduced easy to use

FileFormat– Trenvi, RC file, ORC file


Know MR more

• Even you are the pig developer– Deal with MR issues– Write better pig-latin– Sometimes you can only use MR


Know your data & usecases

• Realtime ? Batch ?• Access Pattern ?• Therefore, you can pick right tool


THANK YOU GUYS

Scott Miao, Trend Micro s [email protected] @ takeshi.miao

Documents

Transcript of Scott Miao, Trend Micro s [email protected] @ takeshi.miao