Intridea ajn-rttos OA NYC Summit

Anthony NyströmFellow, Managing Director of Engineering

Tuesday, June 18, 13

What is Intridea?


What is Intridea?We design and develop apps: Web, Mobile and Data


What is Intridea?We design and develop apps: Web, Mobile and Data

Founded inWashington, DC


What is Intridea?We work with cool clients – really!

We design and develop apps: Web, Mobile and Data



What is Intridea?

40+ Intrideans:Designers/Developers/Scientists+ Smart biz folks

We work with cool clients – really!




What is Intridea?



We work from anywhere!




What is Intridea?






We are growing


What is Intridea?




We hire the best and the smartest



We are growing


The guy on stage

Intridean:



Data Science in the NOW!It takes an army of TOOLS


An Army of Tools you say?


An Army of Tools you say?

• I am going to talk about what NOW means in Data Science

• Databases, Streaming Engines, Query Engines and Interfaces

• We are going to look at many of them and single out a few

• Each has a respected and in some cases competing set of features


Why is NOW in data Special?



Actionable Intelligence & Knowledge




NOW has innate context




NOW has innate context

TIME is THE natural facet for our minds & life!


Why is NOW in data Special?Trends | Patterns | Extraction



Data Centric Trends



Data Centric Trends

Pattern Extraction (ML/NLP)



Data Centric Trends


Signature Extraction (Binary, Encoded)



Data Centric Trends



Not user input data like Google, Yahoo etc.



Data Centric Trends




“I am looking for data that conforms to a learned or known pattern”



Data Centric Trends




“I am looking for data that conforms to a learned or known pattern”

“I am looking for data that matches a predefined signature”


Why is NOW in data Special?Routing | Transformation | Computation



Intelligent Routing



Transformation & Computation

Intelligent Routing




Intelligent Routing“I need to replicate/fork that of criteria x portions of this data

stream”




Intelligent Routing“I need to replicate/fork that of criteria x portions of this data

stream”

“I need to transform certain fields” or “I need to compute a some value on certain fields”


Why is NOW in data Special?Algorithmic Speciality



Concepts

Algorithmic Speciality



Regression

Concepts




Relationships

Regression

Concepts




Relationships

Regression

Concepts


What does a value represent or infer (NLP/ML/k-NN)



Relationships

Regression

Concepts



How is a value related to another value or How can we predict such relations



Relationships

Regression

Concepts



How is a value related to another value or How can we predict such relations

Topological, Ontological, Forest (Evolutionary/Random) (NLP)


Point of Sale System • Terminal• Admin• Tablet


Merck• RT Persona• RT Data• Browser


Where is NOW in data?


Where is NOW in data?

Data Creation Time | Data Consumption Time


Latency


Latency

Standard - NOPE!



Latency

Standard - NOPE!

Depends upon the Medium - YEP!



Latency

Standard - NOPE!

Depends upon the Consumer - YEP!




Latency

Standard - NOPE!

Depends upon the Consumer - YEP!


Depends upon Technology - YEP!



NOW and Latency


NOW and Latency

Real-Time


NOW and Latency

Real-Time

Near Real-Time


NOW and Latency

Real-Time

Some-Time

Near Real-Time


NOW and Latency

Real-Time

Some-Time

Data that is consumed immediately after creation

Near Real-Time


NOW and Latency

Real-Time

Some-Time

Data is consumed within seconds/minutes


Near Real-Time


NOW and Latency

Real-Time

Some-TimeData is consumed when requested & is NOT RT nor NRT

Data is consumed within seconds/minutes


Near Real-Time


Physiological Latency


Perception:

Research suggests that the human retina transmits data to the brain at the rate of 10 million bits per second, which is close to that of 10 base Ethernet connection!

We can perceive changes in reality at ~ 13-15 frames per second (fps, or Hz), Our perception of reality fully refreshes itself ~ once every 77



Perception:



Stock Exchange ~ 5-100 milliseconds (ms)



Web Sites ~ 50-400 milliseconds (ms)

Perception:







Perception:



Games (FPS) ~ 10-150 milliseconds (ms)





Perception:



Games (FPS) ~ 10-150 milliseconds (ms)

Social/Games ~ 200 ms -1 second




Real-Time (DB’s, Index’s, FS’s)


Real-Time (DB’s, Index’s, FS’s)No particular order



• MySQL

No particular order



• MySQL

• SQL Server

No particular order



• MySQL

• PostgreSQL

• SQL Server

No particular order


https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CC8QFjAA&url=http%3A%2F%2Fwww.postgresql.org%2F&ei=h1M9UdObPInQ2AWL_YDQDg&usg=AFQjCNHhuOZnVQteX8H9LuxN9CJBVzl4ZA&bvm=bv.43287494,d.b2U



• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server

No particular order




• Mongo


• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server

No particular order




• Mongo


• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server

• Elastic Search (Lucene)

No particular order




• Mongo


• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server

• Elastic Search (Lucene)• Solr

No particular order




• Mongo


• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server


• HDFS

No particular order




• Mongo


• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server


• HDFS

• HBase

No particular order




• Mongo


• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server


• HDFS

• Oracle

• HBase

No particular order






• Mongo


• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server


• HDFS

• Oracle

• ERTFS

• HBase

No particular order






• Mongo


• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server


• Redis

• HDFS

• Oracle

• ERTFS

• HBase

No particular order






• Mongo


• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server


• Redis

• HDFS

• Oracle

• ERTFS

• HBase

• Cassandra

No particular order






• Mongo


• MySQL

• PostgreSQL

• Neo4j (Graph)

• SQL Server


• Redis

• HDFS

• Oracle

• ERTFS

• HBase

• Cassandra• Riak

No particular order






HBase


HBase

Regions and HDFS


HBase

Regions and HDFS

Scaling


HBase

Regions and HDFS

Hadoop

Scaling


HBase

Regions and HDFS“Regions” Data files for regions are stored in HDFS and replicated to multiple nodes in the cluster. As well, allocation in to the cluster is

rather automatic

Hadoop

Scaling


HBase

Regions and HDFS“Regions” Data files for regions are stored in HDFS and replicated to multiple nodes in the cluster. As well, allocation in to the cluster is

rather automatic

Hadoop

ScalingFault Tolerance

Commodity Machines


HBase

Regions and HDFS

Runs on top of HadoopMapReduce Integration

“Regions” Data files for regions are stored in HDFS and replicated to multiple nodes in the cluster. As well, allocation in to the cluster is

rather automatic

Hadoop

ScalingFault Tolerance

Commodity Machines


Cassandra


Cassandra

Always Writable


Cassandra

Always Writable

Scaling


Cassandra

Always Writable

More...

Scaling


Cassandra

Always WritableEven when internally the write fails. However, the data will eventually

become consistent (Tunable)

More...

Scaling


Cassandra

Always WritableEven when internally the write fails. However, the data will eventually

become consistent (Tunable)

More...

ScalingCan span data centers

Peer-to-Peer communication between nodes (Gossip)


Cassandra

Always Writable

Supports MapReduceSupports Range Queries

Even when internally the write fails. However, the data will eventually become consistent (Tunable)

More...

ScalingCan span data centers

Peer-to-Peer communication between nodes (Gossip)


Redis


Redis

Transactions


Redis

Transactions

An evolutionary Key-Value Store


Redis

Transactions

Pub-Sub



Redis

TransactionsAtomic operations (MULTI/EXEC/Discard) Queue your operations and

EXEC/Commit as transaction. Allows for Roll-back support.

Pub-Sub



Redis

TransactionsAtomic operations (MULTI/EXEC/Discard) Queue your operations and

EXEC/Commit as transaction. Allows for Roll-back support.

Pub-Sub

An evolutionary Key-Value StoreSupports complex types that are closely related to fundamental data

structures. No need for abstraction layer.


Redis

Transactions

Publish - Push messages to a channelSubscribe - Listen to a channel

Atomic operations (MULTI/EXEC/Discard) Queue your operations and EXEC/Commit as transaction. Allows for Roll-back support.

Pub-Sub

An evolutionary Key-Value StoreSupports complex types that are closely related to fundamental data

structures. No need for abstraction layer.


Near Real-Time & Real-Time


Near Real-Time & Real-TimeQueries and Streams



• Storm

Queries and Streams



• Storm

• Kafka

Queries and Streams



• Storm

• Drill/Dremel

• Kafka

Queries and Streams



• Storm

• Drill/Dremel

• Hadoop

• Kafka

Queries and Streams


• MapReduce


• Storm

• Drill/Dremel

• Hadoop

• Kafka

Queries and Streams


• MapReduce


• Storm

• Drill/Dremel

• Hadoop

• MapReduce v2 (YARN)

• Kafka

Queries and Streams


• MapReduce


• Storm

• Drill/Dremel

• Hadoop


• Pig • Kafka

Queries and Streams


• MapReduce


• Storm

• Drill/Dremel

• Hadoop


• Pig

• Hive

• Kafka

Queries and Streams


• MapReduce


• Storm

• Drill/Dremel

• Hadoop • Cascalog


• Pig

• Hive

• Kafka

Queries and Streams


• MapReduce


• Storm

• Drill/Dremel

• Hadoop • Cascalog


• Pig

• Hive

• Kafka

• DataTurbine

Queries and Streams


MapReduce/Hadoop


MapReduce/Hadoop

Scale


MapReduce/Hadoop

Scale

Development


MapReduce/Hadoop

Scale

Batch

Development


MapReduce/Hadoop

Scale100’s to 1000’s of server nodes

Extreme and cheapSimple programming model

Batch

Development


MapReduce/Hadoop

Scale100’s to 1000’s of server nodes

Extreme and cheapSimple programming model

Batch

DevelopmentJava, Python, Grep & Others...


MapReduce/Hadoop

Scale

Complex Multi-Step Processing

100’s to 1000’s of server nodesExtreme and cheap

Simple programming model

Batch

DevelopmentJava, Python, Grep & Others...


Storm


Storm

FAST


Storm

FAST

Integration


Storm

FAST

Assurance

Integration


Storm

FASTOver a million tuples processed per second per node

Assurance

Integration


Storm

FASTOver a million tuples processed per second per node

Assurance

IntegrationIntegrates with any queueing system and any database system

Handles the parallelization, partitioning, and retrying on failures when necessary


Storm

FAST

Scalable, Fault-Tolerant, Guarantees your data will be processed!

Over a million tuples processed per second per node

Assurance

IntegrationIntegrates with any queueing system and any database system

Handles the parallelization, partitioning, and retrying on failures when necessary


CQL/StreamQL/SparQL/QL-RTDB/



Languages



Languages

Scalable



Languages

SQL Idioms

Scalable



LanguagesHuman Readable

SQL Idioms

Scalable



LanguagesHuman Readable

SQL Idioms

ScalableSimultaneous n Queries upon both stream data and static



Languages

All support to a large degree what you would expect from SQL

Human Readable

SQL Idioms

ScalableSimultaneous n Queries upon both stream data and static


PIG


PIG

Language


PIG

Language

Parallelization


PIG

Language

Underneath

Parallelization


PIG

LanguageHigh Level and easy to understand (Pig Latin)

Underneath

Parallelization


PIG

LanguageHigh Level and easy to understand (Pig Latin)

Underneath

ParallelizationIt is trivial to achieve parallel execution of simple, "embarrassingly

parallel" data analysis tasks


PIG

Language

Essentially a MapReduce sequence compiler

High Level and easy to understand (Pig Latin)

Underneath

ParallelizationIt is trivial to achieve parallel execution of simple, "embarrassingly

parallel" data analysis tasks


PIG


PIGExample Pig Script


PIG


PIGThat same example using MR Java code


The perfect Army!


The perfect Army!

In Memory


The perfect Army!

In Memory

Identify and Plan


The perfect Army!

In Memory

Consumer

Identify and Plan


The perfect Army!

In MemoryKeep as much as you can IN MEMORY! Think Redis...

Consumer

Identify and Plan


The perfect Army!

In MemoryKeep as much as you can IN MEMORY! Think Redis...

Consumer

Identify and PlanWhat data can be batch processed and what can’t! Think

Hadoop and Storm (for stream) and HBase (for adhoc)


The perfect Army!

In Memory

Who is the data consumer? Person or Process? Think Pig or xQL’s for both!

Keep as much as you can IN MEMORY! Think Redis...

Consumer

Identify and PlanWhat data can be batch processed and what can’t! Think

Hadoop and Storm (for stream) and HBase (for adhoc)


www.intridea.com


[email protected]@AnthonyNystrom

Thank You Gracias Merci Danke


http://www.intridea.com

http://www.intridea.com

mailto:[email protected]

mailto:[email protected]

Intridea ajn-rttos OA NYC Summit

Documents

Transcript of Intridea ajn-rttos OA NYC Summit