IMCSummit 2015 - Day 1 IT Business Track - The Great Debate: Scale Out vs Scale Up: Is it either, or...

© Hitachi Data Systems Corporation 2015. All rights reserved.1 © Hitachi Data Systems Corporation 2015. All rights reserved.

Michael Hay – VP & Chief Engineer GISDTWITTER: @mihay42

The Great Debate: Scale Out vs Scale Up: Is it either, or …or both?

© Hitachi Data Systems Corporation 2015. All rights reserved.2

A CONTRARIAN TIMEA CONTRARIAN TIME

BLURAY ARCHIVEBLURAY ARCHIVE

FLASH ARCHIVEFLASH ARCHIVE

SOURCE – URL: https://flic.kr/p/dA2Dc | AUTHOR: Rberteig | LICENSE: Creative Commons

SOURCE – URL: BluRayDiscB ack.p ng | AUTHOR: Cdnomad| LICENSE: Public Domain

A NO-SQL FIRM WITH IN-MEMORY FOCUS

VOLATILITY | UNCERTAINTY | COMPLEXITY | AMBIGUITY


ITILITIL

WATERFALLWATERFALL

AGILEAGILE

AGILEAGILE

DEVOPSDEVOPS

ITERATIVE…

ITERATIVE…

ENGINEERING/DEVELOPMENTENGINEERING/DEVELOPMENT

IT & OPERATIONSIT & OPERATIONS

20012001

20092009

20112011

20192019


engineered systems!

!converged stacks!

chef!

puppet

cfeng

ine!c

apist

rano !scale-up primitives!

scale-out primitives!

!gan

glia!p

axos!

!ans

ible!ca

ctai!

papert

rail!s

ensu

!ant!jenkins

maven!gradel!stash

!nag

ios!ic

inga

nova!glance

QUALITY

OPERATIONS

!nebula!cinder!swift

!restful!python!ruby !google flow

! storm

!cql!s3 !m

ongo!yarn !cassandra

DEVELOPMENT

DEVOPS CONTINUOUS DEPLOYMENTCONTINUOUS DEPLOYMENT

deployments permonth

deployments per year

people deploy to production

SOURCE: ETSY’s 2012 DevOps statistics reported at SXSW


SHARED NOTHINGEVENTUALLY CONSISTENTDATA VARIANCE

SCALE-OUT, SCALE-UP DEFINEDSCALE-OUT, SCALE-UP DEFINED

SHARED EVERYTHINGCONSISTENTPERFECT DATA

10sof nodes10s

of nodes

≥100sof nodes

≥100sof nodes


SHARED NOTHINGEVENTUALLY CONSISTENTDATA VARIANCE

SCALE-OUT, SCALE-UP DEFINEDSCALE-OUT, SCALE-UP DEFINED

SHARED EVERYTHINGCONSISTENTPERFECT DATA

10sof nodes10s

of nodes

≥100sof nodes

≥100sof nodes

NO VARIANCE WITH A PROBABILTY OF 1 FOR CONCLUSION

THINK: BANK ACCOUNT BALANCES, FINANCIAL TRADES, COMPONENT FAILURES ON CRITICAL INFRASTRUCTURES, OR ESSENTIALLY ANYTHING WITH RISK AND REGULATION

VARIANCE WITH A PROBABILTY LESS THAN 1 FOR CONCLUSION

THINK: SENTIMENT ANALYSIS, PREDICTIVE MAINTENANCE, PROCESS CONTROLS OR ANYTHING WHERE DECISIONS CAN BE MADE WITH SOME DEGREE OF RISK TOLERANCE


SCALE-OUT, SCALE-UP A CAGE MATCHSCALE-OUT, SCALE-UP A CAGE MATCH

WINNER?WINNER?


SCALE-OUT, SCALE-UP ON THE SAME…SCALE-OUT, SCALE-UP ON THE SAME…

EAM!EAM!


PUTTING IT ALL TOGETHERPUTTING IT ALL TOGETHER

scale-upscale-out

① Acquire packet data to discern indicators of packet loss and persist

F87AB:01001 !F87AA:01000 !F87A9:00111 !F87A8:00110 !F87A7:00100 !

HITACHI MARS

HITACHI STREAM

② Lookup customer name, ID, etc. from HANA in real-time APIAPI③ Spin up Hadoop to compute sentiment on Twitter data by customer

API

④ Combine all data to try and correlate packet behaviors with sentiment

APIAPIAPI


PUTTING IT ALL TOGETHERPUTTING IT ALL TOGETHER① Acquire packet data to discern indicators of packet loss and persist

F87AB:01001 !F87AA:01000 !F87A9:00111 !F87A8:00110 !F87A7:00100 !

HITACHI MARS

HITACHI STREAM

② Lookup customer name, ID, etc. from HANA in real-time APIAPI③ Spin up Hadoop to compute sentiment on Twitter data by customer

API

④ Combine all data to try and correlate packet behaviors with sentiment

APIAPIAPI

LATENCY LEADS TO POOR QUALITY LATENCY LEADS TO POOR QUALITY

scale-upscale-out


EDGE CORE

AGUMENT WITH EDGE SCALE-INAGUMENT WITH EDGE SCALE-IN① Spin up packet acquisition & WAN-ACC engines on the edge

② Load the relevant models and queriesAPIAPI③ Decommission packet acquisition in the core

API APIAPIAPI

scale-in (to the) edge

F87AB:01001 !F87AA:01000 !F87A9:00111 !F87A8:00110 !F87A7:00100 !

HITACHI MARS

HITACHI STREAM HITACHI

STREAM

F87AB:01001 !F87AA:01000 !F87A9:00111 !F87A8:00110 !F87A7:00100 !

HITACHI MARS

scale-upscale-out


FABRICFABRIC NETWORKNETWORK

IN-MEMORY COMPUTE PRESSURES NETWORKS IN-MEMORY COMPUTE PRESSURES NETWORKS

IMPLICATION: NEW (STORAGE) ARCHITECTURES ARE REQUIRED

GigaBYTES/sec[UNIT: 8 bits]

GigaBYTES/sec[UNIT: 8 bits]

GigaBITS/sec[UNIT: 1 bit]

GigaBITS/sec[UNIT: 1 bit]

THROUGHPUTTHROUGHPUT

0.1’s MICROseconds[ORDER: 0.0000001s]0.1’s MICROseconds[ORDER: 0.0000001s]

10’s MICROseconds[ORDER: 0.00001s]

10’s MICROseconds[ORDER: 0.00001s]

LATENCYLATENCY

FABRICS (e.g. PCIe, IB) LEAD SERIAL NETWORK RATES


(IN 2020 ACCORDING TO CISCO)

INTERNET OF THINGSINTERNET OF THINGS


OROR

OROR

OROR

OROR

IS IN-MEMORY COMPUTE WAN COMPATIBLE? IS IN-MEMORY COMPUTE WAN COMPATIBLE?

10Gbpsleased line10Gbpsleased line


Transition to Multi-Modal Architectures

TRANSITION

DATABASE

APPLICATION SERVER

WEB PRESENTATION

APPLICATION(S)/SOURCE(S)

WAREHOUSE/MARTS

CUBES

IN MEMORY

ANALYSIS…

THING/ SOURCEANALYSIS

DATA LAKES/OBJECT CLOUDSANALYSIS ANALYSIS ANALYSIS

LOW LATENCY

DEEP CAPACITY

Applications and sources at real-time and near real-time focus that are latency intolerant

Capacity optimized persistence and data lakes with embedded analytics

PERSISTENT STORE

SCRATCH

PROCESSING

HPC DWH 3 TIER



TRANSITION

DATABASE

APPLICATION SERVER

WEB PRESENTATION


WAREHOUSE/MARTS

CUBES

IN MEMORY

ANALYSIS…



LOW LATENCY

DEEP CAPACITY



PERSISTENT STORE

SCRATCH

PROCESSING

HPC DWH 3 TIER

PROCESSING:MEMORY:

STORAGE:I/O & SENSORS:

4000 MIPS (64bit)0.5 GB8 GB1 GigE | ACCELEROMETER | … E

CU



TRANSITION

DATABASE

APPLICATION SERVER

WEB PRESENTATION


WAREHOUSE/MARTS

CUBES

IN MEMORY

ANALYSIS…



LOW LATENCY

DEEP CAPACITY



PERSISTENT STORE

SCRATCH

PROCESSING

HPC DWH 3 TIER

COMPACT U’sFLASH TRENDING

KVM/DOCKEREVO RAIL

storage (memory) control

scale-in (to the edge):stream, filtering/winnowing, control systems, small edge applications



TRANSITION

DATABASE

APPLICATION SERVER

WEB PRESENTATION


WAREHOUSE/MARTS

CUBES

IN MEMORY

ANALYSIS…



LOW LATENCY

DEEP CAPACITY



PERSISTENT STORE

SCRATCH

PROCESSING

HPC DWH 3 TIER

scale-out compu-store

scale-out obj-store

scale-up: object-dnssap-hanagraphkvs

MEMORIESMEMORIES

storagecontrol



TRANSITION

IN MEMORY

ANALYSIS…



LOW LATENCY

DEEP CAPACITY



PERSISTENT STORE

SCRATCH

PROCESSING

HPC

scale-out hyper-converged

scale-out obj-store

scale

-up

objec

t-dns

MEM

OR

IES

MEM

OR

IES


CREDITS for images/materials

1. Hadoop Coffee Image

‒ Author: http://www.flickr.com/photos/yukop/

‒ License: http://creativecommons.org/licenses/by-sa/2.0/

2. Waterfall photo

‒ Author: https://www.flickr.com/photos/thart2009/

‒ License: https://creativecommons.org/licenses/by/2.0/

‒ URL: https://flic.kr/p/9FqqQF

3. Agile development

‒ Author: https://www.flickr.com/photos/improveit/

‒ License: https://creativecommons.org/licenses/by/2.0/

‒ URL: https://flic.kr/p/Nt7cB

4. BluRay Icon

‒ Author: Jordan (http://jrdng.deviantart.com/?rnrd=8324)

‒ License: http://commons.wikimedia.org/wiki/Main_Page

IMCSummit 2015 - Day 1 IT Business Track - The Great Debate: Scale Out vs Scale Up: Is it either, or...

Technology

Transcript of IMCSummit 2015 - Day 1 IT Business Track - The Great Debate: Scale Out vs Scale Up: Is it either, or...