AWS를 활용한 Big Data 실전 배치 사례 :: 이한주 :: AWS Summit Seoul 2016
-
Upload
amazon-web-services-korea -
Category
Technology
-
view
887 -
download
6
Transcript of AWS를 활용한 Big Data 실전 배치 사례 :: 이한주 :: AWS Summit Seoul 2016
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
베스핀글로벌이한주대표
2016.5.17
AWS를활용한 Big Data 실전배치사례
모니터링, IOT의 Big Data 배치및Data Visualization 사례
About HanJoo Lee
CEO & Cofounder , BESPIN GLOBAL
General Partner,
1998 Cofounded
IT Infra entrepreneur
Built and Operated 12 Datacenters100,000+ 서버운영
11 Countries with 12 Datacenters around the world: 1998년 시작
Australia Belgium Canada France Germany Korea Netherlands USA UK India Rumania
Sydney Antwerp Vancouver Paris Frankfurt Seoul Amsterdam Chicago London Mumbai Bucharest
Hannover Tampa / Austin
Fort Lauderdale
About Bespin Global
베스핀글로벌Managed Service Provider한국에서최초로 AWS MSP 인증받은회사한국에는나와있는 3개 MSP 인증중 2개가 Bespin
• Cloud 전략수립• Cloud Architecture• Cloud Migration• Cloud 운영• Hybrid IT Management• 한국과중국에서 140명
About Bespin Global
Cloud의길잡이
Data lifecycle in Cloud Infra
클라우드 인프라
데이터 애플리케이션
데이터 입력 가공 분석 활용
• IoT를 통해 세상의 모든
것들을 디지털 데이터
로 수집
• 빅데이터 기술을 통해
과거의 텍스트 중심의
정형 데이터뿐만 아니
라 동화상, 음성 등 비
정형 데이터까지 처리
• 대규모 데이터를 실시
간 분석해 결과를 추출
하고 의사결정 진행
• 디지털 골든크로스로서
인간의 두뇌 이상의 의
사결정과 작업을 컴퓨
터가 스스로 수행
머신러닝
빅데이터
IoT
Bespin Global
김성수 상무
Building Big Data Backend Using Native AWS Services
WhaTap
김성조 CTO
Building Big Data System Using Proprietary File System while Using AWS
N3N
김호민 대표
Building Big Data Visualization Using Splunk and AWS
Big Data for SaaS based Monitoring Services
Bespin Global: How to build and Operate Big Data System on AWS?
Avoid unplanned Downtime
Monitoring Process
Server
DBMS
Application
Monitoring Target
분석 DB
Basic Process
Advanced Process
(사후 처리)
Metric
Data 수집 및모니터링
Big DataProcessing
※ Ca사 모니터링 솔루션 UIM 적용
Threshold Alarm
사전 예방 활동
Scale up
Scale out
(Time Series, Correlation, …)
알람 or 장애 조치
Monitoring Dashboard
Building Big Data System on On-Premise
R Analysis( Time Series / Correlation )
Pre detection
DatabaseDistributed Messaging System
…..
M N1 N2 Nn
Cluster ComputingFramework
Yarn
…..
HDFS
Time Series Database
N2 Nn…..M N1
Analysis Platform
……
Data Sources
Application
Server
DBMS
Log
Age
nt
…
Applications
BI
Marketing
Advertisement
…
Limited Scalability
Internal IT Resources to Manage Cluster (Tuning and monitoring, etc…)
Upfront capital expense
NoSQL
…
사전 진단/예방
ProducerStream
Collector
ConsumerStream
Building Big Data System on AWS
Data Sources
Application
Server
DBMS
Log
Age
nt
…
ProducerStream
Kinesis
Streaming Data Platform
EMR AWS Elastic Search
Time Series Database
RDS
Pre detection
KCL
Kinesis S3 Connector
Consumer
S3Lambda
Archiving Data
EMR
Redshift
RDS
An
alysis (BI So
lutio
ns)
Application
Upsell Analysis
Marketing
Advertisement
…
Elastic and Highly Scalable
Don’t Manage Cluster (and AWS has tuned Services)
Easy to Use and Deploy to Multiple Locations
No upfront capital expense – Pay as you go
Cluster monitoring
Cluster monitoring
R Analysis( Time Series / Correlation )
R on EMR
Consumer
사전 진단/예방
On-Premise vs AWS
R Analysis( Time Series / Correlation )
Database
Streaming Data Platform
Kinesis
Time Series Database
Elastic Search
Easy to Use and Managed Console
RDS
Database
Complex to Use Managed Cost+
Cluster Managed Cost (Tuning and monitoring, etc…)+
Elastic Scalability
Don’t Managed Cluster (and AWS has tuned Services)
Upfront capital expense
Pay as you go
Distributed Messaging System
…..
M N1 N2 Nn
Cluster ComputingFramework
Yarn
…..
Time Series Database
N2 Nn…..M N1
Limited Scalability
EMR
R Analysis( Time Series / Correlation )
R on EMR
WhaTap: Running Cloud Based Monitoring Service on AWS
Monitoring Issue Changes
Testing Open Stable
Application Physical SystemNeeds
Unknown bottlenecks
Known issues
Monitoring Solutions
Needs
Testing Open Stable
APM(분석, 문제 해결, 실시간)
SMS(관제, 경고 이벤트)
The differences on Cloud
Dedicated Unix Infra
개별 서버 성능
서버 관리 유연성
서버(노드) 수
시스템 처리 성능
부분 오류 가능성
Cloud Infra
서비스 안정성
Monitoring Service(SaaS based)
Master
security group security
1
2
200 Slave
Monitoring Service
APM on Premise vs on Cloud
• Data collection per several seconds• Active Transaction Analysis• All Transaction Profiling
On Premise Solution
• Data collection per 10 seconds• Active Transaction Analysis• Selective Profiling & Integrative Analysis
On Cloud Service
사용자접속정보,
트랜잭션, 자원, 튜닝정보를
하나의관점으로비교분석할수
있어야합니다. Archiving
TechUser (browser)
Transaction
IP, 도시/국가,
접속 매체, OS,
최근 방문자,
액티브 사용자
TPS, Response
time, Error
time, URL, SQL
Resource
CPU, Heap,
Disk, GC
Tuning
History 비교분석
Active Stack 분석
Hit Map 분석
Integrated analysis of perf. data
EC2 instance
DataServer
EBS
Backup (S3)
StaticsticsRDS (MySql)
Scalable APM (AWS based)
EC2 instance
DataServer
EBS
EC2 instance
DataServer
EBS
EC2 instance
DataServer
EBS
Scalable!!
Elastic Load Balancing
Amazon Route 53
Auto Scaling group
EC2 instance
DataServer
EBS
Backup (S3)
Project/Tenant MgmtRDS (MySql)
StaticsticsRDS (MySql)
ElastiCache(REDIS)
Elastic Load Balancing
Amazon Route 53
Auto Scaling group
Region (Tokyo)
Elastic Load Balancing
Auto Scaling group
EC2 instance
DataServer
EBS
Backup (S3)
StaticsticsRDS (MySql)
Region (US.West)
ElastiCache(REDIS)
Elastic Load Balancing
Auto Scaling group
Region (Tokyo)
Region (US.West)
Elastic Search
Scalable APM (AWS based)
APM to SMS on Cloud
Needs
Testing Open Stable
APM• 분석, 문제 해결• 실시간• 대용량 데이터
SMS• 관제, 문제 인지• 경고 이벤트• 소규모 데이터
SUPPORTS
MOBILE
LOWER
COST
SHORTER
IMPLEMENTATION
PERIOD
SMS on Cloud
1 Month
EXISTING MONITORING
CLOUD MONITORING
5 Minutes
$10000 + α
EXISTING MONITORING
CLOUD MONITORING
$100/mon
EXISTING MONITORING
CLOUD MONITORING
In Office
Anywhere
WhaTap TokyoWhaTap Virginia
WhaTap Oregon
출처 : https://rctom.hbs.org/
WhaTap can service to global customers
Big Data Visualization on AWS
N3N: What We Do
N3N provides IOT Visualization for Fortune 500 Companies and Cities Governments.
N3N - Physical Visualization
Global
Asia
Korea
Seoul
Suwon
Z: Hierarchies
Daejeon
X: Dependencies Y: Relationships
N3N - Logical Visualization
Business Unit
Service
Application
X: Dependencies Y: Relationships
ApplicationData Bases
Z: Hierarchies
N3N - Business Impact
Quantitative BenefitsKPI Before After Result Remarks
MTTD, MTTR 2 days 1.5 days -0.5 day CISCO Stat, Splunk .conf 2014
Big Data Solution Usage Rate 5% 100% +95%
Improvements in Stability & availability
+25%
Reduction in Operational Cost +10%
25% 95% 25% 10%
Wrap Up
Visualization Big data for Biz Decision
Monitoring Log
IOT event Log
System Event Log
AWS event Log
VisualizedAnalysis On AWS
Wrap Up
- Big Data on AWS –Monitoring Use Case
- Not only for Monitoring
- Merchandise, Logistics, Turnover, Marketing, etc.
- Best Practice & Consulting from Bespin Global
WE ARE HIRING!!!!• KOREA• CHINA• JAPAN• USAPLEASE JOIN US
감사합니다!
HELPING YOU ADOPT CLOUD.