Streaming platform Kafka in SK planet

107
Streaming Platform Apache Kafka Data Platform 본부 Data Infrastructure강병수

Transcript of Streaming platform Kafka in SK planet

Page 1: Streaming platform Kafka in SK planet

Streaming Platform Apache Kafka

Data Platform 본부

Data Infrastructure팀

강병수

Page 2: Streaming platform Kafka in SK planet

Intro

Data Infrastructure팀의 Kafka

Kafka Summit 2017 NY

적용기

Page 3: Streaming platform Kafka in SK planet

Intro

Page 4: Streaming platform Kafka in SK planet
Page 5: Streaming platform Kafka in SK planet
Page 6: Streaming platform Kafka in SK planet
Page 7: Streaming platform Kafka in SK planet
Page 8: Streaming platform Kafka in SK planet
Page 9: Streaming platform Kafka in SK planet

상품의구매전환율을실시간통계로뽑고싶어요

광고비용을실시간으로집계하고싶어요

현재접속해있는유저상태에따라마케팅을하고싶어요

현재상품의품절여부를알고싶어요

Page 10: Streaming platform Kafka in SK planet

사업적니즈들을기술적으로어떻게서포팅해야할지고민이늘어난다.

외래문물을수용하러...

Page 11: Streaming platform Kafka in SK planet

Systems Track Streams Track Pipelines Track

Data Processing at LinkedIn with Apache Kafka Portable Streaming Pipelines with Apache Beam Capture the Streams of Database Changes

Kafka in the Enterprise: What if it Fails? The Best Thing Since Partitioned Bread :

Rethinking Stream Processing with Apache

Kafka’s new Streams API

Billions of Messages a Day - Yelp’s Real-time Data

Pipeline

Apache Kafka Core Internals : A Deep Dive Microservices with Kafka : An Introduction to Kafka

Streams with a Real-Life Example

California Schemin! How the Schema Registry has

Ancestry Basking in Data

Simplifying Omni-Channel Retail at Scale Hanging Out with Your Past Self in VR: Time-

Shifted Avatar Replication Using Kafka Streams

Achieving Predictability And Compliance With The

Data Distribution Hub at Bank of New York Mellon

How to Lock Down Apache Kafka and Keep Your

Streams Safe

Building Advanced Streaming Applications using

the Latest from Apache Flink and Kafka

Every Message Counts: Kafka as Foundation for

Highly Reliable Logging at Airbnb

Running Hundreds of Kafka Clusters with 5 People The Data Dichotomy: Rethinking Data & Services

with Streams

The Source of Truth: Why the New York Times

Stores Every Piece of Content Ever Published in

Kafka

Hardening Kafka for New Use Cases with Venice Easy, Scalable, Fault-tolerant Stream Processing

with Kafka and Spark’s Structured Streaming

Single Message Transformations Are Not the

Transformations You’re Looking For

Introducing Exactly Once Semantics in Apache

Kafka

Scalable Real-time Complex Event Processing at

Uber

Cloud Native Data Streaming Microservices with

Spring Cloud and Kafka

Page 12: Streaming platform Kafka in SK planet

Systems Track Streams Track Pipelines Track

Data Processing at LinkedIn with Apache Kafka Portable Streaming Pipelines with Apache Beam Capture the Streams of Database Changes

Kafka in the Enterprise: What if it Fails? The Best Thing Since Partitioned Bread :

Rethinking Stream Processing with Apache

Kafka’s new Streams API

Billions of Messages a Day - Yelp’s Real-time Data

Pipeline

Apache Kafka Core Internals : A Deep Dive Microservices with Kafka : An Introduction to Kafka

Streams with a Real-Life Example

California Schemin! How the Schema Registry has

Ancestry Basking in Data

Simplifying Omni-Channel Retail at Scale Hanging Out with Your Past Self in VR: Time-

Shifted Avatar Replication Using Kafka Streams

Achieving Predictability And Compliance With The

Data Distribution Hub at Bank of New York Mellon

How to Lock Down Apache Kafka and Keep Your

Streams Safe

Building Advanced Streaming Applications using

the Latest from Apache Flink and Kafka

Every Message Counts: Kafka as Foundation for

Highly Reliable Logging at Airbnb

Running Hundreds of Kafka Clusters with 5 People The Data Dichotomy: Rethinking Data & Services

with Streams

The Source of Truth: Why the New York Times

Stores Every Piece of Content Ever Published in

Kafka

Hardening Kafka for New Use Cases with Venice Easy, Scalable, Fault-tolerant Stream Processing

with Kafka and Spark’s Structured Streaming

Single Message Transformations Are Not the

Transformations You’re Looking For

Introducing Exactly Once Semantics in Apache

Kafka

Scalable Real-time Complex Event Processing at

Uber

Cloud Native Data Streaming Microservices with

Spring Cloud and Kafka

서비스 Database를실시간입수하는방법들

Kafka를이용한 Microservice

Kafka Streams 소개

그외에도.. 안정적으로 System을운영하는사례들이나,

하둡데이터와실시간이벤트를함께활용하는방법들

Page 13: Streaming platform Kafka in SK planet

자세한내용과팀에어떻게적용하고있는지는뒤에서얘기하고,

먼저, Kafka를모르시는분들에위해서간단하게설명을하려고합니다.

Page 14: Streaming platform Kafka in SK planet

Apache Kafka

publish / subscribe

distributed system

stable store

low latency

high throughput

real-time processing

Page 15: Streaming platform Kafka in SK planet

Apache Kafka

Producer

BrokerProducer

Producer

Consumer

Consumer

Consumer

Page 16: Streaming platform Kafka in SK planet

Apache Kafka

BrokerProducertopic : syrup

data : 캣츠

Page 17: Streaming platform Kafka in SK planet

Apache Kafka

BrokerProducer

Disk

topic : syrup

data : 캣츠

Page 18: Streaming platform Kafka in SK planet

Apache Kafka

BrokerProducer Consumer

Disk

topic : syrup

data : 캣츠topic : syrup

data : 캣츠

Page 19: Streaming platform Kafka in SK planet

Apache Kafka

Producer

BrokerProducer Consumer

topic : 11st

data : 전자담배

topic : syrup

data : 캣츠topic : syrup

data : 캣츠

Disk

Page 20: Streaming platform Kafka in SK planet

Apache Kafka

Producer

BrokerProducer

Consumer

Consumer

Disk

topic : syrup

data : 캣츠topic : syrup

data : 캣츠

topic : 11st

data : 전자담배topic : 11st

data : 전자담배

Page 21: Streaming platform Kafka in SK planet

Apache Kafka

Producer

BrokerProducer

Consumer

Consumer

Consumer

topic : 11st

data : 전자담배

topic : 11st

data : 전자담배

Disk

topic : syrup

data : 캣츠topic : syrup

data : 캣츠

topic : 11st

data : 전자담배

Page 22: Streaming platform Kafka in SK planet

Apache Kafka

Producer

BrokerProducer

Consumer

Consumer

ConsumerProducer

topic : OCB

data : 500p

Disk

topic : syrup

data : 캣츠topic : syrup

data : 캣츠

topic : 11st

data : 전자담배topic : 11st

data : 전자담배

topic : 11st

data : 전자담배

Page 23: Streaming platform Kafka in SK planet

Apache Kafka

변경이불가능한이벤트로그를스트림으로처리한다.

Producer 와 Consumer는의존도가없다.

Disk를사용하기때문에안정적이고, 잘못됐을때돌아올수있다.

millisecond 단위의 low latency를보장한다.

Page 24: Streaming platform Kafka in SK planet

Disk를사용하는데어떻게빠르지?운영체제는 Disk I/O 효율을위하여 Page Cache라는것을사용한다.

Page 25: Streaming platform Kafka in SK planet

Apache Kafka

메모리사용상태를보면 190GB중 165GB를 Page Cache로사용

Page 26: Streaming platform Kafka in SK planet

Apache Kafka

CPU IO wait 역시거의발생하지않음

Page 27: Streaming platform Kafka in SK planet

ProcessingKafka Streams를이용하여 ETL이가능하다.

StoreDisk를이용하는안정적인저장소다.

Pub / Sub자유롭게데이터를넣고땡길수있다.

Page 28: Streaming platform Kafka in SK planet

Data Infrastructure팀의 Kafka

Page 29: Streaming platform Kafka in SK planet

App log

Web log

Server log

DB data

Data Play를하기위해서...

Page 30: Streaming platform Kafka in SK planet

App log

Web log

Server log

DB data

원시시대

FTP

Page 31: Streaming platform Kafka in SK planet

FTP로전송중에실패하면?하둡에전송된일부를지우고재전송한다.

제시간에 File이생성되지않으면?cron이동작하지않을테니손으로다시돌린다..

App이나 Web같은실시간로그는어떻게?무엇인가대책이필요하다.

Page 32: Streaming platform Kafka in SK planet

App log

Web log

Server log

DB data

2011년에 Kafka를도입

Page 33: Streaming platform Kafka in SK planet

전송중실패부분부터재전송가능

파일로뽑고읽어서보낼것없이이벤트가발생하면즉시전송

App, Web Client 로그도이벤트건 by 건적재가능

현재는대부분의로그를 Kafka를통해입수한다.

Page 34: Streaming platform Kafka in SK planet

38000건 / Sec 33억건 / day

IN : 92 MB / Sec 8TB / day

OUT : 310 MB / Sec 25TB / day

현재 Main Cluster는평균적으로

Page 35: Streaming platform Kafka in SK planet

Data Infrastructure팀의 Kafka

Recopick

3ea

Global

11st

5ea

11st

3ea

Main

10ea

Dev

5ea

Mirror

5ea

커머스

AWS

보라매

커머스

일산

MirrorMaker

MirrorMaker

MirrorMaker

Page 36: Streaming platform Kafka in SK planet

Data Infrastructure팀의 Kafka

Recopick

3ea

Global

11st

5ea

11st

3ea

Main

10ea

Dev

5ea

Mirror

5ea

커머스

AWS

보라매

커머스

일산

서비스로그

MirrorMaker

MirrorMaker

MirrorMaker

실시간추천

Hadoop

Monitoring 검색인덱싱

서비스로그

서비스로그

서비스로그

CEP

Page 37: Streaming platform Kafka in SK planet

클러스터들을혼자구성및운영하려니고충이많이생김

Page 38: Streaming platform Kafka in SK planet

IDC 및클러스터별로입수상태및현황을파악해야 했고,

문제가생겼을때어디쪽의문제인지추적이되어야했고,

장애시복구포인트나복구는잘되는지를알아야한다.

또한언제어떤 Topic이생겼는지등의이력관리가되어야하고,

문제제기가들어왔을때증명도해야한다...

Page 39: Streaming platform Kafka in SK planet
Page 40: Streaming platform Kafka in SK planet

Data Infrastructure팀의 Kafka

운영상편의를위하여.... IDC 통합모니터링개발

Page 41: Streaming platform Kafka in SK planet

Data Infrastructure팀의 Kafka

Kafka 상태모니터

Page 42: Streaming platform Kafka in SK planet

Data Infrastructure팀의 Kafka

Zookeeper 데이터수집

Page 43: Streaming platform Kafka in SK planet

Data Infrastructure팀의 Kafka

통합 Transfer

Page 44: Streaming platform Kafka in SK planet

운영이편해졌다. 하지만...

Page 45: Streaming platform Kafka in SK planet

서비스 DB를실시간으로어떻게입수하지?

real-time streaming 환경은되어있는데실제서비스는몇없다….

MicroService를 Kafka와함께쓰는게추세라던데?

실시간으로통계를뽑으려면 Hadoop으로는한계가있지..

Page 46: Streaming platform Kafka in SK planet

고민해결의꿈을안고 Kafka Summit 2017 NY

Page 47: Streaming platform Kafka in SK planet

Kafka Streams

Page 48: Streaming platform Kafka in SK planet

Kafka Streams

Kafka project의 streaming library

2016년에 release

기존의다른 Streaming Framework에비하여큰이점들이있다.

그러나, 나온지가얼마안되서증명이안됐다.

Page 49: Streaming platform Kafka in SK planet

Kafka Streams

눈부신성공사례로어느정도증명이...

Line corps. 내부파이프라인에 Kafka Streams적용하기https://engineering.linecorp.com/en/blog/detail/80

Peak time : 100만메시지 / sec

Page 50: Streaming platform Kafka in SK planet

Kafka Consumer

BrokerProducertopic : 11st

data : 전자담배data : 화분

Page 51: Streaming platform Kafka in SK planet

Kafka Consumer

Broker

Consumerconsumergroup : group-1

topic : 11st

data : 전자담배

Consumer

consumergroup : group-1

topic : 11st

data : 화분

Producertopic : 11st

data : 전자담배data : 화분

Page 52: Streaming platform Kafka in SK planet

Kafka Consumer

Broker

Consumerconsumergroup : group-1

topic : 11st

data : 전자담배

Consumer

consumergroup : group-1

topic : 11st

data : 화분

Producertopic : 11st

data : 전자담배data : 화분

Scalability

Load balancing

Page 53: Streaming platform Kafka in SK planet

Kafka Consumer

BrokerProducertopic : 11st

data : 자전거data : 전동킥보드

Page 54: Streaming platform Kafka in SK planet

Kafka Consumer

Broker

Consumerconsumergroup : group-1

topic : 11st

Consumer

consumergroup : group-1

topic : 11st

data : 자전거전동킥보드

Producertopic : 11st

data : 자전거data : 전동킥보드

Page 55: Streaming platform Kafka in SK planet

Kafka Consumer

Broker

Consumerconsumergroup : group-1

topic : 11st

Consumer

consumergroup : group-1

topic : 11st

data : 자전거전동킥보드

Producertopic : 11st

data : 자전거data : 전동킥보드

High Availability

Page 56: Streaming platform Kafka in SK planet

Kafka Streams

Consumer, Producer

State

1m

2m

3m

4m

Join

Time Window

Page 57: Streaming platform Kafka in SK planet

Kafka Streams

비슷한역할을수행하던것들

Page 58: Streaming platform Kafka in SK planet

Kafka Streams

Kafka Streams의장점은?

Page 59: Streaming platform Kafka in SK planet

Kafka Streams

Kafka Streams의장점은?

라이브러리이다.

Yarn과같은 resource manager가필요없다.

오직데몬을돌릴수있는서버만있으면된다.

소스가가볍다.

H/A, Scalability, Load Balancing 을구현할필요가없다. Kafka에서다해준다.

생산성이좋아진다.

디버깅이쉽다. 비즈니스로직에만집중해서개발을하면된다.

Page 60: Streaming platform Kafka in SK planet

Kafka Streams로쉽고좋은 Streaming Application을만들수있다.

Page 61: Streaming platform Kafka in SK planet

Microservice

Page 62: Streaming platform Kafka in SK planet

하나의레파지토리에 100만라인의코드였다.

모노리딕앱 : 옐프메인2011

2013

2017

Kafka를백본으로, 마이크로서비스로옮겼다.

70개이상의 production 서비스였다.

2013년보다 R&D에 100만달러를아끼게됐다.

코드의복잡성이줄어들고, downtime이줄어들었다.

Page 63: Streaming platform Kafka in SK planet

Monolithic service Micro service

Page 64: Streaming platform Kafka in SK planet

서비스끼리통신이발생한다.

Page 65: Streaming platform Kafka in SK planet

서비스는필연적으로복잡해진다.

Page 66: Streaming platform Kafka in SK planet

서비스하나의장애가다른서비스에도영향을준다

Page 67: Streaming platform Kafka in SK planet

시스템의복잡도를줄이고 Coupling을줄일수없을까?

Page 68: Streaming platform Kafka in SK planet

Kafka를 Backbone으로활용하면해결할수있다.

Page 69: Streaming platform Kafka in SK planet

Asynchronous하게 데이터를던지고받는다

A B

DC

E

B - getItem(1) A - 수영복C - 양말, 물안경, 수영복….

B - top10

Page 70: Streaming platform Kafka in SK planet

Asynchronous하게 데이터를던지고받는다

A B

DC

E

A - 수영복

C - 양말, 물안경, 수영복,

…...

Page 71: Streaming platform Kafka in SK planet

하나의서비스에장애가나도다른서비스는모른다

Page 72: Streaming platform Kafka in SK planet

또한복구가됐을때, 처리못한 request를모두불러올수있다.

Page 73: Streaming platform Kafka in SK planet

Message Queue는서비스의복잡도를확연하게줄여준다.

Publish / Subscribe 구조는 Coupling을줄여준다.

Page 74: Streaming platform Kafka in SK planet

Kafka를이용한 Microservice를어떻게만들수있을까?

Page 75: Streaming platform Kafka in SK planet

출처 : 넷플릭스 마이크로서비스 가이드https://www.youtube.com/watch?feature=youtu.be&v=OczG5FQIcXw&app=desktop

Microservice의단위는?

Page 76: Streaming platform Kafka in SK planet

Consumer, Producer

State

1m

2m

3m

4m

Join

Time Window

Kafka Streams를이용해서만들수있다.

Page 77: Streaming platform Kafka in SK planet

Kafka에서 Data 입출력이쉽고,

Scalability,

Load Balancing,

H/A 구성

Page 78: Streaming platform Kafka in SK planet

Spring Cloud의 Kafka binder를사용해서쉽게만들수있다.

Page 79: Streaming platform Kafka in SK planet

Kafka를사용하면 Microservice의복잡한문제들을꽤많이해결해준다.

Page 80: Streaming platform Kafka in SK planet

Change Data Capture

Page 81: Streaming platform Kafka in SK planet

11번가상품 DB

11번가주문 DB

Syrup

유저 DB

서비스 DB

Page 82: Streaming platform Kafka in SK planet

11번가상품 DB

11번가주문 DB

Syrup

유저 DB

서비스 DB

Data Application

Data Analysis

Page 83: Streaming platform Kafka in SK planet

서비스 DB는 Kafka를통해적재를못하고있다.

Page 84: Streaming platform Kafka in SK planet

PK Data

A 2

B 2

결과

실시간트랜잭션Update A 1 -> 2

Delete C

Insert B 1

Update B 2

DB

Event Log들과다르게 DB는최종변경내용이저장된다.

Page 85: Streaming platform Kafka in SK planet

/data/test

A 1

A 2

B 1

B 2

Hadoop

실시간트랜잭션Update A 1 -> 2

Delete C

Insert B 1

Update B 2

DB

Hadoop은상태변경을처리하기에적합하지않다.

Page 86: Streaming platform Kafka in SK planet

RDB Hadoop

매일또는매시간Full dump batch

그래서우리는

Page 87: Streaming platform Kafka in SK planet

RDB Hadoop

Data Analysis

Data Application

매일또는매시간Full dump batch

Page 88: Streaming platform Kafka in SK planet

RDB Hadoop

최신상태가필요해..

Full dump에만 6시간이걸려…

File dump가없으면연락후재입수

매일또는매시간Full dump batch

Page 89: Streaming platform Kafka in SK planet

실시간으로데이터베이스변경분을반영할수없을까?

Page 90: Streaming platform Kafka in SK planet

Database의실시간트랜잭션로그는어떻게뽑지?

뽑을수있다면데이터를어떻게순서보장을하면서전송을할까?

Hadoop이아닌어떤 Repository를이용해야할까?

실시간상태를 Hadoop에서도볼수있어야하는데..

넘어야할산들

Page 91: Streaming platform Kafka in SK planet

우리에게도움을줄회사들..

Page 92: Streaming platform Kafka in SK planet

Database의실시간트랜잭션로그는어떻게뽑지?

넘어야할산들

Redo Log, Undo Log parser - Log Miner, OGG, XStreams etc...

Bin log parser - Open-Replicator Link : https://github.com/whitesock/open-replicator

Page 93: Streaming platform Kafka in SK planet

뽑을수있다면데이터를어떻게순서보장을하면서전송을할까?

넘어야할산들

실시간로그를입수하면서

유실없이

메시지간순서보장을할수있다.

Page 94: Streaming platform Kafka in SK planet

Hadoop이아닌어떤 Repository를이용해야할까?

넘어야할산들

Page 95: Streaming platform Kafka in SK planet
Page 96: Streaming platform Kafka in SK planet

실시간상태를 Hadoop에서도볼수있어야하는데..

넘어야할산들

Snapshot to HDFS

또는 Hive와연동

Page 97: Streaming platform Kafka in SK planet

RDB Hadoop

Change

Data

Capture

Page 98: Streaming platform Kafka in SK planet

RDB Hadoop

Change

Data

Capture

Data Application

Data Analysis

Page 99: Streaming platform Kafka in SK planet

Data 분석시, 또는 Data Application에서

실시간으로서비스 DB의상태를사용할수있다.

Page 100: Streaming platform Kafka in SK planet

이제다시 DI Cluster로돌아가보면

Page 101: Streaming platform Kafka in SK planet

상품의구매전환율을실시간통계로뽑고싶어요

광고비용을실시간으로집계하고싶어요

현재접속해있는유저상태에따라마케팅을하고싶어요

현재상품의품절여부를알고싶어요

Page 102: Streaming platform Kafka in SK planet

현재상태로돌아가보면

Sentinel

Rake

Server

Kafka

Hadoop Hive

Log

Agent

RakeAPI

Mirror

Maker

Collector

Other

Streaming

Apps

Page 103: Streaming platform Kafka in SK planet

Router

위내용들을적용하여...

Sentinel

Rake

Server

Kafka

Hadoop

Druid

Hbase

Redis

Hive

Log

Agent

RakeAPI

Mirror

Maker

RDB

Collector

Other

Streaming

Apps

실시간 OLAP

CDC

OGG

유저상태

Page 104: Streaming platform Kafka in SK planet
Page 105: Streaming platform Kafka in SK planet

사업적니즈를함께풀어가기위한노력들

유저가여러번클릭한상품에대해서즉시구매까지전환되도록할수없을까?

실시간통계를보면서수익이많이나는광고를내보내면좋을것같은데..

현재접속해있는유저를대상으로마케팅을하면어떨까?

Page 106: Streaming platform Kafka in SK planet

사업적니즈를함께풀어가기위한노력들

사업적요건들을 Data Infrastructure Cluster안에서해결해나갈수있도록

고민과노력중!

Page 107: Streaming platform Kafka in SK planet

함께데이터의활용가치를높여나가자