KafkaとAWS Kinesisの比較
16
-
Upload
yoshiyasu-saeki -
Category
Technology
-
view
2.349 -
download
0
Transcript of KafkaとAWS Kinesisの比較
• • Kafka Kinesis
• • •
•
2
• 2011/04
• 2015/09
• • Druid (2015; Kindle Direct Publishing) • RDB NoSQL (2016; ; HBase )
• • (WebDB Forum 2014)
• Spark Streaming (Spark Meetup
December 2015)
3
Kafka Kinesis
• Apache • Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis • Stream • Firehose • Analytics
: Pub/Sub : : (SQL)
4
Kafka Kinesis
• Apache • Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis • Stream • Firehose • Analytics
: Pub/Sub : : (SQL)
4
:
5
: <100ms
: ~8000msgs/sec
Kinesis
[Kafka] [Kinesis]
6
[Kafka] [Kinesis]
6
[Kafka] [Kinesis]
Kafka Connect
Kafka-rest Kafka-Pixy Kastle
AWS API Gateway
HTTP API
ETL ETL 7
[Kafka] [Kinesis]
Kafka Connect
Kafka-rest Kafka-Pixy Kastle
AWS API Gateway
HTTP API
ETL ETL
OSS
•Kafka Streams •PipelineDB
AWS
•Kinesis Analytics
7
[Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8
[Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8
[ ]
http://insightdataengineering.com/blog/ingestion-comparison/
X
Kafka Kinesis
• Kafka or Kinesis
• Kafka OSS
Kinesis
•
9
Kafka Kinesis
• (< 5000msgs/sec) Kinesis
• Kafka
• Kinesis
• Hadoop Kafka
• Kinesis
• Kafka
10
• OSS → Kafka
• → Kinesis
• API
• • SNS Bot IoT
•
11
