Amazon Kinesis Data Streams - 開発者ガイド · Amazon Kinesis Data Streams 開発者ガイド...
Transcript of Amazon Kinesis Data Streams - 開発者ガイド · Amazon Kinesis Data Streams 開発者ガイド...
-
Amazon Kinesis Data Streams
-
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams: Copyright 2018 Amazon Web Services, Inc. and/or its affiliates. All rights reserved.
Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any mannerthat is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks notowned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored byAmazon.
-
Amazon Kinesis Data Streams
Table of ContentsAmazon Kinesis Data Streams ...................................................................................................... 1
Kinesis Data Streams ....................................................................................................... 1Kinesis Data Streams ................................................................................................. 2 ............................................................................................................................. 2 ....................................................................................................................... 2
......................................................................................................... 2 .................................................................................................................................. 3
Data Streams ............................................................................................................................. 5Kinesis Data Stream ......................................................................... 5 ............................................................................................................... 6 ............................................................................................................... 6
.......................................................................................................................... 7 .......................................................................................................................... 8
........................................................................................................................................ 8 .......................................................................................................................................... 8
API ........................................................................................................................ 9 .................................................................................................................. 9
.......................................................................................................................................... 10 ............................................................................................................................ 10
AWS .................................................................................................. 10 ............................................................................... 11 .......................................................................................................... 11
: ............................................................................... 11Kinesis Data Streams ............................................. 12 .......................................................................................................................... 12 1: ................................................................. 12 2: ......................................... 13 3: ................................................................. 16 4: ................................................................................................... 16
: CLI ..................................................................................... 17AWS CLI ......................................................................................... 17 ............................................................................ 19
: ............................................................................ 24 .......................................................................................................................... 24 1: ................................................................................... 25 2: IAM .......................................................................... 26 3: .............................................................. 29 4: ................................................................................ 30 5: ................................................................................ 33 6: () .............................................................. 36 7: ......................................................................................................... 37
.................................................................................................................... 39 ...................................................................................................................... 39
Kinesis Data Streams ........................................................................... 39 ....................................................................................................... 40
.................................................................................................................. 41 .................................................................................................................. 42 ............................................................................................. 43 ............................................................................................................... 43 ............................................................................................. 44
........................................................................................... 44 ................................................................................................................. 452 ................................................................................................. 46 .......................................................................................................... 47
iii
-
Amazon Kinesis Data Streams
............................................................................................................... 49 ............................................................................................................... 49
....................................................................................................................... 50 ........................................................................................ 50 ....................................................................................................................... 50Kinesis Data Streams .......................................... 51AWS CLI .......................................................................... 51Kinesis Data Streams API ................................................... 52
......................................................................................................... 52CloudWatch ......................................................................... 52CloudWatch ................................................................... 62AWS CloudTrail Amazon Kinesis Data Streams API ...................... 63CloudWatch KCL ............................................................................. 66CloudWatch KPL .............................................................................. 74
......................................................................................................................... 78 .................................................................................................................... 79Kinesis Data Streams ..................................................................................... 80Kinesis Data Streams Amazon (ARN) ..................................................... 80Kinesis Data Streams ..................................................................................... 80
......................................................................................................... 82Kinesis Data Streams ............................................................... 82 ................................................ 83 ............................................................................ 84 KMS ...................................................................... 85 KMS .............................................. 86KMS .................................................................... 87
VPC .............................................................................. 87Kinesis Data Streams VPC ........................................... 87Kinesis Data Streams VPC .................................. 87 .............................................................................................. 88
....................................................................................... 88 ........................................................................................................ 90
KPL .............................................................................................................................. 90KPL ................................................................................................................... 91KPL .................................................................................................... 91KPL ................................................................................................ 92KPL .................................................................................................... 92Kinesis Producer Library Amazon Trust Services (ATS) ................................ 92KPL ........................................................................ 93KPL .................................................................................................... 93KPL .................................................................................... 95Kinesis data stream ....................................................................................... 96KPL ...................................................................................................................... 97 ................................................................................................. 98Kinesis Data Firehose KPL ............................................................................. 100
API ............................................................................................................................. 100 ............................................................................................. 101
................................................................................................................. 105 ........................................................................................................................ 105 ....................................................................... 106 ................................................................................................ 106 ......................................................................................................... 107 ..................................... 109 .................................................................... 110 CLI ............................................................................................... 113
........................................................................................................... 113 ..................................... 114
iv
-
Amazon Kinesis Data Streams
KMS .............................................................. 115 ....................................................................................................................... 115
......................................................................................................... 115KPL ................................................................................... 116
................................................................................................... 117 .............................................................................................................. 118
Kinesis Client Library 1.x ....................................................................................... 118Kinesis Client Library 2.0 ....................................................................................... 133API ..................................................................................................................... 137
............................................................................ 141Kinesis Client Library 2.0 ....................................................................................... 142API ..................................................................................................................... 146AWS ................................................................................ 147
Kinesis Client Library 1.x 2.x ................................................................................ 148 ................................................................................................ 149 ............................................................................. 152 ............................................................................................................... 153Amazon Kinesis ................................................................................. 154 ......................................................................................................... 156 ................................................................................................... 156
........................................................................................................... 157Kinesis Kinesis Data Streams .............................................................................................................................. 157 .......... 157 ..................................... 158GetRecords Records ......................... 158 .......................................................................... 159 ............................................................................. 159 KMS .............................................................. 160
....................................................................................................................... 160 ..................................................................................................................... 160 ......................................................................................................... 161Kinesis Producer Library AWS Lambda ............................................................. 162 ................................................................................... 162 ......................................................................................................... 163 ............................................................................................................... 165 .................................................................... 166
............................................................................................................................ 168AWS ................................................................................................................................ 170
v
-
Amazon Kinesis Data Streams Kinesis Data Streams
Amazon Kinesis Data Streams Amazon Kinesis DataStreams Kinesis Data Streams Kinesis Data Streams Kinesis Client Library Amazon EC2 AWS Kinesis Data Streams Amazon Kinesis DataStreams
Kinesis Data Streams Kinesis Data FirehoseKinesis Kinesis Data Analytics Kinesis
AWS AWS AWS ?
Kinesis Data Streams (p. 1) Kinesis Data Streams (p. 2) (p. 2) Kinesis Data Streams (p. 2) (p. 5) Kinesis Data Streams (p. 7) Kinesis Data Streams (p. 8) Kinesis Data Streams (p. 8)
Kinesis Data Streams Kinesis Data Streams IT
Kinesis Data Streams
KinesisData Streams
Kinesis Data Streams
1
streaming-data/kinesis/streams/kinesis/streams/big-data/streaming-data/
-
Amazon Kinesis Data Streams Kinesis Data Streams
KinesisData Streams
Kinesis Data Streams Directed Acyclic Graphs (DAG) Kinesis Data Streams Kinesis Data Streams
Kinesis Data Streams Kinesis Data Streams MapReduce
Kinesis () 1 Kinesis Data Streams Kinesis Data Streams MapReduce Kinesis Data Streams
Kinesis Data Streams 2 Amazon DynamoDB 2 Amazon Simple Storage Service (Amazon S3) DynamoDB
Kinesis Client Library KinesisData Streams
Amazon EMR Kinesis Kinesis
Kinesis Data Streams Amazon Kinesis Data Streams
Kinesis Data Streams Kinesis Data Streams Kinesis Data Streams (Amazon EC2 Amazon Kinesis Data Firehose
2
emr-kinesis.html
-
Amazon Kinesis Data Streams
) Amazon DynamoDBAmazon RedshiftAmazon S3 AWS
Kinesis Data Streams Kinesis Data StreamKinesis data stream (p. 4)Kinesis Data Streams (p. 4)
Kinesis data stream (p. 3) (p. 4) (p. 4) BLOB () Kinesis Data Streams BLOB BLOB 1 MB
24 IncreaseStreamRetentionPeriod 168 (7 ) DecreaseStreamRetentionPeriod 24 24 Amazon Kinesis Data Streams
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams Amazon Kinesis Data Streams Application (p. 4)
3
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_IncreaseStreamRetentionPeriod.htmlhttps://docs.aws.amazon.com/kinesis/latest/APIReference/API_DecreaseStreamRetentionPeriod.htmlhttps://aws.amazon.com/kinesis/pricing/
-
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams ApplicationAmazon Kinesis Data Streams application EC2
2 Amazon Kinesis Data Streams (p. 117)
Kinesis Data Streams AWS 1
1 1 5 1 2 MB 1 1,000 1 1 MB ()
Kinesis Data Streams 256 Unicode MD5 128
client.putRecords client.putRecord KinesisData Streams
Note
Kinesis Client LibraryKinesis Client Library Kinesis Client Library Kinesis ClientLibrary Amazon DynamoDB 1
4
-
Amazon Kinesis Data Streams Data Streams
Kinesis Client Library 2 Amazon Kinesis Data Streams (p. 117)
Amazon Kinesis Data Streams application AWS Amazon DynamoDB Amazon CloudWatch
Amazon Kinesis Data Streams Kinesis Data Streams AWS KMS (p. 82)
Note
the section called KMS (p. 86)Note
AWS Key Management Service (AWS KMS) AWS Key Management Service
Amazon Kinesis Data Streams Kinesis Data Streams
Kinesis Data Streams (p. 8)
(p. 7) (p. 8)
Kinesis Data Stream AWS UpdateShardCount API / Kinesis Data Streams
( KB ) (average_data_size_in_KB)
1 (records_per_second)
5
http:///kms/pricingAPI_UpdateShardCount.html
-
Amazon Kinesis Data Streams
() Kinesis Data Streams (number_of_consumers)
KB (incoming_write_bandwidth_in_KB)average_data_size_in_KB records_per_second
KB (outgoing_read_bandwidth_in_KB)incoming_write_bandwidth_in_KB number_of_consumers
(number_of_shards)
number_of_shards = max(incoming_write_bandwidth_in_KB/1000, outgoing_read_bandwidth_in_KB/2000)
Kinesis Data Streams Kinesis Data Streams API AWSCommand Line Interface (AWS CLI)
1. AWS https://console.aws.amazon.com/kinesis Kinesis
2. 3. [] 4. [Kinesis ] [Kinesis
]
[Kinesis ] [] [Creating] [] [Active]
5. []
Kinesis Data Streams API
Kinesis Data Streams API (p. 39)
AWS CLI
AWS CLI create-stream
Kinesis Data Streams Kinesis Data Streams API AWS CLI
Note
6
https://console.aws.amazon.com/kinesiskinesis/create-stream.html
-
Amazon Kinesis Data Streams
1. https://console.aws.amazon.com/kinesis/ Amazon Kinesis 2. 3. []
4. [] []
5. [] []
KMS Kinesis aws/kinesis AWS KMS AWS KMS AWS KMS thesection called KMS (p. 86)
6. [] []
7. [] [] the section calledCloudWatch (p. 52)
API API
AddTagsToStream DecreaseStreamRetentionPeriod DisableEnhancedMonitoring EnableEnhancedMonitoring IncreaseStreamRetentionPeriod RemoveTagsFromStream StartStreamEncryption StopStreamEncryption UpdateShardCount
AWS CLI AWS CLI Kinesis CLI
Kinesis Data Streams Amazon Kinesis Kinesis data stream (p. 8)
Important
Kinesis Data Streams (p. 49)
7
kinesis/API_AddTagsToStream.htmlAPI_DecreaseStreamRetentionPeriod.htmlAPI_DisableEnhancedMonitoring.htmlAPI_EnableEnhancedMonitoring.htmlAPI_IncreaseStreamRetentionPeriod.htmlAPI_RemoveTagsFromStream.htmlAPI_StartStreamEncryption.htmlAPI_StopStreamEncryption.htmlAPI_UpdateShardCount.htmlkinesis/index.html
-
Amazon Kinesis Data Streams
BLOB
(p. 101) (Java )Kinesis DataStreams API PutRecords PutRecord put-record
Kinesis Data Streams Amazon Kinesis Data Streams application Kinesis
Amazon Simple Storage Service (Amazon S3)Amazon RedshiftAmazon Elasticsearch Service(Amazon ES)Splunk Kinesis Data Firehose Kinesis Data Firehose Amazon Kinesis Firehose Amazon Kinesis Data Streams (p. 117)
Amazon Machine Image (AMI) Amazon EC2 Auto Scaling Amazon EC2 Auto Scaling EC2 Auto Scaling EC2 AutoScaling CPU EC2 Amazon EC2 Auto Scaling
Kinesis Data Streams Amazon Kinesis Data Streams
1 1 1 MiB ()
1 1,000 5,000 1 5 GiB 500 AWS UpdateShardCount API
AWS () () 500 200
Base64 1 MiB
8
API_PutRecords.htmlAPI_PutRecord.htmlkinesis/put-record.htmlbasic-create.htmlhttps://docs.aws.amazon.com/autoscaling/ec2/userguide/API_UpdateShardCount.html
-
Amazon Kinesis Data Streams API
GetRecords 1 10 MiB 10,000GetRecords 1
1 5 10,000 10 MiB
GetRecords 2 MiB/GetRecords 10 MiB 5
API AWS API Kinesis Data Streams API API Amazon Kinesis API ReferenceAPI
API
1. AWS () 2. Kinesis Data Streams
9
API_GetRecords.htmlAPI_GetRecords.htmlhttps://docs.aws.amazon.com/kinesis/latest/APIReference/https://console.aws.amazon.com/support/v1#/case/create%3FissueType=service-limit-increase%26limitType=service-code-kinesis
-
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams Kinesis Data StreamsAmazon Kinesis Data Streams (p. 1)
Amazon Kinesis Data Streams (p. 10) : Amazon Kinesis Data Streams (p. 11) : AWS CLI Amazon Kinesis Data Streams (p. 17) : Kinesis Data Streams (p. 24)
Amazon Kinesis Data Streams Amazon Kinesis Data Streams
AWS (p. 10) (p. 11) (p. 11)
AWS (AWS) AWS AWS (Kinesis Data Streams )
AWS AWS
AWS
1. https://aws.amazon.com/ [AWS ]
Note
AWS AWS [Sign in to a different account ()] IAM [Sign-in using root account credentials ()] [ AWS ]
2.
10
-
Amazon Kinesis Data Streams
Kinesis Data Streams
Amazon Kinesis API Reference Kinesis Data Streams Java Amazon Kinesis Data Streams API AWS SDK for Java (p. 100) AWS SDK for Java Kinesis Data Streams API (p. 137) (p. 39)
GoJavaJavaScript.NETNode.jsPHPPython Ruby AWS SDK KinesisData Streams AWS SDK for Java Kinesis Data Streams GitHub
Kinesis Client Library (KCL) KCL Kinesis Data Streams JavaNode.js.NETPythonRuby (p. 117)
AWS Command Line InterfaceKinesis Data Streams AWS CLI AWS
KCL Java
Java 1.7 (Java SE 7 JDK) Java Oracle Java SE
Apache Commons (HTTP ) Jackson JSON
AWS SDK for Java Apache Commons Jackson SDK for Java Java 1.6 Kinesis Client Library Java 1.7
: Amazon Kinesis Data Streams
Amazon Kinesis Data Streams KinesisData Streams (p. 5) (p. 7) (p. 8) (Amazon Kinesis Data Streams (p. 1))
JavaScript N DynamoDB
11
https://docs.aws.amazon.com/kinesis/latest/APIReference/https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/developers/getting-started/java/sdkforbrowser/developers/getting-started/net/developers/getting-started/nodejs/developers/getting-started/php/https://github.com/boto/botodevelopers/getting-started/ruby/https://github.com/aws/aws-sdk-java/tree/master/src/sampleshttp://www.oracle.com/technetwork/java/javase/downloads/index.htmlhttp://www.oracle.com/technetwork/java/javase/downloads/index.htmlsdkforjava/
-
Amazon Kinesis Data Streams Kinesis Data Streams
Kinesis Data Streams Kinesis Data Streams URL Kinesis data stream URL
10 N 3 2
AWS CloudFormation AWS CloudFormation AWS Amazon EC2 Amazon DynamoDB
Note
Kinesis Data Streams AWS AWS 3: (p. 16)
Kinesis Data Streams
AWS (p. 5) (p. 7) (p. 8)
1: AWS AWS CloudFormation Kinesis data stream HTTPS
1. AWS CloudFormation 2. [] URL []
12
cloudformation/freecloudformation/home?region=us-east-1#/stacks/new?stackName=KinesisDataVisSampleApp&templateURL=https:%2F%2Fs3.amazonaws.com%2Fkinesis-demo-bucket%2Famazon-kinesis-data-visualization-sample%2Fkinesis-data-vis-sample-app.template
-
Amazon Kinesis Data Streams 2:
3. [Specify Details ()] t2.micro T2 VPC AWS VPC [InstanceType] m3.medium []
4. [] EC2 []
5. [Review ()] [I acknowledge that this template might cause AWS CloudFormation tocreate IAM resources ( AWS CloudFormation IAM )] [Create ()]
CREATE_IN_PROGRESS KinesisDataVisSample CREATE_COMPLETE
2:
Kinesis Data Stream (p. 13) (p. 14) (p. 15)
Kinesis Data Stream (p. 5) 1
AWS CloudFormation AWSCloudFormation CreateStream
1. [KinesisDataVisSample] 2. [Outputs ()] URL URL http://ec2-xx-xx-xx-
xx.compute-1.amazonaws.com3. 10
[Kinesis Data Streams Data VisualizationSample] 10 URL 1 2
13
https://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/master/src/main/static-content/cloudformation/kinesis-data-vis-sample-app.template#L74-80API_CreateStream.html
-
Amazon Kinesis Data Streams 2:
1. https://console.aws.amazon.com/kinesis Kinesis 2. ( KinesisDataVisSampleApp-KinesisStream-[randomString])
3. 4.
(p. 7) Kinesis data stream PutRecord
PutRecord
14
https://console.aws.amazon.com/kinesis/API_PutRecord.html
-
Amazon Kinesis Data Streams 2:
6 URL 2 URL HttpReferrerKinesisPutter 99132 Kinesis Data Streams 3 PutRecord pair.getResource HttpReferrerStreamWriter 85 92 6 URL
Kinesis Data Streams EC2 EC2 EC2 EC2
1. https://console.aws.amazon.com/ec2/) Amazon EC2 2. [] 3.
KinesisDataVisSample
4. []
(p. 8) Kinesis data stream GetShardIterator GetRecords
ID
() DynamoDB DynamoDB
2 URL N N 3 2 Kinesis Data Streams DynamoDB
Amazon DynamoDB
1. https://console.aws.amazon.com/dynamodb/ DynamoDB 2. [Tables ()] 3. 2
15
https://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/master/src/main/java/com/amazonaws/services/kinesis/samples/datavis/producer/HttpReferrerKinesisPutter.java#L99-132https://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/d7fbcb994caad606635414b01c4eeada15c04f0b/src/main/java/com/amazonaws/services/kinesis/samples/datavis/HttpReferrerStreamWriter.java#L85-92https://console.aws.amazon.com/ec2//API_GetShardIterator.html/API_GetRecords.htmlhttps://console.aws.amazon.com/dynamodb/
-
Amazon Kinesis Data Streams 3:
KinesisDataVisSampleApp-KCLDynamoDBTable-[randomString] KinesisDataVisSampleApp-CountsDynamoDBTable-[randomString]
N 4. Select the KinesisDataVisSampleApp-KCLDynamoDBTable-[randomString] 2
(leaseKey) (checkpoint) (leaseOwner)
5. Select the KinesisDataVisSampleApp-CountsDynamoDBTable-[randomString] (referrerCounts)
Kinesis (KCL)
Kinesis Client Library (KCL) KCL KCL
KCL KCL GetShardIterator IRecordProcessor KCL KCL KCL GetRecords IRecordProcessor
HttpReferrerCounterApplication 9298 KCL KCL
HttpReferrerCounterApplication 104108 IRecordProcessor KCL KCL
CountingRecordProcessor 186 203 IRecordProcessor N
3: 2 AWS AWS CloudFormation
1. https://console.aws.amazon.com/cloudformation AWS CloudFormation 2. 3. [][Delete Stack ()] 4. [Yes, Delete]
AWS CloudFormation [DELETE_IN_PROGRESS] AWS CloudFormation
4: GitHub
16
https://github.com/awslabs/amazon-kinesis-clienthttps://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/master/src/main/java/com/amazonaws/services/kinesis/samples/datavis/HttpReferrerCounterApplication.java#L92-98https://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/master/src/main/java/com/amazonaws/services/kinesis/samples/datavis/HttpReferrerCounterApplication.java#L104-108https://github.com/awslabs/amazon-kinesis-data-visualization-sample/blob/d7fbcb994caad606635414b01c4eeada15c04f0b/src/main/java/com/amazonaws/services/kinesis/samples/datavis/kcl/CountingRecordProcessor.java#L186-203https://console.aws.amazon.com/cloudformation/https://github.com/awslabs/amazon-kinesis-data-visualization-sample
-
Amazon Kinesis Data Streams : CLI
Kinesis Data Streams API Amazon Kinesis Data StreamsAPI AWS SDK for Java (p. 100)AWS SDK for Java Kinesis Data Streams API (p. 137) (p. 39)
SDK Kinesis Data Streams AWS SDKfor Java
: AWS CLI AmazonKinesis Data Streams
AWS Command Line Interface Amazon Kinesis DataStreams Kinesis Data Streams Kinesis data stream
For CLI access, you need an access key ID and secret access key. Use IAM user access keys instead ofAWS access keys. IAM lets you securely control access to AWS services andresources in your AWS account. For more information about creating access keys, see Understanding andGetting Your Security Credentials in the AWS General Reference.
IAM IAM
() Kinesis Data Streams
AWS CLI (p. 17) (p. 19)
AWS CLI AWS CLI Windows LinuxOS XUnix AWS CLI
Windows
1. AWS Command Line Interface Windows MSI
2. MSI 3.
Linux, macOS, or Unix
Python 2.6.5 AWS Command Line Interface
1. pip
17
developers/getting-started/java/developers/getting-started/java/https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.htmlhttps://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.htmlget-set-up-for-amazon-ec2.html#create-an-iam-user#ak_region#ak_regioninstalling.htmlinstalling.htmlhttps://pip.pypa.io/en/latest/installing.html
-
Amazon Kinesis Data Streams AWS CLI
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"sudo python get-pip.py
2. pip AWS CLI
sudo pip install awscli
aws help
Kinesis Data Streams Kinesis Data Streams AWS CLI
aws kinesis help
Kinesis Data Streams
AVAILABLE COMMANDS
o add-tags-to-stream
o create-stream
o delete-stream
o describe-stream
o get-records
o get-shard-iterator
o help
o list-streams
o list-tags-for-stream
o merge-shards
o put-record
o put-records
o remove-tags-from-stream
o split-shard
o wait
Amazon Kinesis API Kinesis DataStreams API create-stream CreateStream API
AWS CLI
18
-
Amazon Kinesis Data Streams
AWS CLI aws configure AWS CLI AWS CLI 1
aws configureAWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLEAWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYDefault region name [None]: us-west-2Default output format [None]: json
AWS CLI 4 AWS ID AWS
Note
AWS CLI AWS
JSONtexttable JSON
aws configure AWSCommand Line Interface AWS
AWS CLI Kinesis data stream Kinesis Data Streams (p. 2): AmazonKinesis Data Streams (p. 11)
Note
Kinesis Data Streams AWS Kinesis Data Streams AWS 4: (p. 23)
1: (p. 19) 2: (p. 21) 3: (p. 21) 4: (p. 23)
1: Foo
19
cli-chap-getting-set-up.html#cli-signupcli-chap-getting-set-up.html#cli-signuprande.htmlcli-chap-getting-started.html
-
Amazon Kinesis Data Streams
aws kinesis create-stream --stream-name Foo --shard-count 1
--shard-count 1
aws kinesis describe-stream --stream-name Foo
{ "StreamDescription": { "StreamStatus": "CREATING", "StreamName": "Foo", "StreamARN": "arn:aws:kinesis:us-west-2:account-id:stream/Foo", "Shards": [] }}
CREATING
{ "StreamDescription": { "StreamStatus": "ACTIVE", "StreamName": "Foo", "StreamARN": "arn:aws:kinesis:us-west-2:account-id:stream/Foo", "Shards": [ { "ShardId": "shardId-000000000000", "HashKeyRange": { "EndingHashKey": "170141183460469231731687303715884105727", "StartingHashKey": "0" }, "SequenceNumberRange": { "StartingSequenceNumber": "49546986683135544286507457935754639466300920667981217794" } } ] }}
"StreamStatus": "ACTIVE" list-streams
aws kinesis list-streams
:
{ "StreamNames": [ "Foo" ]}
20
-
Amazon Kinesis Data Streams
2: put-record "testdata"
aws kinesis put-record --stream-name Foo --partition-key 123 --data testdata
{ "ShardId": "shardId-000000000000", "SequenceNumber": "49546986683135544286507457936321625675700192471156785154"}
3: ( get-record ) get-shard-iterator
aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name Foo
aws kinesis Kinesis Data Streams API GetShardIterator API ()
{ "ShardIterator": "AAAAAAAAAAHSywljv0zEgPX4NyKdZ5wryMzP9yALs8NeKbUjp1IxtZs1Sp+KEd9I6AJ9ZG4lNR1EMi+9Md/nHvtLyxpfhEzYvkTZ4D9DQVz/mBYWRO6OTZRKnW9gd+efGN2aHFdkH1rJl4BL9Wyrk+ghYG22D2T1Da2EyNSH1+LAbK33gQweTJADBdyMwlo5r6PqcP2dzhg="}
() get 300 get-shard-iterator
get-records Kinesis Data Streams API GetRecordsGetRecords
get-records ()
aws kinesis get-records --shard-iterator AAAAAAAAAAHSywljv0zEgPX4NyKdZ5wryMzP9yALs8NeKbUjp1IxtZs1Sp+KEd9I6AJ9ZG4lNR1EMi+9Md/nHvtLyxpfhEzYvkTZ4D9DQVz/mBYWRO6OTZRKnW9gd+efGN2aHFdkH1rJl4BL9Wyrk+ghYG22D2T1Da2EyNSH1+LAbK33gQweTJADBdyMwlo5r6PqcP2dzhg=
21
API_GetShardIterator.htmlAPI_GetRecords.html
-
Amazon Kinesis Data Streams
bash Unix ()
SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name Foo --query 'ShardIterator')
aws kinesis get-records --shard-iterator $SHARD_ITERATOR
PowerShell ()
aws kinesis get-records --shard-iterator ((aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name Foo).split('"')[4])
get-records ()
{ "Records":[ { "Data":"dGVzdGRhdGE=", "PartitionKey":"123, "ApproximateArrivalTimestamp": 1.441215410867E9, "SequenceNumber":"49544985256907370027570885864065577703022652638596431874" } ], "MillisBehindLatest":24000, "NextShardIterator":"AAAAAAAAAAEDOW3ugseWPE4503kqN1yN1UaodY8unE0sYslMUmC6lX9hlig5+t4RtZM0/tALfiI4QGjunVgJvQsjxjh2aLyxaAaPr+LaoENQ7eVs4EdYXgKyThTZGPcca2fVXYJWL3yafv9dsDwsYVedI66dbMZFC8rPMWc797zxQkv4pSKvPOZvrUIudb8UkH3VMzx58Is="}
get-records ()
testdata put-record Base64 AWS CLI Kinesis Data Streams Base64 Base64 raw stdout Base64 (https://www.base64decode.org/ ) dGVzdGRhdGE= testdata AWS CLI (describe-stream list-streams)Kinesis (KCL) Base64 KCL Kinesis Client Library 1.x (p. 118)
get-records / NextShardIterator () get-records 300
22
https://www.base64decode.org/
-
Amazon Kinesis Data Streams
get-records get-shard-iterator
MillisBehindLatest GetRecords () 24 168 (7 )
get-records NextShardIterator KCL
get-records ()
{ "Records": [], "NextShardIterator": "AAAAAAAAAAGCJ5jzQNjmdhO6B/YDIDE56jmZmrmMA/r1WjoHXC/kPJXc1rckt3TFL55dENfe5meNgdkyCRpUPGzJpMgYHaJ53C3nCAjQ6s7ZupjXeJGoUFs5oCuFwhP+Wul/EhyNeSs5DYXLSSC5XCapmCAYGFjYER69QSdQjxMmBPE/hiybFDi5qtkT6/PsZNz6kFoqtDk="}
4:
aws kinesis delete-stream --stream-name Foo
describe-stream
aws kinesis describe-stream --stream-name Foo
delete
{ "StreamDescription": { "StreamStatus": "DELETING", "StreamName": "Foo", "StreamARN": "arn:aws:kinesis:us-west-2:account-id:stream/Foo", "Shards": [] }}
describe-stream not found
A client error (ResourceNotFoundException) occurred when calling the DescribeStream operation: Stream Foo under account 112233445566 not found.
23
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html
-
Amazon Kinesis Data Streams :
: Kinesis Data Streams
Amazon Kinesis Data Streams Kinesis Data Streams
Important
Kinesis Data Streams AWS Kinesis Data Streams Amazon DynamoDB DynamoDB AWS 7: (p. 37)
2015 2 25 5 ()
(: Amazon Elastic Compute Cloud (Amazon EC2))
() Kinesis Data Streams AWS
(p. 24) 1: (p. 25) 2: IAM (p. 26) 3: (p. 29) 4: (p. 30) 5: (p. 33) 6: () (p. 36) 7: (p. 37)
: Kinesis Data Streams (p. 24)
Amazon Web Services Kinesis Data Streams (p. 2): Amazon KinesisData Streams (p. 11)
24
#ak_region#ak_region
-
Amazon Kinesis Data Streams 1:
:Amazon Kinesis Data Streams (p. 11):AWS CLI Amazon Kinesis Data Streams (p. 17)
AWS AWS
IAM IAM AWS IAM AWS AWS How Do I GetSecurity Credentials?(AWS General Reference)
IAM IAM
Java 7 Java Development Kit (JDK) Oracle Java SE
Eclipse Java IDE
AWS SDK for Java Eclipse IDE AWSToolkit for Eclipse
Kinesis Client Library (KCL) 1.2.1 GitHub Kinesis Client Library (Java)
1: (p. 25)
1: : Kinesis Data Streams (p. 24)
1. AWS https://console.aws.amazon.com/kinesis Kinesis
2. [] 3. 4. [Kinesis ] 5. (: StockTradeStream) 6. 1[] 7. [Kinesis ]
[Kinesis ] CREATING ACTIVE [] []
25
https://console.aws.amazon.com/console/homehttps://docs.aws.amazon.com/IAM/latest/UserGuide/console.htmlhttps://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.htmlhttps://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.htmlget-set-up-for-amazon-ec2.html#create-an-iam-userhttp://www.oracle.com/technetwork/java/javase/downloads/index.htmlhttp://www.oracle.com/technetwork/java/javase/downloads/index.htmlhttps://www.eclipse.org/downloads/sdk-for-java/eclipse/eclipse/https://github.com/awslabs/amazon-kinesis-clienthttps://console.aws.amazon.com/kinesis
-
Amazon Kinesis Data Streams 2: IAM
Kinesis Data Streams ()
[Kinesis ] [] 1
1 20 1 250 1 () 5,000
()
Kinesis Data Streams (p. 5) (p. 39)
2: IAM (p. 26)
2: IAM AWS AWS Identity and Access Management (IAM) AWS IAM
Kinesis Data Streams
DescribeStream Kinesis data stream
PutRecordPutRecords Kinesis data stream Kinesis Data Streams
26
PoliciesOverview.html
-
Amazon Kinesis Data Streams 2: IAM
DescribeStream Kinesis data stream
GetRecordsGetShardIteratorKinesis data stream Kinesis Data Streams
CreateTableDescribeTableGetItemPutItemScanUpdateItemAmazon DynamoDB
Kinesis (KCL) DynamoDB
DeleteItem Amazon DynamoDB
Kinesis Data Streams
PutMetricData Amazon CloudWatch
KCL CloudWatch
IAM 1 2
IAM
1. Amazon (ARN) ARN [ ARN] []ARN
arn:aws:kinesis:region:account:stream/name
(us-west-2 )
AWS ID ()name
1: (p. 25) (StockTradeStream)
2. ()DynamoDB ARN
arn:aws:dynamodb:region:account:table/name
KCL StockTradesProcessor
3. IAM [] (https://console.aws.amazon.com/iam/home#policies) [] IAM [][]
4. [] [] 5. AWS [Amazon Kinesis]
27
using-regions-availability-zones.html#concepts-regions-availability-zonesusing-regions-availability-zones.html#concepts-regions-availability-zonesbilling/home?#/accountiam/home#policies
-
Amazon Kinesis Data Streams 2: IAM
6. DescribeStreamGetShardIteratorGetRecordsPutRecord PutRecords
7. 1 ARN 8. []
AWS ARN
Amazon DynamoDB CreateTableDeleteItemDescribeTableGetItemPutItemScanUpdateItem 2 ARN
Amazon CloudWatch PutMetricData *
ARN (*) PutMetricData CloudWatch
9. [Next Step] 10. [] StockTradeStreamPolicy []
{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt123", "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:PutRecord", "kinesis:PutRecords", "kinesis:GetShardIterator", "kinesis:GetRecords" ], "Resource": [ "arn:aws:kinesis:us-west-2:123:stream/StockTradeStream" ] }, { "Sid": "Stmt456", "Effect": "Allow", "Action": [ "dynamodb:*" ], "Resource": [ "arn:aws:dynamodb:us-west-2:123:table/StockTradesProcessor" ] }, { "Sid": "Stmt789", "Effect": "Allow", "Action": [ "cloudwatch:PutMetricData" ], "Resource": [ "*" ] } ]
28
-
Amazon Kinesis Data Streams 3:
}
IAM
1. https://console.aws.amazon.com/iam/ IAM 2. [] [] 3. [User name] StockTradeStreamUser 4. [] [] [: ]
5. [Attach existing policies directly] 6. [: ]
7. [] 8. [ ID] [] []
9.
~/.aws/credentials
[default]aws_access_key_id=access keyaws_secret_access_key=secret access key
IAM
1. IAM [] [] 2. [StockTradeStreamPolicy] [] 3. [StockTradeStreamUser] []
3: (p. 29)
3: the section called : (p. 24) () ()
1. 2. IDE
3.
Amazon Kinesis Client Library (KCL) AWS SDK Apache HttpCore Apache HttpClient
29
https://console.aws.amazon.com/iam/iam/home?#policies
-
Amazon Kinesis Data Streams 4:
Apache Commons Lang Apache Commons Logging Guava (Java Google ) Jackson Annotations Jackson Core Jackson Databind Jackson Dataformat: CBOR Joda Time
4. IDE IDE
(the section called 4: (p. 30))
(p. 30)
4: : Kinesis Data Streams (p. 24)
StockTrade
StockTrade () ID
JSON StockTrade 1 (:
{ "tickerSymbol": "AMZN", "tradeType": "BUY", "price": 395.87, "quantity": 16, "id": 3567129045}
StockTradeGenerator
StockTradeGenerator getRandomTrade()
StockTradesWriter
main StockTradesWriter Kinesis Data Streams
30
-
Amazon Kinesis Data Streams 4:
1. 2. AmazonKinesisClientBuilder 3. 4. AmazonKinesis 5. (
)6. StockTradeGenerator.getRandomTrade()
sendStockTrade 100
sendStockTrade StockTradesWriter
private static void sendStockTrade(StockTrade trade, AmazonKinesis kinesisClient, String streamName) { byte[] bytes = trade.toJsonAsBytes(); // The bytes could be null if there is an issue with the JSON serialization by the Jackson JSON library. if (bytes == null) { LOG.warn("Could not get JSON bytes for stock trade"); return; } LOG.info("Putting trade: " + trade.toString()); PutRecordRequest putRecord = new PutRecordRequest(); putRecord.setStreamName(streamName); // We use the ticker symbol as the partition key, explained in the Supplemental Information section below. putRecord.setPartitionKey(trade.getTickerSymbol()); putRecord.setData(ByteBuffer.wrap(bytes));
try { kinesisClient.putRecord(putRecord); } catch (AmazonClientException ex) { LOG.warn("Error sending record to Amazon Kinesis.", ex); }}
PutRecord API trade JSON
1
byte[] bytes = trade.toJsonAsBytes();
PutRecordRequest (putRecord )
PutRecordRequest putRecord = new PutRecordRequest();
PutRecord BLOB putRecord setXxxx()
putRecord.setStreamName(streamName);putRecord.setPartitionKey(trade.getTickerSymbol());putRecord.setData(ByteBuffer.wrap(bytes));
31
-
Amazon Kinesis Data Streams 4:
1 (p. 101)
putRecord (put )
kinesisClient.putRecord(putRecord);
if (bytes == null) { LOG.warn("Could not get JSON bytes for stock trade"); return;}
put try/catch
try { kinesisClient.putRecord(putRecord);} catch (AmazonClientException ex) { LOG.warn("Error sending record to Amazon Kinesis.", ex);}
KinesisData Streams put put
LOG.info("Putting trade: " + trade.toString());
Kinesis Data Streams API (PutRecord) PutRecords (p. 101)
1. (IAM ) ~/.aws/credentials
2. StockTradeWriter
StockTradeStream us-west-2
us-west-2
Feb 16, 2015 3:53:00 PM com.amazonaws.services.kinesis.samples.stocktrades.writer.StockTradesWriter sendStockTradeINFO: Putting trade: ID 8: SELL 996 shares of BUD for $124.18
32
-
Amazon Kinesis Data Streams 5:
Feb 16, 2015 3:53:00 PM com.amazonaws.services.kinesis.samples.stocktrades.writer.StockTradesWriter sendStockTradeINFO: Putting trade: ID 9: BUY 159 shares of GE for $20.85Feb 16, 2015 3:53:01 PM com.amazonaws.services.kinesis.samples.stocktrades.writer.StockTradesWriter sendStockTradeINFO: Putting trade: ID 10: BUY 322 shares of WMT for $90.08
Kinesis Data Streams
5: (p. 33)
5: : Kinesis Data Streams (p. 24) (p. 30)1 KinesisClient Library (KCL) Kinesis Client Library 1.x (p. 118)
StockTradesProcessor
~/.aws/credentials RecordProcessorFactory RecordProcessor
StockTradeRecordProcessor RecordProcessorFactory (:
) KCL ()
Kinesis Data Streams RecordProcessor
StockTradeRecordProcessor
RecordProcessor initializeprocessRecordsshutdown 3
Kinesis Client Library initialize shutdown processRecords processRecord
processRecord reportStats resetStats
processsRecords processRecord
33
-
Amazon Kinesis Data Streams 5:
1 reportStats() resetStats()
1 checkpoint()
60 (p. 35)
StockStats
addStockTrade(StockTrade): StockTrade toString():
StockTradeRecordProcessor
1. processRecord StockTrade
StockTrade trade = StockTrade.fromJsonAsBytes(record.getData().array());if (trade == null) { LOG.warn("Skipping record. Unable to parse record into StockTrade. Partition Key: " + record.getPartitionKey()); return;}stockStats.addStockTrade(trade);
2. reportStats
System.out.println("****** Shard " + kinesisShardId + " stats for last 1 minute ******\n" + stockStats + "\n" + "****************************************************************\n");
3. resetStats stockStats
stockStats = new StockStats();
1. (p. 30)
2. (IAM ) ~/.aws/credentials
3. StockTradesProcessor
34
-
Amazon Kinesis Data Streams 5:
StockTradesProcessor StockTradeStream us-west-2
us-west-2
1 1
****** Shard shardId-000000000001 stats for last 1 minute ****** Most popular stock being bought: WMT, 27 buys. Most popular stock being sold: PTR, 14 sells. ****************************************************************
Kinesis Client Library 1.x (p. 118) KinesisClient Library 1 1 KCL KCL
KCL Kinesis Data Streams
(p. 30)Kinesis Data Streams API put Kinesis Data Streams ()
1 KCL KCL
6: () (p. 36)
35
-
Amazon Kinesis Data Streams 6: ()
6: () : Kinesis Data Streams (p. 24)
1 3 StockStats
1.
// Ticker symbol of the stock that had the largest quantity of shares sold private String largestSellOrderStock; // Quantity of shares for the largest sell order trade private long largestSellOrderQuantity;
2. addStockTrade
if (type == TradeType.SELL) { if (largestSellOrderStock == null || trade.getQuantity() > largestSellOrderQuantity) { largestSellOrderStock = trade.getTickerSymbol(); largestSellOrderQuantity = trade.getQuantity(); } }
3. toString
public String toString() { return String.format( "Most popular stock being bought: %s, %d buys.%n" + "Most popular stock being sold: %s, %d sells.%n" + "Largest sell order: %d shares of %s.", getMostPopularStock(TradeType.BUY), getMostPopularStockCount(TradeType.BUY), getMostPopularStock(TradeType.SELL), getMostPopularStockCount(TradeType.SELL), largestSellOrderQuantity, largestSellOrderStock); }
()
****** Shard shardId-000000000001 stats for last 1 minute ****** Most popular stock being bought: WMT, 27 buys. Most popular stock being sold: PTR, 14 sells. Largest sell order: 996 shares of BUD. ****************************************************************
7: (p. 37)
36
-
Amazon Kinesis Data Streams 7:
7: Kinesis data stream Amazon DynamoDB ""
1. 2. https://console.aws.amazon.com/kinesis Kinesis 3. (StockTradeStream)4. [] 5. https://console.aws.amazon.com/dynamodb/ DynamoDB 6. StockTradesProcessor
Kinesis Data Streams (processRecord(Record) ) Kinesis Data Streams Kinesis Data Streams
() 1 1 ( 1 1 )
() AmazonEC2 Auto Scaling
Amazon S3/DynamoDB/Amazon Redshift/Storm
AWS Kinesis Data Streams AWS
Kinesis Data Streams API Amazon Kinesis Data Streams
API AWS SDK for Java (p. 100)AWS SDK for Java Kinesis Data Streams API (p. 137) (p. 39)
37
https://console.aws.amazon.com/kinesishttps://console.aws.amazon.com/dynamodb/https://github.com/awslabs/amazon-kinesis-connectors
-
Amazon Kinesis Data Streams 7:
Kinesis Client Library Kinesis Client Library 1.x (p. 118)
(p. 160)
38
-
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams API AWS SDK for Java Kinesis data stream
Java Kinesis Data Streams API Kinesis Data Streams API AWS SDK
(p. 39) (p. 41) (p. 42) (p. 43) (p. 43) (p. 44) (p. 49) Amazon Kinesis Data Streams (p. 49) Amazon Kinesis Data Streams (p. 52) Amazon Kinesis Data Streams IAM (p. 78) (p. 82) Amazon Kinesis Data Streams VPC (p. 87) Kinesis (p. 88)
Kinesis data stream
Kinesis Data Streams Kinesis data stream Java
AmazonKinesisClientBuilder clientBuilder = AmazonKinesisClientBuilder.standard(); clientBuilder.setRegion(regionName);clientBuilder.setCredentials(credentialsProvider);clientBuilder.setClientConfiguration(config); AmazonKinesis client = clientBuilder.build();
39
sdk-for-java/developers/getting-started/
-
Amazon Kinesis Data Streams
AWS General Reference Kinesis Data Streams
Kinesis Data Streams Kinesis Data Streams CreateStreamRequest
CreateStreamRequest createStreamRequest = new CreateStreamRequest();createStreamRequest.setStreamName( myStreamName );createStreamRequest.setShardCount( myStreamSize );
AWS 2 AWS 2 AWS 2 2 2
AWSKinesis DataStream (p. 5)
createStreamRequest createStream createStream ACTIVE describeStream describeStream describeStream try/catch
client.createStream( createStreamRequest );DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();describeStreamRequest.setStreamName( myStreamName );
long startTime = System.currentTimeMillis();long endTime = startTime + ( 10 * 60 * 1000 );while ( System.currentTimeMillis() < endTime ) { try { Thread.sleep(20 * 1000); } catch ( Exception e ) {} try { DescribeStreamResult describeStreamResponse = client.describeStream( describeStreamRequest ); String streamStatus = describeStreamResponse.getStreamDescription().getStreamStatus(); if ( streamStatus.equals( "ACTIVE" ) ) { break; } // // sleep for one second // try { Thread.sleep( 1000 ); } catch ( Exception e ) {} } catch ( ResourceNotFoundException e ) {}
40
#ak_region
-
Amazon Kinesis Data Streams
}if ( System.currentTimeMillis() >= endTime ) { throw new RuntimeException( "Stream " + myStreamName + " never went active" );}
Kinesis Data Streams AWS AWS AWS 1 Kinesis Data Streams AWS
ListStreamsRequest listStreamsRequest = new ListStreamsRequest();listStreamsRequest.setLimit(20); ListStreamsResult listStreamsResult = client.listStreams(listStreamsRequest);List streamNames = listStreamsResult.getStreamNames();
ListStreamsRequest setLimit 20 listStreams setLimit KinesisData Streams listStreamsRequest listStreams listStreams ListStreamsResult getStreamNames streamNames Kinesis Data Streams getHasMoreStreams
while (listStreamsResult.getHasMoreStreams()) { if (streamNames.size() > 0) { listStreamsRequest.setExclusiveStartStreamName(streamNames.get(streamNames.size() - 1)); } listStreamsResult = client.listStreams(listStreamsRequest); streamNames.addAll(listStreamsResult.getStreamNames());}
getHasMoreStreams listStreamsRequest listStreams setExclusiveStartStreamName listStreams setExclusiveStartStreamName listStreams streamNames
listStreams
CREATING ACTIVE UPDATING DELETING
41
-
Amazon Kinesis Data Streams
describeStream (p. 39)
1 ListShards
import software.amazon.awssdk.services.kinesis.KinesisAsyncClient;import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
import java.util.concurrent.TimeUnit;
public class ShardSample {
public static void main(String[] args) {
KinesisAsyncClient client = KinesisAsyncClient.builder().build();
ListShardsRequest request = ListShardsRequest .builder().streamName("myFirstStream") .build();
try { ListShardsResponse response = client.listShards(request).get(5000, TimeUnit.MILLISECONDS); System.out.println(response.toString()); } catch (Exception e) { System.out.println(e.getMessage()); } }}
POM
4.0.0
kinesis.data.streams.samples shards 1.0-SNAPSHOT org.apache.maven.plugins maven-compiler-plugin 8 8
42
API_ListShards.html
-
Amazon Kinesis Data Streams
software.amazon.awssdk kinesis 2.0.0
describeStream getShards 1 getHasMoreShards getStreamDescription true getShards getHasMoreShards false getShards EXPIRED (EXPIRED ) (p. 48)
DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();describeStreamRequest.setStreamName( myStreamName );List shards = new ArrayList();String exclusiveStartShardId = null;do { describeStreamRequest.setExclusiveStartShardId( exclusiveStartShardId ); DescribeStreamResult describeStreamResult = client.describeStream( describeStreamRequest ); shards.addAll( describeStreamResult.getStreamDescription().getShards() ); if (describeStreamResult.getStreamDescription().getHasMoreShards() && shards.size() > 0) { exclusiveStartShardId = shards.get(shards.size() - 1).getShardId(); } else { exclusiveStartShardId = null; }} while ( exclusiveStartShardId != null );
Kinesis Data Streams DeleteStreamRequest
DeleteStreamRequest deleteStreamRequest = new DeleteStreamRequest();deleteStreamRequest.setStreamName(myStreamName);client.deleteStream(deleteStreamRequest);
ResourceNotFound
43
-
Amazon Kinesis Data Streams
Important
UpdateShardCount API
Amazon Kinesis Data Streams Kinesis Data Streams Kinesis Data Streams
2 1 2 2 1 1 1 2
()
() () Amazon CloudWatch IAM API Kinesis Data Streams IAM Amazon Kinesis Data Streams IAM (p. 78)
(p. 44) (p. 45) 2 (p. 46) (p. 47)
Amazon Kinesis Data Streams () ()
1
44
API_UpdateShardCount.html
-
Amazon Kinesis Data Streams
Kinesis Data Streams Amazon CloudWatch 1
putRecordRequest.setPartitionKey( String.format( "myPartitionKey" ) );
Kinesis Data Streams MD5 MD5
ID ID getShardId putRecordResults putRecords putRecordResult putRecord
String shardId = putRecordResult.getShardId();
ID
Amazon Kinesis Data Streams MD5
shard.getHashKeyRange().getStartingHashKey();shard.getHashKeyRange().getEndingHashKey();
1 1/3 1 2/3
myStreamName shard splitShardRequest ID
SplitShardRequest splitShardRequest = new SplitShardRequest();splitShardRequest.setStreamName(myStreamName);splitShardRequest.setShardToSplit(shard.getShardId());
45
http://en.wikipedia.org/wiki/MD5http://en.wikipedia.org/wiki/MD5
-
Amazon Kinesis Data Streams 2
setNewStartingHashKey Kinesis Data Streams Kinesis Data Streams splitShard
BigInteger startingHashKey = new BigInteger(shard.getHashKeyRange().getStartingHashKey());BigInteger endingHashKey = new BigInteger(shard.getHashKeyRange().getEndingHashKey());String newStartingHashKey = startingHashKey.add(endingHashKey).divide(new BigInteger("2")).toString();
splitShardRequest.setNewStartingHashKey(newStartingHashKey);client.splitShard(splitShardRequest);
(p. 47)
2 2 1 1 2
2 2 2 2 1 276381 1 382454 2 1 276454
2 1 276381 1 455560 2 1 ( 382454)
OPEN () MD5 (CLOSED ) (p. 48)
CLOSED OPEN (CLOSED ) null
if( null == shard.getSequenceNumberRange().getEndingSequenceNumber() ) { // Shard is OPEN, so it is a possible candidate to be merged.}
CLOSED
shard.getHashKeyRange().getEndingHashKey();
2
46
-
Amazon Kinesis Data Streams
2 myStreamName shard1 shard2 2
mergeShardsRequest setStreamName setShardToMerge setAdjacentShardToMerge 2 Kinesis Data Streams mergeShards
MergeShardsRequest mergeShardsRequest = new MergeShardsRequest();mergeShardsRequest.setStreamName(myStreamName);mergeShardsRequest.setShardToMerge(shard1.getShardId());mergeShardsRequest.setAdjacentShardToMerge(shard2.getShardId());client.mergeShards(mergeShardsRequest);
(p. 47)
Amazon Kinesis Data Streams
(p. 47) (p. 48)
splitShard mergeShards (p. 39)
DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();describeStreamRequest.setStreamName( myStreamName );
long startTime = System.currentTimeMillis();long endTime = startTime + ( 10 * 60 * 1000 );while ( System.currentTimeMillis() < endTime ) { try { Thread.sleep(20 * 1000); } catch ( Exception e ) {} try { DescribeStreamResult describeStreamResponse = client.describeStream( describeStreamRequest ); String streamStatus = describeStreamResponse.getStreamDescription().getStreamStatus(); if ( streamStatus.equals( "ACTIVE" ) ) { break; } // // sleep for one second //
47
-
Amazon Kinesis Data Streams
try { Thread.sleep( 1000 ); } catch ( Exception e ) {} } catch ( ResourceNotFoundException e ) {}}if ( System.currentTimeMillis() >= endTime ) { throw new RuntimeException( "Stream " + myStreamName + " never went active" );}
Kinesis Data Streams KinesisData Streams API getShardIterator getRecords (p. 137) Kinesis Client Library
Note
(p. 49)
OPEN CLOSED EXPIRED
OPEN: OPEN
CLOSED: CLOSED
EXPIRED: EXPIRED getStreamDescription().getShards EXPIRED (p. 49)
ACTIVE getRecordsResult.getNextShardIterator nullKinesis Client Library
48
-
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams Kinesis data stream Kinesis data stream 24 168
IncreaseStreamRetentionPeriod 168 DecreaseStreamRetentionPeriod 24 DescribeStream
AWS CLI
aws kinesis increase-stream-retention-period --stream-name retentionPeriodDemo --retention-period-hours 72
Kinesis Data Streams 24 48 23 55 24
Kinesis Data Streams DecreaseStreamRetentionPeriod
API
24 Amazon Kinesis Data Streams
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams AWS ()
(p. 50) (p. 50) (p. 50) Kinesis Data Streams (p. 51) AWS CLI (p. 51)
49
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_IncreaseStreamRetentionPeriod.htmlhttps://docs.aws.amazon.com/kinesis/latest/APIReference/API_DecreaseStreamRetentionPeriod.htmlhttps://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.htmlhttps://docs.aws.amazon.com/kinesis/latest/APIReference/API_DecreaseStreamRetentionPeriod.htmlhttps://aws.amazon.com/kinesis/pricing/
-
Amazon Kinesis Data Streams
Kinesis Data Streams API (p. 52)
Kinesis Data Streams AWS CLI Kinesis Data Streams API
: : : : :
AWS AWS () AWS () AWS Billing and Cost Management
() 50
aws: AWS AWS
1128 Unicode Unicode (_ . / = +- @)
50
-
Amazon Kinesis Data Streams Kinesis Data Streams
0255 Unicode
Unicode (_ . / = + - @)
Kinesis Data Streams Kinesis Data Streams
1. Kinesis Data Streams
2. [] 3. [] []
1. Kinesis Data Streams
2. [] 3. [] [] 4. [] []
[]
[] (p. 50)
5. []
1. Kinesis Data Streams
2. [Stream List] 3. [] [] [] 4. [] []
AWS CLI AWS CLI
add-tags-to-stream
list-tags-for-stream
51
kinesis/add-tags-to-stream.htmlkinesis/list-tags-for-stream.html
-
Amazon Kinesis Data Streams Kinesis Data Streams API
remove-tags-from-stream
Kinesis Data Streams API Kinesis Data Streams API
AddTagsToStream
ListTagsForStream
RemoveTagsFromStream
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams
CloudWatch (p. 52) Kinesis Data Streams Amazon CloudWatch
Kinesis (p. 62) Kinesis CloudWatch
API (p. 63) Kinesis Data Streams AWS CloudTrail API Amazon S3
Kinesis (p. 66) Kinesis Client Library (KCL) KCL
Kinesis (p. 74) Kinesis Producer Library (KPL) KPL
Amazon CloudWatch Amazon Kinesis DataStreams Amazon Kinesis Data Streams Amazon CloudWatch Kinesis CloudWatch PutRecords.Bytes GetRecords.Bytes
1 CloudWatch 2
Kinesis
52
kinesis/remove-tags-from-stream.htmlAPI_AddTagsToStream.htmlAPI_ListTagsForStream.htmlAPI_RemoveTagsFromStream.html
-
Amazon Kinesis Data Streams CloudWatch
() 1
() 1EnableEnhancedMonitoring
Amazon CloudWatch
Amazon Kinesis Data Streams Kinesis Data Streams 2 CloudWatch EnableEnhancedMonitoring
CloudWatch Amazon CloudWatch CloudWatch
(p. 53) (p. 58) Amazon Kinesis Data Streams (p. 60) Amazon Kinesis Data Streams (p. 61)
AWS/Kinesis
Kinesis Data Streams 1 CloudWatch
GetRecords.Bytes Kinesis MinimumMaximum Average GetRecords
: OutgoingBytes
: StreamName
: MinimumMaximumAverageSumSamples
:
53
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_EnableEnhancedMonitoring.htmlcloudwatchcloudwatchhttps://docs.aws.amazon.com/kinesis/latest/APIReference/API_EnableEnhancedMonitoring.htmlhttps://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#Statistic
-
Amazon Kinesis Data Streams CloudWatch
GetRecords.IteratorAge GetRecords.IteratorAgeMilliseconds
GetRecords.IteratorAgeMillisecondsKinesis GetRecords ()GetRecords Minimum Maximum Kinesis
: IteratorAgeMilliseconds
: StreamName
: MinimumMaximumAverageSamples
:
GetRecords.Latency GetRecords
: StreamName
: MinimumMaximumAverage
:
GetRecords.Records MinimumMaximum Average GetRecords
: OutgoingRecords
: StreamName
: MinimumMaximumAverageSumSamples
: Count
GetRecords.Success GetRecords
: StreamName
: AverageSumSamples
: Count
54
-
Amazon Kinesis Data Streams CloudWatch
IncomingBytes Kinesis PutRecord PutRecords MinimumMaximum Average put
: IncomingBytes
: StreamName
: MinimumMaximumAverageSumSamples
:
IncomingRecords Kinesis PutRecord PutRecords MinimumMaximum Average put
: IncomingRecords
: StreamName
: MinimumMaximumAverageSumSamples
: Count
PutRecord.Bytes PutRecord Kinesis
: StreamName
: MinimumMaximumAverageSumSamples
:
PutRecord.Latency PutRecord
: StreamName
: MinimumMaximumAverage
:
PutRecord.Success Kinesis PutRecord Average
: StreamName
: AverageSumSamples
: Count
55
-
Amazon Kinesis Data Streams CloudWatch
PutRecords.Bytes PutRecords Kinesis
: StreamName
: MinimumMaximumAverageSumSamples
:
PutRecords.Latency PutRecords
: StreamName
: MinimumMaximumAverage
:
PutRecords.Records Kinesis PutRecords
: StreamName
: MinimumMaximumAverageSumSamples
: Count
PutRecords.Success Kinesis 1 PutRecords
: StreamName
: AverageSumSamples
: Count
ReadProvisionedThroughputExceeded GetRecords Average
Minimum 1
Maximum 0 ()
:ReadProvisionedThroughputExceeded
: StreamName
: MinimumMaximumAverageSumSamples
: Count
56
-
Amazon Kinesis Data Streams CloudWatch
SubscribeToShard.RateExceeded 1 : StreamNameConsumerName
SubscribeToShard.Success SubscribeToShard 5 5 1
: StreamNameConsumerName
SubscribeToShardEvent.Bytes MinimumMaximum Average
: OutgoingBytes
: StreamNameConsumerName
: MinimumMaximumAverageSumSamples
:
SubscribeToShardEvent.MillisBehindLatestSubscribeToShard
: StreamNameConsumerName
: MinimumMaximumAverageSamples
:
SubscribeToShardEvent.Records MinimumMaximum Average
: OutgoingRecords
: StreamNameConsumerName
: MinimumMaximumAverageSumSamples
: Count
SubscribeToShardEvent.Success
: StreamNameConsumerName
: MinimumMaximumAverageSumSamples
: Count
57
-
Amazon Kinesis Data Streams CloudWatch
WriteProvisionedThroughputExceededPutRecord PutRecords Average
Minimum
Maximum 0 ()
:WriteProvisionedThroughputExceeded
: StreamName
: MinimumMaximumAverageSumSamples
: Count
AWS/Kinesis
Kinesis 1 CloudWatch Kinesis Amazon CloudWatch Amazon CloudWatch 1
IncomingBytes PutRecord PutRecords MinimumMaximum Average put
: IncomingBytes
: StreamNameShardId
: MinimumMaximumAverageSumSamples
:
IncomingRecords PutRecord PutRecords MinimumMaximum Average put
: IncomingRecords
58
cloudwatch/pricing/
-
Amazon Kinesis Data Streams CloudWatch
: StreamNameShardId
: MinimumMaximumAverageSumSamples
: Count
IteratorAgeMilliseconds GetRecords ()GetRecords Minimum Maximum Kinesis 0 ()
:GetRecords.IteratorAgeMilliseconds
: StreamNameShardId
: MinimumMaximumAverageSamples
:
OutgoingBytes MinimumMaximum Average GetRecords SubscribeToShard
: GetRecords.Bytes
: StreamNameShardId
: MinimumMaximumAverageSumSamples
:
OutgoingRecords MinimumMaximum Average GetRecords SubscribeToShard
: GetRecords.Records
: StreamNameShardId
: MinimumMaximumAverageSumSamples
: Count
59
-
Amazon Kinesis Data Streams CloudWatch
ReadProvisionedThroughputExceeded GetRecords 1 1 5 1 1 2 MB Average
Minimum 1
Maximum 0 ()
:ReadProvisionedThroughputExceeded
: StreamNameShardId
: MinimumMaximumAverageSumSamples
: Count
WriteProvisionedThroughputExceededPutRecord PutRecords 1 1 1,000 1 1 1 MB Average
Minimum
Maximum 0 ()
:WriteProvisionedThroughputExceeded
: StreamNameShardId
: MinimumMaximumAverageSumSamples
: Count
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams
StreamName Kinesis
ShardId Kinesis ID
60
-
Amazon Kinesis Data Streams CloudWatch
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams Kinesis Data Streams
GetRecords.IteratorAgeMilliseconds ( 24 7 ) 50% CloudWatch (p. 159)
ReadProvisionedThroughputExceeded Average
WriteProvisionedThroughputExceededReadProvisionedThroughputExceeded (PUT) Average
PutRecord.SuccessPutRecords.Success CloudWatch PUT KinesisProducer Library (KPL) PutRecords.Success
GetRecords.Success CloudWatch
CloudWatch CloudWatch API Kinesis Data Streams
CloudWatch
1. https://console.aws.amazon.com/cloudwatch/ CloudWatch 2. 3. 4. [ CloudWatch ] [Kinesis ] 5. [MetricName] [StreamName]
: [] [] CloudWatch 5 [] IncomingBytes CloudWatch [] GetRecords.Bytes
6. () CloudWatch
61
https://console.aws.amazon.com/cloudwatch/
-
Amazon Kinesis Data Streams CloudWatch
AWS CLI
list-metrics get-metric-statistics
CloudWatch CLI
mon-list-metrics mon-get-stats
CloudWatch API
ListMetrics GetMetricStatistics
Amazon CloudWatch Kinesis Data Streams AWSKinesisAgent CloudWatch Kinesis Data Streams CPU CPU cloudwatch.endpoint (p. 107)
CloudWatch Kinesis Data Streams CloudWatch
BytesSent Kinesis Data Streams
:
RecordSendAttempts PutRecords ()
: Count
RecordSendErrors PutRecords ()
: Count
ServiceErrors () PutRecords
: Count
62
cloudwatch/list-metrics.htmlcloudwatch/get-metric-statistics.htmlcli-mon-list-metrics.htmlcli-mon-get-stats.htmlAPI_ListMetrics.htmlAPI_GetMetricStatistics.html
-
Amazon Kinesis Data Streams AWS CloudTrail Amazon
Kinesis Data Streams API
AWS CloudTrail Amazon Kinesis DataStreams API Amazon Kinesis Data Streams AWS CloudTrail Kinesis DataStreams API KinesisData Streams Kinesis Data Streams API Amazon S3 CloudTrail [Event history ()] CloudTrail IP KinesisData Streams
CloudTrail AWS CloudTrail User Guide
CloudTrail AWS Kinesis Data Streams CloudTrail AWS [Event history ()] AWS CloudTrail
CloudTrail Amazon S3 AWS AWS Amazon S3 AWS CloudTrail
CloudTrail CloudTrail Amazon SNS CloudTrail CloudTrail
Kinesis Data Streams CloudTrail
AddTagsToStream CreateStream DecreaseStreamRetentionPeriod DeleteStream DeregisterStreamConsumer DescribeStream DescribeStreamConsumer DisableEnhancedMonitoring EnableEnhancedMonitoring IncreaseStreamRetentionPeriod ListStreamConsumers ListStreams ListTagsForStream
63
view-cloudtrail-events.htmlview-cloudtrail-events.htmlcloudtrail-create-and-update-a-trail.htmlcloudtrail-aws-service-specific-topics.html#cloudtrail-aws-service-specific-topics-integrationsgetting_notifications_top_level.htmlreceive-cloudtrail-log-files-from-multiple-regions.htmlcloudtrail-receive-logs-from-multiple-accounts.htmlcloudtrail-receive-logs-from-multiple-accounts.htmlAPI_AddTagsToStream.htmlAPI_CreateStream.htmlAPI_DecreaseStreamRetentionPeriod.htmlAPI_DeleteStream.htmlAPI_DeregisterStreamConsumer.htmlAPI_DescribeStream.htmlAPI_DescribeStreamConsumer.htmlAPI_DisableEnhancedMonitoring.htmlAPI_EnableEnhancedMonitoring.htmlAPI_IncreaseStreamRetentionPeriod.htmlAPI_ListStreamConsumers.htmlAPI_ListStreams.htmlAPI_ListTagsForStream.html
-
Amazon Kinesis Data Streams AWS CloudTrail Amazon
Kinesis Data Streams API
MergeShards RegisterStreamConsumer RemoveTagsFromStream SplitShard StartStreamEncryption StopStreamEncryption UpdateShardCount
ID
AWS Identity and Access Management (IAM)
.
AWS
CloudTrail userIdentity
: Kinesis Data Streams Amazon S3 CloudTrail1 CloudTrail API
CreateStreamDescribeStreamListStreamsDeleteStreamSplitShardMergeShards CloudTrail
{ "Records": [ { "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:16:31Z", "eventSource": "kinesis.amazonaws.com", "eventName": "CreateStream", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "shardCount": 1, "streamName": "GoodStream" }, "responseElements": null, "requestID": "db6c59f8-c757-11e3-bc3b-57923b443c1c", "eventID": "b7acfcd0-6ca9-4ee1-a3d7-c4e8d420d99b" },
64
API_MergeShards.htmlAPI_RegisterStreamConsumer.htmlAPI_RemoveTagsFromStream.htmlAPI_SplitShard.htmlAPI_StartStreamEncryption.htmlAPI_StopStreamEncryption.htmlAPI_UpdateShardCount.htmlcloudtrail-event-reference-user-identity.html
-
Amazon Kinesis Data Streams AWS CloudTrail Amazon
Kinesis Data Streams API
{ "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:17:06Z", "eventSource": "kinesis.amazonaws.com", "eventName": "DescribeStream", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "streamName": "GoodStream" }, "responseElements": null, "requestID": "f0944d86-c757-11e3-b4ae-25654b1d3136", "eventID": "0b2f1396-88af-4561-b16f-398f8eaea596" }, { "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:15:02Z", "eventSource": "kinesis.amazonaws.com", "eventName": "ListStreams", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "limit": 10 }, "responseElements": null, "requestID": "a68541ca-c757-11e3-901b-cbcfe5b3677a", "eventID": "22a5fb8f-4e61-4bee-a8ad-3b72046b4c4d" }, { "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:17:07Z", "eventSource": "kinesis.amazonaws.com", "eventName": "DeleteStream", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "streamName": "GoodStream" }, "responseElements": null,
65
-
Amazon Kinesis Data Streams CloudWatch KCL
"requestID": "f10cd97c-c757-11e3-901b-cbcfe5b3677a", "eventID": "607e7217-311a-4a08-a904-ec02944596dd" }, { "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:15:03Z", "eventSource": "kinesis.amazonaws.com", "eventName": "SplitShard", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "shardToSplit": "shardId-000000000000", "streamName": "GoodStream", "newStartingHashKey": "11111111" }, "responseElements": null, "requestID": "a6e6e9cd-c757-11e3-901b-cbcfe5b3677a", "eventID": "dcd2126f-c8d2-4186-b32a-192dd48d7e33" }, { "eventVersion": "1.01", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::012345678910:user/Alice", "accountId": "012345678910", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2014-04-19T00:16:56Z", "eventSource": "kinesis.amazonaws.com", "eventName": "MergeShards", "awsRegion": "us-east-1", "sourceIPAddress": "127.0.0.1", "userAgent": "aws-sdk-java/unknown-version Linux/x.xx", "requestParameters": { "streamName": "GoodStream", "adjacentShardToMerge": "shardId-000000000002", "shardToMerge": "shardId-000000000001" }, "responseElements": null, "requestID": "e9f9c8eb-c757-11e3-bf1d-6948db3cd570", "eventID": "77cf0d06-ce90-42da-9576-71986fec411f" } ]}
Amazon CloudWatch Kinesis Kinesis Client Library (KCL) for Amazon Kinesis Data Streams KCL Amazon CloudWatch CloudWatch []
66
https://docs.aws.amazon.com/kinesis/latest/dev/developing-consumers-with-kcl.htmlhttps://console.aws.amazon.com/cloudwatch/
-
Amazon Kinesis Data Streams CloudWatch KCL
Amazon CloudWatch
KCL CloudWatch Amazon CloudWatch Amazon CloudWatch API Amazon CloudWatch
(p. 67) (p. 67) (p. 67) (p. 68)
KCL
CloudWatch 2
CloudWatch NONESUMMARYDETAILED DETAILED CloudWatch NONE (p. 68)
KCL CloudWatch OperationWorkerIdentifierOperation ShardId
CloudWatch Amazon CloudWatch Amazon CloudWatch
WorkerIdentifier KCL ID WorkerIdentifier CloudWatch KCL WorkerIdentifier ID KCL ID KCL
KinesisClientLibConfiguration KCL MultiLangDaemon metricsLevel metricsEnabledDimensions MultiLangDaemon KCL .properties
67
publishingMetrics.htmlpublishingMetrics.htmlhttps://aws.amazon.com/cloudwatch/pricing/cloudwatch_concepts.html#Dimension
-
Amazon Kinesis Data Streams CloudWatch KCL
NONESUMMARY DETAILED 3 1 CloudWatch KCL OperationShardId WorkerIdentifier
KCL
KCL (p. 68) (p. 71) (p. 73)
KCL Amazon CloudWatch KCL
InitializeTask (p. 68) ShutdownTask (p. 69) ShardSyncTask (p. 70) BlockOnParentTask (p. 70)
InitializeTask
InitializeTask KCL Kinesis Data Streams
KinesisDataFetcher.getIterator.SuccessKCL GetShardIterator
: Detailed
: Count
KinesisDataFetcher.getIterator.Time KCL GetShardIterator
: Detailed
:
RecordProcessor.initialize.Time
: Summary
:
Success
: Summary
68
-
Amazon Kinesis Data Streams CloudWatch KCL
: Count
KCL
: Summary
:
ShutdownTask
ShutdownTask shutdown() 1 2
CreateLease.Success KCL DynamoDB
: Detailed
: Count
CreateLease.Time KCL DynamoDB
: Detailed
:
UpdateLease.Success
: Detailed
: Count
UpdateLease.Time
: Detailed
:
RecordProcessor.shutdown.Time
: Summary
:
Success
: Summary
: Count
69
-
Amazon Kinesis Data Streams CloudWatch KCL
KCL
: Summary
:
ShardSyncTask
ShardSyncTask Kinesis data stream KCL
CreateLease.Success KCL DynamoDB
: Detailed
: Count
CreateLease.Time KCL DynamoDB
: Detailed
:
Success
: Summary
: Count
: Summary
:
BlockOnParentTask
BlockOnParentTask KCL
Success
: Summary
: Count
: Summary
70
-
Amazon Kinesis Data Streams CloudWatch KCL
:
Amazon EC2 Kinesis data stream
RenewAllLeases (p. 71) TakeLeases (p. 72)
RenewAllLeases
RenewAllLeases
RenewLease.Success
: Detailed
: Count
RenewLease.Time
: Detailed
:
CurrentLeases
: Summary
: Count
LostLeases
: Summary
: Count
Success
: Summary
: Count
: Summary
:
71
-
Amazon Kinesis Data Streams CloudWatch KCL
TakeLeases
TakeLeases KCL KCL
ListLeases.Success KCL DynamoDB
: Detailed
: Count
ListLeases.Time KCL DynamoDB
: Detailed
:
TakeLease.Success KCL
: Detailed
: Count
TakeLease.Time
: Detailed
:
NumWorkers
: Summary
: Count
NeededLeases
: Detailed
: Count
LeasesToTake
: Detailed
: Count
TakenLeases
: Summary
: Count
TotalLeases KCL
72
-
Amazon Kinesis Data Streams CloudWatch KCL
: Detailed
: Count
ExpiredLeases
: Summary
: Count
Success TakeLeases
: Summary
: Count
TakeLeases
: Summary
:
ProcessTask
ProcessTask GetRecords processRecords
KinesisDataFetcher.getRecords.SuccessKinesis data stream GetRecords
: Detailed
: Count
KinesisDataFetcher.getRecords.TimeKinesis data stream GetRecords
: Detailed
:
UpdateLease.Success
: Detailed
: Count
UpdateLease.Time
: Detailed
73
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html
-
Amazon Kinesis Data Streams CloudWatch KPL
:
DataBytesProcessed ProcessTask
: Summary
:
RecordsProcessed ProcessTask
: Summary
: Count
ExpiredIterator GetRecords ExpiredIteratorException
: Summary
: Count
MillisBehindLatest ()
: Summary
:
RecordProcessor.processRecords.Time processRecords
: Summary
:
Success
: Summary
: Count
: Summary
:
Amazon CloudWatch Kinesis Kinesis Producer Library (KPL) for Amazon Kinesis Data Streams Amazon CloudWatch CloudWatch [
74
https://docs.aws.amazon.com/kinesis/latest/dev/developing-producers-with-kpl.htmlhttps://console.aws.amazon.com/cloudwatch/
-
Amazon Kinesis Data Streams CloudWatch KPL
] AmazonCloudWatch
KPL CloudWatch Amazon CloudWatch Amazon CloudWatch API Amazon CloudWatch CloudWatch
(p. 75) (p. 75) Amazon CloudWatch (p. 76) (p. 76)
KPL KPL
KPL CloudWatch KPL KPL
CloudWatch 2
CloudWatch NONESUMMARYDETAILED DETAILED NONE
GLOBALSTREAMSHARD SHARD
SHARD ID 2 2 7 CloudWatch 1 1 1
MetricName (GLOBAL): Metric X Metric Y | | ----------------- ------------ | | | |StreamName (STREAM): Stream A Stream B Stream A Stream B
75
publishingMetrics.htmlhttps://aws.amazon.com/cloudwatch/pricing/
-
Amazon Kinesis Data Streams CloudWatch KPL
| | -------- --------- | | | |ShardID (SHARD): Shard 0 Shard 1 Shard 0 Shard 1
( Metric Y)
tuple: GLOBAL STREAM SHARD ID
Amazon CloudWatch KPL KPL KPL CloudWatch
N (N 1 60 )
CloudWatch
KPL 1 1 1 CloudWatch
User RecordsReceived
KPL
: Detailed
:
User RecordsPending
KPL PUT
: Detailed
:
76
-
Amazon Kinesis Data Streams CloudWatch KPL
User Records Put
KPL
: Summary
:
User Records DataPut
: Detailed
:
Kinesis RecordsPut
Kinesis Data Streams ( Kinesis Data Streams)
KPL
: Summary
:
Kinesis RecordsData Put
Kinesis Data Streams
: Detailed
:
Errors by Code ErrorCode StreamName ShardId
Kinesis Data Streams API Kinesis Data Streams 1 Kinesis Data Streams
: Summary
:
All Errors
: Summary
:
77
-
Amazon Kinesis Data Streams
Retries per Record 1
()
: Detailed
:
Buffering Time KPL
: Summary
:
Request Time PutRecordsRequests
: Detailed
:
User Records perKinesis Record
Kinesis Data Streams
: Detailed
:
Amazon KinesisRecords perPutRecordsRequest
PutRecordsRequest Kinesis Data Streams
: Detailed
:
User Records perPutRecordsRequest
PutRecordsRequest 2
: Detailed
:
Amazon Kinesis Data Streams IAM
AWS Identity and Access Management (IAM)
AWS AWS AWS
78
-
Amazon Kinesis Data Streams
AWS AWS AWS AWS
Kinesis Data Streams IAM Kinesis Data StreamsAPI AWS
Kinesis Client Library (KCL) AmazonDynamoDB Amazon CloudWatch KCL DynamoDB CloudWatch KCL CloudWatch KCL Kinesis Client Library 1.x (p. 118)
IAM
AWS Identity and Access Management (IAM) IAM
IAM Amazon DynamoDB Amazon DynamoDB IAM Amazon DynamoDB
IAM Amazon CloudWatch Amazon CloudWatch AWS
(p. 79) Kinesis Data Streams (p. 80) Kinesis Data Streams Amazon (ARN) (p. 80) Kinesis Data Streams (p. 80)
IAM 1 JSON
{ "Statement":[{ "Effect":"effect", "Action":"action", "Resource":"arn", "Condition":{ "condition":{ "key":"value" } } } ]}
79
https://aws.amazon.com/iam/https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started.htmlhttps://docs.aws.amazon.com/IAM/latest/UserGuide/UsingIAMWithDDB.htmlUsingIAMWithDDB.htmlUsingIAM.htmlUsingIAM.html
-
Amazon Kinesis Data Streams Kinesis Data Streams
[Effect]: effect Allow Deny IAM API
[Action]: action API [Resource]:
Amazon ARN [Condition]: condition
IAM IAM Policy Generator IAM Policy Simulator
Kinesis Data Streams IAM IAM API Kinesis Data Streams API kinesis: : kinesis:CreateStreamkinesis:ListStreamskinesis:DescribeStream
"Action": ["kinesis:action1", "kinesis:action2"]
Get
"Action": "kinesis:Get*"
Kinesis Data Streams *
"Action": "kinesis:*"
Kinesis Data Streams API Amazon Kinesis API Reference
Kin