Big “Serverless” Data - Big Data Days · Serverless in big data processing Amazon Kinesis Video...

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Eric JohnsonSenior Developer Advocate - Serverless

AWS

@edjgeek

Big “Serverless” DataPowering Big Data with Serverless

Background Image by Эдуард Ризванов from Pixabay

https://pixabay.com/users/8370633-8370633/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=3227572

https://pixabay.com/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=3227572


Who am I?

• Sr. Developer Advocate – Serverless, AWS

• Serverless / Tooling / Automation Geek

• Software Architect / Solutions Architect

• Husband to Brigitte

• Father to Noah, Jake, Owen

Sophie Anne, & Gracie Mae

• Music lover

• Pizza / Diet Dr. Pepper fanatic


Why are we here?


Serverless in big data processing

Amazon Kinesis Video Streams

Amazon KinesisData Streams

Amazon Kinesis Data Firehose

Amazon KinesisData Analytics

Amazon Athena AWS Lambda Amazon SimpleStorage Service

Amazon DynamoDB

Understanding the role Serverless plays in Big Data


Agenda

Ingestion

Real-time processing

Real-time analytics

Post processing


What is serverless?

No infrastructure provisioning, no management

Automatic scaling

Pay for value Highly available and secure


Ingestion


Ingesting data at scale


Amazon KinesisData Streams


Video Ingestion Data Ingestion


Video ingestion

• Fully managed infrastructure that scales to load

• Offers SDK in C++ and Java

• Supports live and on-demand playback of streams

• Durable storage using Amazon S3

• Works with many forms of time encoded data

• Supports multiple time code based formats



Data ingestion – Kinesis Data Streams

• Uses shards to scale• 1 MB or 1000 records /second/shard ingress

• 2 MB/second/shard egress

• Works with Kinesis Data Analytics

• Can support connected consumers for enhanced fanout

• Can store data up to 168 hours (7 days)

Amazon Kinesis Data Streams


Data ingestion – Kinesis Firehose

• Auto-scales to meet load• Different regions have different capacity

• US East: 5,000 records/second, 2,000

transactions/second, and 5 MiB/second.

• Works with Kinesis Data

Analytics

• Can transform data before

delivery to target

• Stores data up to 24 hours

on failed delivery

Amazon Kinesis Firehose


Data ingestion – Kinesis Firehose

Data Sources Targets

• Firehose PUT APIs• Amazon Kinesis

Agent• AWS IoT• CloudWatch Logs• CloudWatch Events

• Amazon S3• Amazon Redshift• Amazon

Elasticsearch Service


Kinesis Data Stream

Kinesis Data Streams vs. Kinesis Firehose

Kinesis Firehose

Amazon KinesisData Stream

Data Producers010001110010100

01000111001001101010100010010100010100

01000100101110100

010010100010100

010010100010100010010100010100

010010100010100010010100010100

Data Producers


01000111001001101010100010010100010100

010010100010100

010010100010100010010100010100

01000111001001101010100

01000111001001101010100

01000111001001101010100

01000111001001101010100

010001101010100

010001101100


Kinesis Firehose

Kinesis Firehose

Data Producers


01000111001001101010100010010100010100

010010100010100

010010100010100010010100010100

01000111001001101010100

01000111001001101010100

01000111001001101010100

01000111001001101010100

010001101010100

010001101100

Use Kinesis Firehose when you need:• Ability to transform data in the stream• Auto scaling for unpredictable load• Multiple targets for final data


Kinesis Data Stream

Kinesis Data Streams


Data Producers010001110010100

01000111001001101010100010010100010100

01000100101110100

010010100010100

010010100010100010010100010100

010010100010100010010100010100

Use Kinesis Data Streams when:• You have semi-predictable traffic• You need to perform real-time action on

data in the stream


Real-time

processing


Kinesis Data Stream + Lambda


Data Producers

Lambda function

Lambda function

Lambda function

Amazon DynamoDB


AWS IoT Core

Lambda services handles intermittent pollingvia GetRecords API


Kinesis Data Stream + Lambda


Data Producers

Lambda function

Lambda function

Lambda function

Amazon DynamoDB


AWS IoT Core

Lambda services handles intermittent pollingvia GetRecords API

All applications share 2 MB/second/shard egress


Kinesis Data Stream + Enhanced Fanout + Lambda


Data Producers

Lambda function

Lambda function

Lambda function

Amazon DynamoDB


AWS IoT Core

Functions triggered by consumers



Data Producers

Lambda function

Lambda function

Lambda function

Amazon DynamoDB


AWS IoT Core

Functions triggered by consumers

Each consumer provides an individual 2 MB/second/shard egress

Kinesis Data Stream + Enhanced Fanout + Lambda


Video Processing

Amazon Kinesis

Video Streams

Amazon Rekognition video

Amazon SageMaker

S3 Bucket


Video Processing

Amazon Kinesis

Video Streams


Amazon SageMaker

Real time analysis and machine learning

S3 Bucket


Video Processing

Amazon Kinesis

Video Streams


Amazon SageMaker

Real time analysis and machine learning

S3 Bucket

HLS Compatible live oron-demand playback


Video Processing

Amazon Kinesis

Video Streams


Amazon SageMaker

Real time analysis and machine learningHLS Compatible live oron-demand playback

S3 Bucket

Near real-time processing


Real-time

analytics


Amazon Kinesis Data Analytics

• Built-in functions to filter, aggregate, and transform streaming data

• Processes streaming data with sub-second latencies

• Build SQL queries that perform joins, aggregations over time windows and filters

• includes open source libraries based on Apache Flink that enable you to build an application in hours instead of months



Real-time analytics




Stream source can be Kinesis Data Stream or Firehose


Inside Kinesis Data Analytics

Stream

data

-- Create Fail Stream --CREATE OR REPLACE STREAM "FAIL_STREAM" (

sensorId INT,currentTemperature INT,status VARCHAR(10)

);

CREATE OR REPLACE PUMP "FAIL_STREAM_PUMP" AS INSERT INTO "FAIL_STREAM"SELECT "sensorId", "currentTemperature", "status"FROM "SOURCE_SQL_STREAM_001"WHERE "status" SIMILAR TO '%FAIL%';

-- Create Warn Stream --CREATE OR REPLACE STREAM "WARN_STREAM" (


);

CREATE OR REPLACE PUMP "WARN_STREAM_PUMP" AS INSERT INTO "WARN_STREAM"SELECT "sensorId", "currentTemperature", "status"FROM "SOURCE_SQL_STREAM_001"WHERE "status" SIMILAR TO '%WARN%';

FAIL_STREAM

WARN_STREAM



Stream

data



);




);


FAIL_STREAM

WARN_STREAM

Use SQL or Apache Flink to filter data



Stream

data



);




);


FAIL_STREAM

AWS Lambda

• Alert• Diagnose• Remediat

e



Stream

data



);




);


WARN_STREAM


• Dashboards

• Consumer response


Real-time analytics





AWS Lambda

FAIL_STREAM

WARN_STREAM

What about the raw data?


Real-time analytics






AWS Lambda

FAIL_STREAM

WARN_STREAM

Raw Data Archive


Post processing


Serverless data storage

Amazon SimpleStorage Service

Amazon DynamoDB

Amazon Timestream

AmazonQuantum Ledger

Database

Amazon CloudWatc

h





Serverless data storage


Amazon DynamoDB

Amazon Timestream


Database

Amazon CloudWatc

h




How you need to process

your data determines

where to store it


Serverless storage options


Amazon DynamoDB

Amazon Timestream


Database

Amazon CloudWatc

h

• Immutable and transparent

• Cryptographically Verifiable

• Object storage• Unstructured

data

• Structured data• Alerting built in

• NoSQL• Key value or

document data

• Time series database


Post processing – Serverless Tools

Amazon AthenaQuery S3 data with standard SQL expressions

Amazon S3 SelectRetrieve subsets of object data, instead of the entire object.

AWS GlueExtract, transform, and load (ETL) service that works across multiple services.


AWS Glue

Bucket

Bucket

Bucket

Bucket

Bucket

DynamoDB TableDynamoDB

Table

DynamoDB Table

DynamoDB Table

DynamoDB Table

Other non-serverless services• MariaDB

• Microsoft SQL Server• MySQL• Oracle• PostgreSQL

Critical data can be stored in many places


AWS Glue

Bucket

Bucket

Bucket

Bucket

Bucket


Table

DynamoDB Table

DynamoDB Table

DynamoDB Table

Crawler Data Catalog




AWS Glue

Bucket

Bucket

Bucket

Bucket

Bucket


Table

DynamoDB Table

DynamoDB Table

DynamoDB Table




What it is doing • Classifies data to determine the format,

schema, and associated properties of the raw data

• Groups data into tables or partitions – Data is grouped based on crawler heuristics.

• Writes metadata to the Data Catalog


AWS Glue

Bucket

Bucket

Bucket

Bucket

Bucket


Table

DynamoDB Table

DynamoDB Table

DynamoDB Table




This catalog contains meta-data about the data stores. How do I get the data itself in a meaningful way?


Enter: AWS Athena

Bucket

Bucket

Bucket

Bucket

Bucket


Table

DynamoDB Table

DynamoDB Table

DynamoDB Table




Amazon Athena


AWS Athena

Bucket

Bucket

Bucket

Bucket

Bucket


Table

DynamoDB Table

DynamoDB Table

DynamoDB Table


Other non-serverless services• Amazon Aurora

• MariaDB• Microsoft SQL Server• MySQL• Oracle• PostgreSQL

Athena queries Glue Data Catalog

Glue returns data from data source Amazon Athena


AWS Athena


Question

I have HUGE compressed CSV files

stored on Amazon S3.

How do I get small bits of data without

reading the entire file?


Enter: Amazon S3 Select

import boto3

s3 = boto3.client('s3’)

r = s3.select_object_content(

Bucket='jbarr-us-west-2’,

Key='sample-data/airportCodes.csv’,

ExpressionType='SQL’,

Expression="select * from s3object s where s.\"Country (Name)\" like '%United States%’”,

InputSerialization = {'CSV': {"FileHeaderInfo": "Use"}},

OutputSerialization = {'CSV': {}}, )

for event in r['Payload’]:

if 'Records' in event:

records = event['Records']['Payload'].decode('utf-8’)

print(records)

elif 'Stats' in event:

statsDetails = event['Stats']['Details’]


Before S3 Select

Lambda function

Bucket

001010010110110010010101010010100111001001001101100101100101010001001001111001000011001001001111110010000000110110010100110000010100101101100100101010100101001110010010011011001011001010100010010011110010000110010010011111100100000001101100101001100000101001011011001001010101001010011100100100110110010110010101000100100111100100001100100100111111001000000011011001010011000001010010110110010010101010010100111001001001101100101100101010001001001111001000011001001001111110010000000110110010100110000010100101101100100101010100101001110010010011011001011001010100010010011110010000110010010011111100100000001101100101001100000101001011011001001010101001010011100100100110110010110010101000100100111100100001100100100111111001

0000000110110010100110

Entire file returned


001010010110110010010101010010100111001001001101100101100101010001001001111001000011001001001111110010000000110110010100110000010100101101100100101010100101001110010010011011001011001010100010010011110010000110010010011111100100000001101100101001100000101001011011001001010101001010011100100100110110010110010101000100100111100100001100100100111111001000000011011001010011000001010010110110010010101010010100111001001001101100101100101010001001001111001000011001001001111110010000000110110010100110000010100101101100100101010100101001110010010011011001011001010100010010011110010000110010010011111100100000001101100101001100000101001011011001001010101001010011100100100110110010110010101000100100111100100001100100100111111001

0000000110110010100110

After S3 Select

Lambda function

Bucket

Parsed value returned

Up to 400% faster and 80% cheaper


Questions?https://pixabay.com/illustrations/questions-font-who-what-how-why-2245264/


Eric Johnson@edjgeek

Image Source: https://pixabay.com/illustrations/thank-you-polaroid-letters-2490552/

Big “Serverless” Data - Big Data Days · Serverless in big data processing Amazon Kinesis Video...

Documents

Transcript of Big “Serverless” Data - Big Data Days · Serverless in big data processing Amazon Kinesis Video...