© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

AWS Data Transfer ServicesData Ingest Strategies into the AWS Cloud

Ken ChanProduct Business Development Manager

Greater Chinakenchan@amazon.com

Storage is the gravity for cloud applications

Customers have Data Centers

DeployDeploy

Batches and Streams

Direct Connect

Snowball, Snowball Edge,

Snowmobile

3rd Party Connectors

Transfer Acceleration

Storage Gateway Kinesis Firehose

File

Amazon EFS

Block

Amazon EBS (persistent)

Object

Amazon GlacierAmazon S3 Amazon EC2 Instance Store

(ephemeral)

Internet/VPN CloudFront

Long Term Archive

All tiers accessible through a single API

Oldest content trickles down to glacier automatically to save cost

Amazon S3

S3 In-frequent Access

Amazon Glacier

Life

Cyc

le P

olic

ies

OnsiteStorage

Frequently Accessed

New

S3 Life Cycle Policy Use case for Media

Internet/VPN ingest

What is Internet/VPN?

Globally available

Default method of ingesting content into Amazon S3

Simple standards-based (HTTP) connection

Use your existing internet connection

Available in a VPC for VPN connectivity

Acceleration through multipart upload

Data transfer into AWS is free

VPN connections using VPC virtual private gateway•$0.05 per VPN connection-hour•$0.048 per VPN connection-hour for connections to the Tokyo region

How does Internet/VPN ingest work?Accelerate data transfer using

multipart uploadIngest data directly into S3 buckets with existing internet connectivity

S3 bucketAWS Region

and

through the console or API

customer gateway

endpoints

VPN connection

Internet Internet through VPN + VPC

Amazon S3 Transfer Acceleration

What is Transfer Acceleration?

Network- and protocol-based data transfer service

Acceleration of data ingress/egress with S3 buckets

Typically 50% to 300% faster

Feature of S3 enabled at the bucket level

Available in all S3 regions worldwide

No client/server software required

No code changes to your application

No firewall exceptions

Simple pricing model

Ingest & egress with Transfer Acceleration

S3 bucketAWS edge

location

Uploader

Optimizedthroughput!

Uses AWS 59 global edge locations

AWS determines best edge location

Data transfer optimized between edge and customer, and edge and S3

Data is not stored on the edge cache

Customers: Frame.io, Hudl, Viocorp

Problem Statement:• Needed to accelerate customer content ingest into their respective

applications running on AWS• Existing ingest options were proprietary and too expensive

Use of AWS:• S3 and S3 transfer acceleration for massively scalable ingest • S3 for storage, CloudFront and S3 transfer acceleration for ingest

Business Benefits: • Global highly distributed data transport available on demand• Massive scalability and elasticity• Lower TCO for storage and data transport infrastructure

Accelerating media content uploads to their platforms

S3 BucketAWS EdgeLocation

Uploader

OptimizedThroughput!

Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore

500 GB upload from these edge locations to a bucket in Singapore

Tim

e [h

rs]

Public internet

How fast is S3 Transfer Acceleration?S3 transfer acceleration

AWS Direct Connect

What is AWS Direct Connect?

Dedicated, 1 or 10 GE private pipes into AWS

Create private (VPC) or public virtual interfaces to AWS

Reduced data-out rates (data-in still free)

Consistent network performance

At least 1 location to each AWS region

Option for redundant connections

Uses BGP to exchange routing information over a VLAN

Getting Started with DX

Create Connection to issue LOA

LOA

Pass this LOA to our DX partner to get cross connection setup

At the Direct Connect location

CORP

AWS DirectConnect Routers

Customer Router

Colocation

DX Location

Customernetwork`

AWS backbonenetwork

Cross- connect

Customer router

Customer’s network

Demarcation

Dedicated port through Direct Connect partner

CORP

AWS DirectConnect Routers

Colocation

DX Location

Partner network

AWS backbonenetwork

Cross- connect

Customer router

Partnernetwork

Accesscircuit

Demarcation

Partnerequipment

Hybrid cloud storage expansion:Amazon EFS through Direct Connect

“Bursting”File Workloads

Data Migration into EFS

Amazon EFSOn-Premises AWS Direct Connect

AWS Storage Gateway

What is AWS Storage Gateway?

Works with your existing applications

Secure and durable storage in AWS

Low latency for frequently used data

Scalable and cost-effective on-premises storage - $.01/GB written to AWS + S3/Amazon Glacier storage fees

Service connecting an on-premises software appliance with cloud-based storage

Hybrid storage use cases and architectures for AWS Storage Gateway

Enabling cloud workloadsMove data to AWS storage for Big Data, cloud bursting, or migration

Tiered cloud storageEasily add AWS storage to your on-premises environment

Backup, archive, and disaster recoveryCost effective storage in AWS with local or cloud restore

Storage Gateway hybrid storage solutionsEnables using standard storage protocols to access AWS storage services

Customer Premises

StorageGateway

Amazon EBS snapshots

Amazon S3Amazon Glacier

AWS Identity and Access Management (IAM)

AWS Key Management Service (KMS)

AWS CloudTrail

Amazon CloudWatch

Enterprise storage

Devices

Applicationservers

Storage gateway – Files, volumes, and tapes

File gateway NFS (v3 and v4.1) interface **NEW!**On-premises file storage backed by Amazon S3 objects

Volume gateway iSCSI block interfaceOn-premises block storage backed by Amazon S3 with EBS snapshots

Tape gateway iSCSI virtual tape library (VTL) interfaceVirtual tape storage in Amazon S3 and Glacier with VTL management

Detail: AWS File Gateway for S3

NFS Interface Elasticity Amazon S3 Bucket

Easy Integration Cloud ScaleCloud Access

AWS Snowball

What is AWS Snowball? Petabyte-scale data transport

E-ink shipping label

Ruggedized case“8.5G impact”

All data encrypted end-to-end

Rain- and dust-resistant

Tamper-resistant case and

electronics

80 TB10 GE network

AWS storage migration expansion: AWS Snowball

Transfer Capacity Integration Regional

Availability

80TB modelHDFS support3rd party API

HIPAA support

Continue to expand

How it works

How fast is Snowball?• Less than 1 day to transfer 200TB via 3x10G connections with 3

Snowballs, less than 1 week including shipping• Number of days to transfer 200TB via the Internet at typical utilizations

Internet Connection SpeedUtilization 1Gbps 500Mbps 300Mbps 150Mbps

25% 71 141 236 47150% 36 71 118 23675% 24 47 225 157

Customer: Scripps Networks Interactive

Problem Statement:• Need storage platform to manage active archive content• Existing content repository too large to migrate via available

network-based ingest methods

Use of AWS:• S3 and Snowball for massively scalable ingest • S3 for storage, Glacier for content archive• Snowball to securely transport existing media content from on-

premises storage and tape vault

Business Benefits: • Petabyte-scale data transport without increased network costs• Massive scalability and elasticity• Lower TCO for active archive storage

Active archive transport and archival for digital content provider

AWS storage migration expansion:AWS Snowmobile

Hybrid cloud storage expansion: AWS Snowball Edge

On-premises Capacity

On-premises Integration

On-premises Compute

Clustered local storage100TB capacity

NFS and S3-compatible endpoint

AWS Lambda support for local transformation

AWS Snowball Edge

Integrated Storage and Compute

Applications

STG214

Storage Ecosystem Partners

Backup to AWS approaches

Amazon S3

Amazon GlacierAWS

DirectConnect

InternetAmazon S3-IA

Applicationservers

Cloud gateway

Local disk

Mediaserver

Cloud gateway

HTTPS/API

Applicationservers

Backup SW cloud connector

Local diskMedia

server with cloud

connector

HTTPS/API

Hybrid cloud storage ecosystem

BackupAWS Storage Gateway VTL

Direct to Amazon S3

File Systems

Object Storage

Remember to complete your evaluations!

Thank you!