Amazon CloudFront - Developer Guide - AWS · Amazon CloudFront Developer Guide
Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports
-
Upload
blazeclan-technologies-private-limited -
Category
Technology
-
view
2.702 -
download
2
description
Transcript of Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports
![Page 1: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/1.jpg)
Featuring Our
Latest
Reports
![Page 2: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/2.jpg)
AGENDA
2
Introduction To Amazon S3 & CloudFront
Log Processing Using AWS Vs Traditional ways
Log Processing With Cloudlytics- Big Data Approach
Cloudlytics Use cases
Cloudlytics Reports & Live Demo
![Page 3: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/3.jpg)
What is Amazon Simple Storage Service or S3?
• Amazon Simple Storage Service is storage for the
internet
• Stores 2 trillion+ objects
• 1.1 million requests per second at peak
• Each time a request is
made to access a file on S3,
an entry to log file is created
• Average size of each
log entry 550B
3
Image Courtesy: http://threatpost.com/files/2013/03/
![Page 4: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/4.jpg)
What is Amazon CloudFront?
4
1 5
4
3
2
Amazon CloudFront is a
web service for content
delivery
CloudFront
decreases
latencies
for object
downloads
and streamsSupports static
and dynamic
content, including
web pages
Each time a request
is made to access a file
on CloudFront, an entry
to log file is created
Average size of each
log entry 650B
![Page 5: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/5.jpg)
Information Hidden in S3 & CloudFront Logs
5
-
Object Details Download Status
Download /Streaming Time
Number of
Bytes
Transferred
Details about Edge Locations
IP Address of the Requester
Referrer LinkTime Taken to Download Object(S3)
Details about Play, Pause, Stop (Streaming Content on CloudFront)
![Page 6: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/6.jpg)
Uncover the Hidden Information
• Generating logs for Amazon S3 &
CloudFront logs is optional
• Log files are stored in S3 buckets
• CloudFront Log files are compressed
and stored in .gz format
• A log file is generated every hour, but we
have seen varied patterns with multiple files
generated every hour
• No ready solution from AWS to process
these log files
6
S3 logs CloudFront
logs
Logs Stored
in S3
Logs Analyzed by
CLOUDLYTICS
![Page 7: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/7.jpg)
Image Courtesy: www.fao.org
Traditional Log Processing
• Extract data from the source using an ETL tool
• Transform data and load in a data-warehouse
• Takes days to process a few GBs of
log files using traditional hardware
• Alternately use a Hadoop distribution
to process logs
• But maintaining a Hadoop cluster is a huge overhead
7
![Page 8: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/8.jpg)
Log Processing with Cloudlytics
• Cloudlytics - Analyze your Amazon S3 &
CloudFront Logs
• Detailed analysis of your S3 & CloudFront
access patterns
• Dynamic Graphs to get a 360 degree
perspective
• Scalable & Reliable service built using
Amazon EMR & RedShift
• Pay as you go
8
![Page 9: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/9.jpg)
Log Processing – Big Data Approach
• Cloudlytics extracts log files stored
in S3 buckets
• Processes the log files to
transform information
• Stores the processed data in
a data-warehouse
• Graphical and tabular reports generated from data-warehouse
9
![Page 10: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/10.jpg)
10
Cloudlytics Use Cases
![Page 11: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/11.jpg)
Independent Software Vendors (ISVs)
• ISVs distribute downloadable software to
end users across the globe
• ISVs need to ensure that downloads are
fast, helps improve user experience
• ISVs need to track each download for
success and failure
• Identify broken links on the website, helps improve user
experience
• Identify the most popular downloads, focus on popular products
• Identify spam attacks, help reduce bandwidth costs
11
![Page 12: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/12.jpg)
E-Learning Companies
• E-Learning companies distribute educational
content in multiple formats
(ebooks, audio, video etc.)
• Figure out the most popular content
• Figure out end user engagement by querying
number of events per request (Play, Pause, Stop)
• Get a breakup of requests by Operating System and Devices,
develop content creation for specific platforms
12
Image Courtesy: http://www.elifescience.in/images
![Page 13: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/13.jpg)
Media Organizations
• Large number of media assets
available online
• Content does not go any changes
during it’s life cycle
• Some content is extremely popular
while others do not get any views
• Identify the most popular content,
set caching mechanism
• Figure out end user engagement by querying number of events per
request (Play, Pause, Stop)
• Identify the edge locations from which the content is downloaded
the most, optimize billing using CloudFront pricing Class
13
![Page 14: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/14.jpg)
Cloudlytics REPORTS
14
Which Reports Resonate
with Your Business Needs?
![Page 15: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/15.jpg)
Geographic Reports
15
![Page 16: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/16.jpg)
Browser & OS Statistics
16
![Page 17: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/17.jpg)
Detail IP Monitoring
17
![Page 18: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/18.jpg)
Timeline Charts
18
![Page 19: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/19.jpg)
The TOP 10
19
![Page 20: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/20.jpg)
Edge Location Traffic
20
![Page 21: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/21.jpg)
DEMO
21
Image Courtesy: SourceKeyit.com
Let’s Look at
Cloudlytics in
Action
![Page 22: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/22.jpg)
How to get started?
22
So Where do you
Get Started ??
Image courtesy: http://blogs.position2
.com/best-of-the-week-august-24-2012
![Page 23: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/23.jpg)
Get Started in 3 Easy Steps
23
Configure your Log
buckets
Register for Free
Analyze & Generate
Reports
![Page 24: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/24.jpg)
Pricing
• No upfront costs
• Register for free and analyze upto 25MB logs/month FREE
• Pay only for the amount of logs you subscribe
24
![Page 25: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/25.jpg)
Advantage - Cloudlytics
• Scalable & Reliable
• Developed using Amazon Web Services
tools like Amazon EMR & Amazon Redshift
• Developed by BlazeClan Technologies,
leading Consulting Partner with
Amazon Web Services
• Pay as you go service with no contracts and no lock-ins
25
![Page 26: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/26.jpg)
To sum it Up
26
Identifying Popular
Downloads & Streams
Get Geographical Distribution
of Downloads & Streams
Improve User Experience by
Calculating & Optimizing Latencies
Identify Edge Location
Traffic and optimize AWS Billing
Identify Spam
Attacks
Get Streaming Pattern
for Video Content
![Page 27: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/27.jpg)
Our Global User Reach
27
80+ Users and
Counting!
![Page 28: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/28.jpg)
Coming Soon!
28
Image Courtesy: http://www.trophies.com/coming-soon/
![Page 29: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/29.jpg)
Upcoming Webinar
29
Check out Our
Upcoming Webinars
@ blazeclan.com/webinars
![Page 30: Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports](https://reader034.fdocuments.net/reader034/viewer/2022042713/54799923b4af9faa158b4873/html5/thumbnails/30.jpg)
Visit :Cloudytics.com
Contact: [email protected]
Follow Us On :
Our Blog : http://blog.blazeclan.com/
Thank you