AWS Machine Learning · Run the FV3GFS global model on Amazon Web Services, at full resolution and...
Transcript of AWS Machine Learning · Run the FV3GFS global model on Amazon Web Services, at full resolution and...
1© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved | 1© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Where and when?
Matthew Greensmith, Specialist SA
AWS Machine Learning
2© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Weather and climate on AWS - Joint Center for Satellite Data Assimilation (JCSDA)Joint Effort for Data Integration (JEDI) is a next-generation data assimilation (DA) system fornumerical weather prediction (NWP) that is capable and flexible enough to use for both researchand operations. Run the FV3GFS global model on Amazon Web Services, at full resolution andwith the pre-operational configuration.48-node (1,728-core) compute clusters on AWS.
3© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
4© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Our mission at AWS
Put machine learning in the hands of every developer
5© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Where ML doesn’t work
How to recognize AI Snake Oil - https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf
• Statistical Analysis
• What has happened
• Trends
• No outliers
6© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Agenda
3 ways we are seeing ML in weather and climate
• Meta analysis and model tuning
• Effect prediction
• Device integration – efficiency of data collection
7© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Meta Analysis
Scher S, Messori G. Predicting weather forecast uncertainty with machine learning. Q J R Meteorol Soc. 2018;144:2830–2841. https://doi.org/10.1002/qj.3410
“…networks are shown to be useful predictors offorecast uncertainty.”
8© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Extending this to post processing
Rasp S, Lerch S. Neural networks for post-processing ensemble weather forecasts. Monthly Weather Review 2018, 146, 3885-3900.
“Our neural network models significantly outperform state-of-the-art post-processing techniques while being computationally more efficient.”
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
ML Analysis of prediction effect : Jupiter Intelligence
Machine Learning for improving Disaster Management and Response using AWS
Hurricane Irma predicted path Hurricane Irma real path
Source: Weather Channel
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Importance of Social Media Data in Disasters
• Micro-blogging data from crowds
of non-professional participants
during disasters are of significant
value.
Researchers assert that bystanders “on the ground are uniquely positioned to
share information that may not yet be available elsewhere in the information
space...” [Starbird et al., 2010].
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Using locations identified by Comprehend to track hurricane path
Source: Wikipedia
Source: Weather Channel
13© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
IoT is HARD!
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AmazonS3 Bucket
Amazon Redshift
Amazon SageMaker
MQTT
Endpoints Gateway/PLCCloud Enterprise
Applications
Device shadow
RulesEngine
AWS IoTCore
Certificate Authority
DeviceShadow
AWS IoT Greengrass
LambdaFunctions
MessageRouter
Long-range Comms
Amazon FreeRTOS
Local Resources
IoT SDK
AWS IoT Device Management
AWS IoTAnalytics
AWS
Enterprise Users
MQTT
UDP/IP
IoTUsers
EdgeUsers
Cert
WiFi
All AWS
Over-the-air (OTA) Updates
Analytics Data Store
Data Pipelines
Templated Reports
Local Resources
IoT Solution Map
Batch Fleet Provisioning
Real-time Fleet Index & Search
Corp Apps
AWS IoTDevice Defender
Ad-hoc & In-depth Analysis
Risk Mitigation
Detection Profiles
Alerts
Scheduled or Ad-hoc
Audit
MQTT
Things
OTA
Amazon FreeRTOS
Message Broker
IntegratedClient
OTA
BLE
Timeseries
SnowballEdge
AWS IoT SiteWise
3rd Party
On-premises Historian
Secret Manager
Event Detection
OPC Server
AWS IoTEvents
Local Comms
Custom Gateway
Amazon EMR
AmazonQuickSight
AmazonS3
AWS Lambda
Amazon Kinesis
AWS Things Graph
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Data and State Sync Security
Over the Air UpdatesConnectors
Operate devices offline & synchronize data
when reconnected
LocalActions
Simplify device programming
with AWS Lambda
Mutual authentication &
authorization between cloud
and devices
Easily update AWS IoT Greengrass
Core
Machine Learning Inference
Perform ML Inference
locally
Local Resource
Access
AWS Lambda functions can
access & use local resources of a given device
Extend edge devices with
connections to external services
LocalMessages
and Triggers
Enable device communication without a cloud
connection
Secrets Manager
Deploy secrets to edge devices
AWS IoT Greengrass
16© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Bringing machine learning to all developers
A M A Z O N S A G E M A K E R
Collect and prepare
training data
Choose and optimize your ML algorithm
Set up and manage environments
for training
Train and tune model
(trial and error)
Deploy model in
production
Scale and manage the production environment
17© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Collect and prepare
training data
Choose and optimize your ML algorithm
Set up and manage environments
for training
Train and tune model
(trial and error)
Deploy model in
production
Scale and manage the production environment
Pre-built notebooks for
common problems
Built-in, high performance algorithms
One-click training Optimization
One-click deployment
Fully managed with auto-scaling, health checks,
automatic handling of node failures,
and security checks
Bringing machine learning to all developers
A M A Z O N S A G E M A K E R
18© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
FRAMEWORKS INTERFACES INFRASTRUCTURE
AI Services
Broadest and deepest set of capabilities
T H E A W S M L S T A C K
VISION SPEECH LANGUAGE CHATBOTS FORECASTING RECOMMENDATIONS
ML Services
ML Frameworks + Infrastructure
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D& C O M P R E H E N D
M E D I C A L
L E X F O R E C A S TR E K O G N I T I O NI M A G E
R E K O G N I T I O NV I D E O
T E X T R A C T P E R S O N A L I Z E
Ground Truth Notebooks Algorithms + Marketplace Reinforcement Learning Training Optimization Deployment HostingAmazon SageMaker
F P G A SE C 2 P 3& P 3 D N
E C 2 G 4E C 2 C 5
I N F E R E N T I AG R E E N G R A S S E L A S T I CI N F E R E N C E
D L C O N T A I N E R S
& A M I s
E L A S T I C K U B E R N E T E S
S E R V I C E
E L A S T I C C O N T A I N E R
S E R V I C E
19© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
1
Createthe loop
Connect technology initiatives with business outcomes
2
Assess your structured and unstructured data sources
Advance yourdata strategy
?
3
Put machine learning in the hands of your developers
Organize for success
S E T T I N G Y O U R O R G A N I Z A T I O N U P F O R S U C C E S S
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Collaborating on scientific data in the cloud
NOAA- NEXRAD on AWS S3, usage increased 2.3x
greater scientific impact
Do go to aws.amazon.com/earth/
22© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
ML.awsThank you!