Best Practices for Sharing Imageryusing Amazon...
Transcript of Best Practices for Sharing Imageryusing Amazon...
![Page 1: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/1.jpg)
Best Practices for Sharing Imageryusing Amazon Web Services
Peter Becker
![Page 2: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/2.jpg)
• Making Imagery Accessible• Store massive volumes of imagery on inexpensive cloud storage• Use elastic compute for image services
• Focus on Imagery- Large in Volume- Static- Basis for visualization and Analytics
• Note:• Implementation of these workflows currently require additional tools and a component
that is not publically available. Details for access will be provided at the end
Objectives
www.esri.com/landsatonaws
![Page 3: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/3.jpg)
What’s Changed
NAS
StandaloneDirect Access
Enterprise
Cloud
NFS Http
Simple/Object Store
Elasticity
![Page 4: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/4.jpg)
Amazon Storage Options
Ephemeral(80GB)
$0 (inc. with EC2)
NFS
EBS (Elastic Block Storage)1TB/Disk ($100/TB/Month)
Fileshare
NFS
![Page 5: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/5.jpg)
Amazon Storage Options
Ephemeral(80GB)
$0 (inc. with EC2)
NFS
EBS (Elastic Block Storage)1TB/Disk ($100/TB/Month)
S3
Simple Storage$30/TB/Month
99.999999999% Durability99.99% Reliability/year
≈10-50MB/s
Http
![Page 6: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/6.jpg)
Azure Storage Options
TemporaryDisk
(80GB)
NFS+REST
Azure Files5TB, max 60MBps/Share
Azure Blobs500TB/container
max 60MBps/Blob
REST
Blobs
![Page 7: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/7.jpg)
Optimizations
Suitable data structure:- Enable access in single request
- Metadata- Index- Data Tiles
Cache to ephemeral/temporary storage
ObjectStore
• Use Inexpensive Scalable Storage- Minimize Number of requests- Minimize Size of requests- Minimize Repeat requests
• Enable Elasticity• Enable Caching
Http
Driver Driver Driver Enable http access
![Page 8: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/8.jpg)
File Format Considerations / Optimization
Metadata
Metadata
IndexMetadata
Bad
Index
Metadata
Pyramid
Pyramid
Enable separate caching of metadata and index
Non Optimum Access Enables partial
Access
Striped TIF Tiled TIF MRF
Index
Metadata
• Tiling of imagery – Enables partial access• Compression – Reduce storage and transfer – Weigh against additional compute• Data access complexity – Reduce subsequent requests
separCache Locally
Good Optimum
![Page 9: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/9.jpg)
MRF (Meta Raster) Format
• NASA designed format• Split raster into three (or more) files
- MRF – Includes links to Index and DataFile- Index – Simple index to tiles- DataFile – Data file- Auxiliary files – e.g. Aux.xml
• Open Source Implemented in GDAL• Multiple compression options• Multiple modes: Static, Split, Cloned, Caching
• MRF file is seen as a Raster Dataset in ArcGIS• MRF file and index is small and can easily be on a local file system• Data can be on slower storage
Index
Metadata
Pyramid
Datafile
![Page 10: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/10.jpg)
Storage Formats & Compression
• Raw/Striped TIF Sequential access / No compression• Tiled GeoTIF Tiled Access
- Compression Options- None- Lossy JPEG………...…..Fast Lossy compression 8 & 12bit- Lossless Deflate/LZW…Generic lossless compression
• JPEG2000 Tiled Access- Compression Options
- Lossy…………………….Computationally expensive- Lossless…………………Computationally expensive
• MRF Tiled Access- Compression Options
- As geoTIF – JPEG,Deflate- PNG – Generic lossless- LERC – Controlled Lossy – Very Fast – Optimum for higher bit depth and catagorical
Advantages/Disadvantages
![Page 11: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/11.jpg)
Converting Data
• ArcGIS CopyRasters, GDAL_Translate/GDALaddo- Convert rasters, but don’t copy associated metadata
• OptimizeRasters- Python Script (Calls ArcGIS Version of GDAL)
- From any GDAL supported format to MRF or TiledTIF- Parameters to define compression, tilesize etc- Currently command line tool
![Page 12: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/12.jpg)
Getting Imagery to and from the Cloud
• AWS command line• CloudBerry, FileZilla, ….• OptimizeRasters
- Copy and conversion in parallel- Option to creates Cloned MRF, Caching MRF- Check success and retries if necessary
• Import/Export- Ship hard drives for large projects
• Note: File Names on S3 are case sensitive
•
![Page 13: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/13.jpg)
Conversion and UploadingUsing OptimizeRasters
Demo
![Page 14: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/14.jpg)
AccessingCloud Imagery
![Page 15: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/15.jpg)
ArcGIS Prerequisites
• ArcGIS typically access files through a file system• Cannot access files in S3 directly
• Use MRF Driver to point to MRF file that points to data on S3• MRF Driver is custom GDAL Driver for ArcGIS• Available in 10.3.1* (with issues)• Best to download – See later
• Note: Following can be used to Optimize Enterprise implementations as well• Works with data on S3, but also from any local storage
![Page 16: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/16.jpg)
ArcGIS MRF Driver
• Special MRF driver for ArcGIS• Incorporated into ArcGIS 10.3.1* Desktop & Server• 4 Modes
StaticMRF
IndexMetadata
Datafile One Directory
![Page 17: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/17.jpg)
ArcGIS MRF Driver
• Special MRF driver for ArcGIS• Incorporated into ArcGIS 10.3.1* Desktop & Server• 4 Modes
StaticMRF SplitMRF
IndexMetadata
DatafileIndex
Metadata FastExpensiveStorage
SlowCheapStorage
Datafile
![Page 18: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/18.jpg)
ArcGIS MRF Driver
• Special MRF driver for ArcGIS• Incorporated into ArcGIS 10.3.1* Desktop & Server• 4 Modes
StaticMRF SplitMRF ClonedMRF
IndexMetadata
DatafileIndex
Metadata FastExpensiveStoragee.g. Ephemeral
SlowCheapStoragee.g. S3
IndexMetadata
Datafile
Cache
IndexMetadata
VSI Curl / Http
Datafile
![Page 19: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/19.jpg)
ArcGIS MRF Driver
• Special MRF driver for ArcGIS• Incorporated into ArcGIS 10.3.1* Desktop & Server• 4 Modes
StaticMRF SplitMRF ClonedMRF CachingMRF
IndexMetadata
DatafileIndex
Metadata
Datafile
IndexMetadata
Datafile
Cache
Cached File
Any GDALReadable file
IndexMetadata
XXX File
Cache
VSI Curl / Http
IndexMetadata
VSI Curl / Http
![Page 20: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/20.jpg)
Creating Local MRF Files that Point to Source
• Use OptimizeRasters- Creates MRF files on local file system (Ephemeral or EBS)- MRF points to Index and Data Files- Copies Auxiliary files for RasterType- Note MRF files can have any extension (incl. .tif)
• If source is MRF – Use ClonedMRF- Adds caching optimization
• If source is other format – Use CachingMRF- Enables source to be any Image on S3 (or SAN etc)
• Define Cache location as Ephemeral
IndexMetadata
Datafile
Cache
IndexMetadata
XXX File
Cache
IndexMetadata
Auxiliary Files
Auxiliary Files Auxiliary Files
OptimizeRastes
Auxiliary Files
![Page 21: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/21.jpg)
Creating and UsingCachingMRF
Demo
![Page 22: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/22.jpg)
Creating and Using CachingMRF
![Page 23: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/23.jpg)
Creating Mosaic Dataset
• Use Standard Workflow- Create Mosaic Dataset- Add Rasters- Define and Build Overviews- Set Properties- Perform QC- Publish
• More Advance Workflow- Create Source Mosaic Dataset for each project
- As for Standard Workflow- Create Derived Mosaic Dataset
• For more details see: Image Management Workflows- Resource Center landing page http://esriurl.com/6005
- Guidebook in Help System http://esriurl.com/6007
- ArcGIS Online Group http://esriurl.com/6539
![Page 24: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/24.jpg)
Handling Overviews
• Create Overviews on Ephemeral or EBS • Use OptimizeRasters to copy to S3• Delete Overviews• Use Optimize Rasters to copy back MRF files• Repair paths (if necessary)
• (If very large can generate on EBS and copy MRF back to Ephemeral)
![Page 25: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/25.jpg)
Publishing Image Services
• For initial testing use Standard Workflow- Right click Publish as Image Service
• Publishing can be automated
![Page 26: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/26.jpg)
Configuring ArcGIS Server for Imagery in AWS
• Use AutoScaling Group
• Use Silo configuration of ArcGIS Server (each server is its own site)
• Load Balancer – User Regular Amazon Elastic Load Balancer (ELB)
- Turn On Sticky session on using Duration-Based Session Stickiness
- Listen on ports 80 (http) & 443 (https)- Install SSL certification on port 443- Forward Port 80 (ELB) -> 6080 (ArcGIS Server)- Forward Port 443 (EBL) -> 6443 (ArcGIS Server)
• On ArcGIS server turn on http & https
• Create Master Server
• Create AMI of Master Server
• Config AMI to Autoscaling group
• Scale out and scale in based on CPU Load (typically CPU Load >80% or <20% for 5min)
• Set Min and Max Instances
![Page 27: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/27.jpg)
Publish Elastic Image Services• Prepare Base
- Delete all Cache files- ZIP MD and MRF files- Store on S3
• For each machine- Use Deployer Script / Cloud Formation
Template to Start up- Pull Mosaic Datasets, MRF overviews,
MRF source rasters from S3 in a zip file.
• Above can be automated with Cloud Formation templates available from Professional Services
![Page 28: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/28.jpg)
Advanced Techniques
• S3 Security- Use ACL to define Public Read- Else use Bucket Policy to define which IPs etc. can read
• Embed MRF files into Mosaic Dataset- MRF files fully encapsulated in Mosaic Dataset
• PostgreSQL- Multiple machines from same Mosaic Dataset (dynamic updates)
![Page 29: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/29.jpg)
Getting Access
• Currently the scripts are in Prototype state• To get access to OptimzeRasters, MRF Drivers, PPT and doc:
- Send email to - [email protected] Provide GitHub ID- Name & email- Brief explanation of project goals
![Page 30: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/30.jpg)
Thank you…
• Please fill out the session survey in your mobile app• Select• [Best Practices for Sharing Imagery using
Amazon Web Services]in the Mobile App
- Use the Search Feature to quickly find this title
• Click “Technical Workshop Survey”• Answer a few short questions and enter any comments
Note: Incorrect time
![Page 31: Best Practices for Sharing Imageryusing Amazon …proceedings.esri.com/library/userconf/proc15/tech-workshops/tw_398...Best Practices for Sharing Imagery ... Azure Storage Options.](https://reader031.fdocuments.net/reader031/viewer/2022013006/5b2844a37f8b9ad0608b465c/html5/thumbnails/31.jpg)