Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing...
Transcript of Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing...
![Page 1: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/1.jpg)
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 1
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
“cloud”Accessing
cloud - hostedimage data
Day 3. Standard image file format for sharing big image data in the cloud
![Page 2: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/2.jpg)
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 2
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Large data needs chunks & pyramids to be accessible.
Monolithic formats are difficult to access remotely.
Current formats for chunked access tends to be monolithic.
![Page 3: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/3.jpg)
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 3
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Single TIFF file
437 GB
Faas et al. (idr0053, CC BY-NC-SA 3.0)J Cell Biol (2012)921600 x 380928 pixels
Use cases:
1. Sharing beyond current limits, e.g., downloading TBsfrom OMERO.
2. Remotely sharing withouta server.
![Page 4: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/4.jpg)
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 4
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Remote access
…
21 chunks in the browser
Single TIFF file
437 GB
Remote copy
Local copy
Chunk size:
1024 x 1024(1 MB)
Another 21 chunks
Monolithic format Next-generation file format (NGFF)351 GB 88 GB 21 MB22 GB
![Page 5: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/5.jpg)
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 5
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Maturity Expressivity Multi-resolution Performance
TIFF Ubiquitous 2D tiles TIFF spec Linear slowdown
HDF Well-supported 3D+ chunks No spec yet Slow writes *
Zarr/N5 Next-generation 3D+ chunks NGFF spec Highly parallel **
HDFMetadata
MetadataZarr
Suite of open file formats
![Page 6: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/6.jpg)
Scientific providersCommercial providers
“Cloud” providers Remote
LocalO
bject“Cloud” à “Storage”
https://www.openio.io/blog/block-file-object-storage-evolution-computer-storage-systems
Filesystem Object storage
1 €/GB 0.01 €/GB
Gbps Tbps
10 µs 1 ms
I/O intensive Immutable
High frequency Versioned
… …
Software needsto be smarter.
Each object is a chunk.
FilesystemsNFS/SMBGPFSHDDSDD
![Page 7: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/7.jpg)
NGFF
Proprietary File Formats (PFF)
Conversion
Analysis clients & libraries
Upload
Remote
LocalO
bject
On-the-flytranslation
On-the-flytranslation
Upload
![Page 8: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/8.jpg)
Proprietary File Formats (PFF) Analysis clients & libraries
Upload
Remote
LocalO
bject
On-the-flytranslation
On-the-flytranslation
NGFF
Conversion
Upload
![Page 9: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/9.jpg)
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 9
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Conversion
bioformats2raw raw2ometiff
OME-XMLTIFF
Chris Allan & GlencoeConverting Whole Slide Images to OME-TIFF: A New Workflow
glencoesoftware/bioformats2rawglenocesoftware/raw2ometiff
MetadataZarr/N5
![Page 10: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/10.jpg)
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 10
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Specifications:
• Multiscales
• Labels
• HCS Plates
Process:
• Discussions: https://image.sc
• Publication: https://ngff.openmicroscopy.org
• Samples: https://s3.embassy.ebi.ac.uk/idr/zarr
{“Object-per-chunk”
MetadataZarr/N5
![Page 11: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/11.jpg)
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 11
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
PracticalDownload from S3Convert your dataShare your data on S3http://bit.ly/gbi-2021-ngff-1
![Page 12: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/12.jpg)
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 12
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
A few last words on S3Endpoint Bucket Key (Access)
https://s3.embassy.ebi.ac.uk idr zarr/v0.1/zarr/9822151.zarr download
https://s3.embl.de bioim idr0062-tiffs/… download, upload
https://s3.eu-south-1.amazonaws.com … … …
https://s3.embl.de/bioim/s3-browser.html
Unique identifier for an object
idr0062
Z-stacks of TIFFs
idr0023
![Page 13: Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing beyond current limits, e.g., downloading TBs from OMERO. 2. Remotely sharing without](https://reader035.fdocuments.net/reader035/viewer/2022071516/613940a0a4cdb41a985b9547/html5/thumbnails/13.jpg)
the authors,Thanks to the OME teams,
GehlenborgLab (HMS)
& our funders
Hériché et al. (idr0002, CC-BY 4.0)MBoC (2016)
Lamers et al. (idr0083, CC BY 4.0)Science (2020)
Blin et al. (idr0062, CC BY 4.0)PLOS Biology (2019)
Faas et al. (idr0053, CC BY-NC-SA 3.0)J Cell Biol (2012)
VivBDVTomancak Lab(MPI-CBG)
McDole et al. (idr0044, CC BY 4.0)Cell (2018)
N5Saalfeld Lab(Janelia)
https://www.openmicroscopy.org/teams
ZarrAlistair Miles(Oxford)
13
Acknowledgements
KLBKeller Lab(Janelia)