Scientific Computing and Visualization using HDF
-
Upload
the-hdf-eos-tools-and-information-center -
Category
Technology
-
view
245 -
download
0
Transcript of Scientific Computing and Visualization using HDF
![Page 1: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/1.jpg)
Aashish [email protected]
Technical Leaderwith
Patrick O’Leary, Petr Votava (NASA), Dr. Rama Nemani (NASA), Chris Harris,
Chris Kotfila, Doruk Aztek, Andrew Michaelis (NASA)
Scientific Computing and Visualization using HDF
July 19th 2016Durham, NC
![Page 2: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/2.jpg)
What We Do at Kitware?
0pen Source and Open Data is strongly encouraged and practisedat Kitware
![Page 3: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/3.jpg)
It started with VTK
![Page 4: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/4.jpg)
Parallel Processing and Rendering - Paraview
![Page 5: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/5.jpg)
Computer Vision
Object and Building Recognition by
Function (DARPA)
Images, Video, Point
Clouds
Recognition by Function
Content-based
Retrieval
Event & Activity
Recognition
Anomaly Detection
3D Extraction and
Compression
Detection & Tracking
![Page 6: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/6.jpg)
Medical Computing
Quantitative imaging Electronic health records
Vascular analysisSurgical guidance And simulation
Digital pathology Orthopedic analysis
Longitudinal and population shape
analysisInteractive medical applications
and visualizations
![Page 7: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/7.jpg)
Community Adaptation
![Page 8: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/8.jpg)
HDF at KitwareClimate Community High Performance Computing
Extensible Data Model and Format
- Developed to exchange scientific data between HPC codes and tools
- Heavy data is stored using HDF5
Network Common Data Form (NetCDF)
- Most projects use NetCDF4
Medical Community Vision CommunityLeading-edge algorithms for registering and segmenting multidimensional data
![Page 9: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/9.jpg)
ACMEThe Accelerated Climate Modeling for Energy (ACME) project is sponsored by the Earth System Modeling (ESM) program (Biological and Environmental Research) with eight national laboratories and six partner institutions to develop and apply the most complete, leading-edge climate and Earth system models to challenging and demanding climate-change research imperatives.
Most commonly used data format - NetCDF4
Data streaming using OpenDAP
Python Interface for most of the tools
![Page 10: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/10.jpg)
OpenNEXNEX is a platform for scientific collaboration, knowledge sharing and research for the Earth science com
Global Daily Downscaled Projections (NEX-GDDP, NetCDF4)
MODIS-Land and Atmosphere (HDF)
![Page 11: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/11.jpg)
Web VisualizationData processing
Gaia
Gaia
![Page 12: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/12.jpg)
Web VisualizationData processing
Pure JS?
![Page 13: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/13.jpg)
HDF5 File Organization
![Page 14: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/14.jpg)
Preprocessing Simulation Postprocessing
![Page 15: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/15.jpg)
![Page 16: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/16.jpg)
Possible ImprovementsStreaming and Big Data analytics
- Any useful ingestion of HDF data into cluster requires ETL pipeline
- For some tools, computation cannot move close to the data, streaming support is necessary in such cases
- Optimal read/write on cloud storage
Web-Support
- More tools and projects are moving to support web-enabled data analysis and visualization
- Pure JS implementation if possible
![Page 17: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/17.jpg)
Summary● HDF is widely data format for scientific computing,
climate/geospatial visualization, and in other domains at Kitware
● Recently we have started using HDF for information visualization
● We are looking forward to HDF usage on cloud and web-environment
● Kitware is always looking for strong open source collaborations and is committed to push open-source scientific computing to its next level
![Page 18: Scientific Computing and Visualization using HDF](https://reader036.fdocuments.net/reader036/viewer/2022062821/58a083e21a28ab19098b755f/html5/thumbnails/18.jpg)
InformationAashish Chaudhary: [email protected]
LinkedIn: www.linkedin.com/in/aachaudhary
Kitware: http://www.kitware.com
NASA-NEX: https://nex.nasa.gov/nex
Kitware-AIST: https://github.com/OpenGeoscience/nex
HPC Cloud : http://www.kitware.com/publications/item/view/1784
HPCloud Github: https://github.com/Kitware/HPCCloud