Sponsored by the National Science Foundation GENI I&M Workshop NetCDF and Local Data Manager (LDM)...
-
Upload
zayne-seavey -
Category
Documents
-
view
214 -
download
0
Transcript of Sponsored by the National Science Foundation GENI I&M Workshop NetCDF and Local Data Manager (LDM)...
Sponsored by the National Science Foundation
GENI I&M Workshop
NetCDF and Local Data Manager (LDM)
Mike ZinkNovember 4, 2010
http://geni.cs.umass.edu/visehttp://geni.cs.umass.edu/dicloud
http://www.geni.net
Sponsored by the National Science Foundation 2November 3, 2010
Overview
Sponsored by the National Science Foundation 3November 3, 2010
What is NetCDF
NetCDF is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.
Uses HDF5 as data storage layer.
• Also provides read-only access to some HDF4, HDF5 archives.
• Parallel I/O for high performance computing.• Developed and maintained by Unidata at UCAR
Sponsored by the National Science Foundation 4November 3, 2010
Who uses NetCDF?
• NetCDF is widely used in University Earth Science community.
• Used for Intergovernmental Panel on Climate Change (IPCC) data sets.
• Used by NASA and other large data producers.• Used for NEXRAD and CASA radar data
(moments)
Sponsored by the National Science Foundation 5November 3, 2010
NetCDF Example
5
netcdf chickasha.ok-20090527-103631 { dimensions: Radial = UNLIMITED ; / / (1289 currently) Gate = 426 ; variables: double Azimuth(Radial) ; Azimuth:Units = "Degrees" ; double Elevation(Radial) ; Elevation:Units = "Degrees" ; double GateWidth(Radial) ; GateWidth:Units = "Millimeters" ; double StartRange(Radial) ; StartRange:Units = "Millimeters" ; int StartGate(Radial) ; StartGate:Units = "Unitless" ; int Time(Radial) ; Time:Units = "Seconds" ; double TxFrequency(Radial) ; TxFrequency: Units = "Hertz" ; double TxLength(Radial) ; TxLength:Units = "Seconds" ; double TxPower(Radial) ; TxPower:Units = "dBm" ; int AfcSet(Radial) ; AfcSet:Units = "Unitless" ; int GcfState(Radial) ; GcfState:Units = "Unitless" ; float Reflectivity(Radial, Gate) ; Reflectivity:Units = "dBz" ; float Velocity(Radial, Gate) ; Velocity:Units = "MetersPerSecond" ; float SpectralWidth(Radial, Gate) ; SpectralWidth:Units = "MetersPerSecond" ; float DifferentialReflectivity(Radial, Gate) ; DifferentialReflectivity: Units = "dB" ; float DifferentialPhase(Radial, Gate) ; DifferentialPhase:Units = "Degrees" ; float CrossPolCorrelation(Radial, Gate) ; CrossPolCorrelation:Units = "Unitless" ; float NormalizedCoherentPower(Radial, Gate) ; NormalizedCoherentPower:Units = "Unitless" ; float SpecificPhase(Radial, Gate) ; SpecificPhase:Units = "DegreePerKm" ; float HPropagationPhase(Radial, Gate) ; HPropagationPhase:Units = "Radians" ; float VPropagationPhase(Radial, Gate) ; VPropagationPhase:Units = "Radians" ; int GateFlags(Radial, Gate) ; GateFlags:Units = "BitField" ; float CorrectedReflectivity(Radial, Gate) ; CorrectedReflectivity: Units = "dBZ" ; float CorrectedDifferentialReflectivity(Radial, Gate) ; CorrectedDifferentialReflectivity:Units = "dB" ; // g lobal attributes: :NetCDFRevision = "$Id: mdmd.c,v 1.33 2008-02-28 00:11:57 junyent Exp $" ; :RadarName = "chickasha.ok" ; : Latitude = 35.031226436707 ; :Longitude = -97.9566907121578 ; :Height = 353.986269181594 ; :MccId = 124335663 ; :NumGates = 426 ; :S canFlag = 1 ; :S canType = 2 ; :S canId = 29137 ; : PosAccel = 60. ; : PosVelocity = 12. ; :AntennaGain = 37. ; :AntennaBeamwidth = 1.8 ; }
float Reflectivity(Radial, Gate) ; Reflectivity:Units = "dBz" ; float Velocity(Radial, Gate) ; Velocity:Units = "MetersPerSecond" ; float SpectralWidth(Radial, Gate) ; SpectralWidth:Units = "MetersPerSecond" ; float DifferentialReflectivity(Radial, Gate) ; DifferentialReflectivity:Units = "dB" ; float DifferentialPhase(Radial, Gate) ; DifferentialPhase:Units = "Degrees" ; float CrossPolCorrelation(Radial, Gate) ; CrossPolCorrelation:Units = "Unitless" ; float NormalizedCoherentPower(Radial, Gate) ; NormalizedCoherentPower:Units = "Unitless" ; float SpecificPhase(Radial, Gate) ; SpecificPhase:Units = "DegreePerKm" ; float HPropagationPhase(Radial, Gate) ; HPropagationPhase:Units = "Radians" ; float VPropagationPhase(Radial, Gate) ; VPropagationPhase:Units = "Radians" ; int GateFlags(Radial, Gate) ; GateFlags:Units = "BitField" ; float CorrectedReflectivity(Radial, Gate) ; CorrectedReflectivity:Units = "dBZ" ; float CorrectedDifferentialReflectivity(Radial, Gate) ; CorrectedDifferentialReflectivity:Units = "dB" ;
Sponsored by the National Science Foundation 6November 3, 2010
OpeNDAP Client for NetCDF
OPenDAP (http://www.opendap.org/) is a widely supported protocol for access to remote data
Defined and maintained by the OPenDAP organization
Designed to serve as intermediate format for accessing a wide variety of data sources.
Client is now built into netCDF C library.
Sponsored by the National Science Foundation 7November 3, 2010
Local Data Manager (LDM)
• LDM software acquires data and shares them with other nodes
• Data product is treated as a opaque unit, thus nearly any data can be relayed
• LDM can handle data from:– National Weather Service "NOAAport channel 3" data
streams– NEXRAD radar data– lightning data from the National Lightning Detection
Network– GOES satellite imagery
Sponsored by the National Science Foundation 8November 3, 2010
LDM Features
• User configurable– execute an arbitrary program with the data product as
input
• Supports distributed processing– Data ingest functions can be separated from storage
and use functions– LDM servers may be configured to relay subsets of
data among themselves, distributing the storage and processing around the network.
• Extensible– New decoders can be added easily
• Event-driven– Handles data as it arrives
Sponsored by the National Science Foundation 9November 3, 2010
Demo Data Flow• Dynamic end-to-end Nowcasting
– Mapping Nowcast Workflows onto GENI
Archival Storage
Radar Nodes
“raw” live data
Upstream LDM feed
archived netcdf data
Nowcast Processing
aggregatedmulti-radar data
Post to Web
Nowcast images for display
NetCDF over LDM
LDM server
Storage
NetCDF over LDM
LDM server
Mer
ge
NetCDF over LDM
LDM server
Now
casting
Sponsored by the National Science Foundation 10November 3, 2010
Multi-radar NetCDF Data
Nowcast Processing
1. DiCloud Archival Service (S3)2. LDM Data Feed (EC2)
“raw” live data
Generate “raw” live dataViSE/CASA radar nodes
http://stb.ece.uprm.edu/current.jsp
Use proxy to track usage-based spending on Amazon and enforce quotas and limits
http://geni.cs.umass.edu/vise/dicloud.php
1. Ingest mulit-radar data feeds2. Merge and grid multi-radar data2. Generate 1min, 5min, and 10min Nowcasts3. Send results over NLR to Umass4. Repeat
ViSE views steerable radars as shared, virtualized resources
http://geni.cs.umass.edu/vise
Nowcast images for display