www.hdfgroup.org

The HDF Group

1

Introduction to netCDF-4

Elena PourmalThe HDF Group

10/17/2015

www.hdfgroup.org2

Overview

10/17/2015

• What is netCDF?• Data model and formats• Ecosystem

www.hdfgroup.org

WHAT IS NETCDF?

310/17/2015

www.hdfgroup.org4

Background

10/17/2015

• Developed and maintained by Unidata http://www.unidata.ucar.edu/

• Mission: provide data services, tools, and community leadership to advance Earth system science, enhance educational opportunities, and broaden participation

• Funded by NSF through UCAR• Open source software

www.hdfgroup.org5

What is netCDF?

10/17/2015

netCDF = network Common Data Form• Data model

• Scientific data (especially suitable for gridded data)• Widely used in ocean and atmospheric science, climate and

weather modeling• Other disciplines: molecular dynamics, fusion research,

medical imaging• File FORMAT

• Portable, self-describing, etc.• Not just one file format (netcdf, HDF4, HDF5, OPeNDAP, etc.)

• Application programming interfaces (APIs)• C, Java, C++, Fortran• Python, Ruby, Perl, MATLAB, IDL

www.hdfgroup.org6

History of netCDF

10/17/2015

• 1988 – first release of netCDF• 1991 – netCDF 2.0• 1994 – HDFv 3.3 reads netCDF files• 2004 – netCDF 3.6 supports 64-bit offsets• 2008 – netCDF-4 based on HDF5• 2010 and later – parallel, OPeNDAP, HDF4

www.hdfgroup.org

NETCDF DATA MODEL AND FORMATS

710/17/2015

www.hdfgroup.org8

netCDF classic data model

10/17/2015

• Variables • Name, shape, type• N-dim arrays

• Dimension• Name, length

• Attributes • Name, type, value

www.hdfgroup.org9

netCDF classic data model

Data model elements:• Variables • Name, shape, type• N-dim arrays

• Dimensions• Name, length• Can be shared

• Attributes • Name, type, value

Model limitations:• Atomic types only• One extendible

dimension• Flat structureModel advantages:• Widely adopted• Best practices and

conventions

10/17/2015

www.hdfgroup.org10

netCDF classic format

Pros:• Simple (header with

metadata, raw data)• Suitable for parallel

access (pnetcdf from Argonne)

• Aside: could easily support SWMR

Cons:• No support for

compression• No extensibility in

multiple dimension• XDR-based (big-endian)• Costly to add more

variables

10/17/2015

www.hdfgroup.org11

netCDF enhanced data model

Data model elements:• Variables• Dimensions• Attributes • Groups

Model limitations:• Complex• Harder to use and not

widely adopted• Emerging best practices

and conventionsModel advantages:• Rich collection of

datatypes• Hierarchical data

organization

10/17/2015

www.hdfgroup.org12

Example of ncdump output

netcdf OMI-Aura_L2-example {dimensions:

PRESSURE = 18 ;DATETIME = 2 ;independent_465 = 465 ;independent_22 = 22 ;independent_7 = 7 ;independent_2 = 2 ;

variables:float PRESSURE(PRESSURE) ;

PRESSURE:MissingValue = -999.f ;PRESSURE:standard_name = "air_pressure" ;PRESSURE:VAR_NAME = "PRESSURE" ;PRESSURE:VAR_DESCRIPTION = "Pressure at retrieval layer" ;PRESSURE:units = "hPa" ;PRESSURE:valid_range = -Infinityf, Infinityf ;PRESSURE:VAR_DEPEND = "PRESSURE" ;

short APrioriCovarianceMatrix(DATETIME, independent_465) ;APrioriCovarianceMatrix:MissingValue = -32767s ;APrioriCovarianceMatrix:UniqueFieldDefinition = "OMI-Specific" ;APrioriCovarianceMatrix:Offset = 0. ;

10/17/2015

www.hdfgroup.org13

Example of h5dump output

/PRESSURE Dataset {18/18} Attribute: CLASS scalar Type: 16-byte null-terminated ASCII string Data: "DIMENSION_SCALE" Attribute: MissingValue {1} Type: native float Data: -999 Attribute: NAME scalar Type: 9-byte null-terminated ASCII string Data: "PRESSURE" Attribute: REFERENCE_LIST {9} Type: struct { "dataset" +0 object reference "dimension" +8 native int } 16 bytes Data: (0) {DATASET-1:14192, 1}, {DATASET-1:14192, 2}, {DATASET-1:30064, 1}, {DATASET-1:32126, 1}, {DATASET-1:32472, 1}, {DATASET-1:34237, 1}, (6) {DATASET-1:49807, 1}, {DATASET-1:55542, 1}, {DATASET-1:62105, 1}

10/17/2015

www.hdfgroup.org14

netCDF-4 - netCDF extended format and library

netCDF-4 uses HDF5 as a storage layerPros:• Compression• Chunking storage• Parallel I/O• Extensibility (more

objects can be added easily)

• No limitation on number objects

Cons:• Complex as HDF5• Clunky support for

dimensions• Not easy to tune for

performance since some HDF5 parameters are hidden

10/17/2015

www.hdfgroup.org

netCDF-4 Architecture

10/17/2015 15

HDF5 Library

netCDF-4Library

netCDF-3Interface

netCDF-3applications

netCDF-3applications

netCDF-4applications

netCDF-4applications

HDF5applications

HDF5applications

netCDFfiles

netCDFfiles

netCDF-4HDF5 files

HDF5files

www.hdfgroup.org

NETCDF ECOSYSTEMReal Strength

1610/17/2015

www.hdfgroup.org17

Ecosystem

10/17/2015

Conventions

Standards: NASA,

OGC, FGDC

Tools and utilities

Servers, clients,

services: THREDDS, OPeNDAP

Analysis and

visualization: IDV, IDL,

MATLAB

Simple model

and diverse

APIs

www.hdfgroup.org18

Power of CF conventions

10/17/2015

NPP data visualized with IDV

Without CF metadata With CF metadata and dimensions

www.hdfgroup.org10/17/2015 19

www.hdfgroup.org20

Questions

10/17/2015

?

Thank you!