Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt...
-
Upload
daniella-francis -
Category
Documents
-
view
218 -
download
0
Transcript of Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt...
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate Data Formats
Deniz Bozkurt
web.itu.edu.tr/bozkurtdITU Eurasia Institute of Earth Sciences
The International Training Course on“Climate Analysis and Applications”
Alanya10-19 October 2011
The International Training Course on“Climate Analysis and Applications”
Alanya10-19 October 2011
Eurasia Institute of Earth SciencesIstanbul Technical University
Datasets
Climate Datasets
Eurasia Institute of Earth SciencesIstanbul Technical University
Raw data
“abcd12345”
Input system
Digital representation “11010001”
Climate Datasets
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate Datasets
• Modeled data: gridded multi-dimendisonal data– Consisting of variables, dimensions and attributes
• Observations (station, reanalysis)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate Datasets
• Climate data can be stored as different formats in different platforms:
• Character format
• Packed binary format
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate Data Formats - Character Format
• ASCII (American Standard Code for Information Interchange)
• It refers to a “text” file that is readable by the naked eye (it only contains the letters a-z, numbers, carriage returns, and punctuation marks.
• Each character is stored as digital
============
Binary100100011001011101100110110011011110101100010000011101111100111111001011011001100100
Hello, world
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ASCII
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ASCII
• Factors restrict the use of ASCII:
• Weak data storage and huge data size:
“9” 1 byte, “679.43” 6 byte, -0.123456E+05 13 byte
• No or not practical metadata
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – Packed Binary
• A common data format for saving and storing of large size of the data
• “3.1” 24 bits (ASCII) can be packed as 5 bit
• Can be processed very quickly and effectively in data analysis
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – Packed Binary
Eurasia Institute of Earth SciencesIstanbul Technical University
Some standard data formats
• Self-describing data format– All the information about data is stored inside of the data
– No need for an additional information for the identification of the contents of the file
– Machine independent
• Can be readable by various applications– NCL, IDL, GrADS, Matlab, Ferret, C, C++, Java, Fortran
– Komut satırı operatörler: NCO, CDO
• Most commonly used format in the oceanographic and atmospheric science for observational data and numerical modeling (IPCC datasets, NASA, NOAA, NCAR, UCAR…)
NetCDF (network Common Data Form) Format
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
metadata
dimensions
variables
Variableproperties
attributes
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
Data storage in NetCDF format
• 3D (x,y,t) or 4D (x,y,z,t)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
Data storage in NetCDF format
141 241 341
131 231 331
121 221 321
111 211 311
441
431
421
411
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
netcdf mynetcdf {
dimesions:
x=4;
y=4;
time=UNLIMITED;
variables:
float x(x);
float y(y);
int time(time);
float temperature(time,x,y);
data:
x = 10, 20, 30, 40;
y = 110, 120, 130, 140;
time = 31, 59, 90;
}
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
Data storage in NetCDF formatnetcdf mynetcdf {
dimesions:
x=4;
y=4;
time=UNLIMITED;
variables:
float x(x);
float y(y);
int time(time);
float temperature(time,x,y);
data:
x = 10, 20, 30, 40;
y = 110, 120, 130, 140;
time = 31, 59, 90;
Temperature = 111, 211, 311, 411;
}
141 241 341
131 231 331
121 221 321
441
431
421
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
time = 1
x = 1 to 4
y = 1
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
Data storage in NetCDF formatnetcdf mynetcdf {
dimesions:
x=4;
y=4;
time=UNLIMITED;
variables:
float x(x);
float y(y);
int time(time);
float temperature(time,x,y);
data:
x = 10, 20, 30, 40;
y = 110, 120, 130, 140;
time = 31, 59, 90;
Temperature = 111, 211, 311, 411, 121, 221, 321, 421, 131, 231, 331, 431, 141, 241, 341, 441;
}
time = 1
x = 1 to 4
y = 1 to 4
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
Data storage in NetCDF formatnetcdf mynetcdf {
dimesions:
x=4;
y=4;
time=UNLIMITED;
variables:
float x(x);
float y(y);
int time(time);
float temperature(time,x,y);
data:
x = 10, 20, 30, 40;
y = 110, 120, 130, 140;
time = 31, 59, 90;
Temperature = 111, 211, 311, 411, 121, 221, 321, 421, 131, 231, 331, 431, 141, 241, 341, 441, 112, 212, 312, 412, 122, 222, 322, 422, 132, 232, 332, 432, 142, 242, 342, 442, 113, 213, 313, 413, 123, 223, 323, 423, 133, 233, 433, 143, 243, 343, 443;
}
time = 1 to 3
x = 1 to 4
y = 1 to 4
Y
X
Time
Eurasia Institute of Earth SciencesIstanbul Technical University
4D (x,y,z,t)
141 241 341
131 231 331
121 221 321
111 211 311
441
431
421
411
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
Altitude
Climate data formats - NetCDF
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
some NetCDF commands
– ncdump file_name | less, to see the entire contents of the file
– ncdump –h file_name | less, to see the front portion of the data
– ncdump –v var_name file_name | less, to see the specific data within the file
– ncl_filedump to see the contents of different data formats
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - GRIB
GRIB (GRIdded Binary) Format
• World Meteorological (WMO) standard, commonly used in meteorology
• Bit-oriented data format: very efficient for transmission/archival
- netCDF (float) 2-4 times larger than corresponding GRIB
• two dimensional horizontal grids only
- each grid has scale/offset for optimal precision
• requires external parameter table (s)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - GRIB
• Basically consists of:– Product definition section(PDS)
• Creater of the data, unit of the data, time stamp, vertical system of the data
– Grid description section (GDS), optional• Grid description, map projections
– Bit map section (BMS), optional• Defines which grid points will be presented
– Binary data section (BDS)• Contains the packed binary data
– End section
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - GRIB
• Files are typically 1/2 to 1/3 of the size of normal binary files:
• Y: gridded data
• D: scale factor
• R: reference value
• E: binary scale factor
€
Y ∗10D = R + X ∗2E( )
Eurasia Institute of Earth SciencesIstanbul Technical University
• NetCDF data:• Raster
• Table
• Feature
• Directly readable
Climate data formats – ArcGIS Shapefile
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ArcGIS Shapefile
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ArcGIS Shapefile
Raster
Feature
Table
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ArcGIS Shapefile
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ArcGIS Shapefile
http://www.gisclimatechange.org/