Www.hdfgroup.org The HDF Group 10/17/151 HDF5 Tools Tutorial ICALEPCS 2015.
HDF5 Tools Update
-
Upload
colton-wise -
Category
Documents
-
view
28 -
download
0
description
Transcript of HDF5 Tools Update
HDF5 Tools Update
Peter Cao - The HDF [email protected]
November 6, 2007
This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 2
Outline
Overview of current tools
New tools in HDF5
HDF-Java 2.4 release
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 3
Readers • h5dump, h5diff, h5ls• new tools: h5check, h5stat
HDF5 Command Line Tools
Writers•h5repack, h5repart, h5import, h5jam/h5unjam•new tool: h5copy, h5mkgrp
Converters•h4toh5, h5toh4, gif2h5, h52gif
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 4
Where to start
Tools located at bin/ directory
1. Use “-h” option for help
2. Online help http://www.hdfgroup.org/hdf5tools.html
3. Email to THG help desk [email protected]
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 55
h5dump
Dumps file content to
stdout in
• ASCII text• XML • Binary
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 6
h5dump -H SDS.h5
HDF5 "SDS.h5" { GROUP "/" { GROUP "Floats" { DATASET "FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } } } DATASET "IntArray" { DATATYPE H5T_STD_I32LE DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) } } }}
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 7
h5dump -d /Floats/FloatArray SDS.h5
HDF5 "SDS.h5" {DATASET "/Floats/FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 )
} DATA { (0,0): 0.01, 0.02, 0.03, (1,0): 0.1, 0.2, 0.3, (2,0): 1, 2, 3, (3,0): 10, 20, 30 }}}
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 8
h5dump -x SDS.h5
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 9
h5dump Binary Output
-b F, --binary=[MEMORY|FILE|LE|BE]
0000000 00000000 00000001 00000002 00000003
00000004 000000050000030 0000000a 0000000b 0000000c 0000000d
0000000e 0000000f0000060 00000014 00000015 00000016 00000017
00000018 000000190000110 0000001e 0000001f 00000020 00000021
00000022 000000230000140 00000028 00000029 0000002a 0000002b
0000002c 0000002d
h5dump -d /IntArray -o out_le.bin -b LE SDS.h5
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 10
h5diff
BetweenShow Differences
two objects or two files
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 11
h5diff SDS.h5 SDS2.h5
Dataset: </IntArray> and </IntArray>5 differences found
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 12
Report differences
Dataset: </IntArray> and </IntArray>position IntArray IntArray difference ------------------------------------------------------
[ 0 0 ] 0 10 10
[ 1 0 ] 10 100 90
[ 2 0 ] 20 200 180
[ 3 0 ] 30 300 270
[ 4 0 ] 40 400 360
5 differences found
h5diff SDS.h5 SDS2.h5 -r /IntArray
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 13
h5repack
Copies a file to a new file with • Compression filter• Data layout• Other user options
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 14
h5repack: new filters
-f {GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE}
For exampleh5repack -i SDS.h5 -o out.h5 -f GZIP=6
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 15
h5repack: data layout
-l {CHUNK| COMPA| CONTI}
For example
h5repack -i SDS.h5 -o out.h5 -l CHUNK=2x3
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 16
h5repack: latest version of file format
h5repack options File size in bytes
Original file 3,167,264
--latest 2,267,346
--latest –compact=9 2,162,254
--latest –compact=9 –ssize=50:dtype 299,882
The testing file was created with HDF5 1.6.5. The root group of the file contains 100 groups. Each group at the root contains 9 sub groups. Each sub group contains one compound dataset. The compound dataset has 40 members of 32 bit integer
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 17
h5repart
Repartitions a file into a family of files
For example
h5repart -m 200m int16kx16k.h5 part200m%d.h5
977 MB
200 MB part200m0.h5
200 MB part200m1.h5
200 MB part200m2.h5
200 MB part200m3.h5
177 MB part200m4.h5
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 18
h5import
PATH work/First-set INPUT-CLASS TEXTFP RANK 3 DIMENSION-SIZES 5 2 4 OUTPUT-CLASS FP OUTPUT-SIZE 64 OUTPUT-ARCHITECTURE IEEE OUTPUT-BYTE-ORDER LE CHUNKED-DIMENSION-SIZES 2 2 2 MAXIMUM-DIMENSIONS 8 8 -1
GROUP "/" { GROUP "work" { DATASET "First-set" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) } DATA { (0,0,0): 1.01, 1.02, 1.03, 1.04, (0,1,0): 1.11, 1.12, 1.13, 1.14, (1,0,0): 1.21, 1.22, 1.23, 1.24, (1,1,0): 1.31, 1.32, 1.33, 1.34, (2,0,0): 1.41, 1.42, 1.43, 1.44, … } } }}}
Imports binary/ASCII data into an HDF5 file• h5import infile -c config_file [infile -c
config_file2 ...] -outfile outfile
For exampleh5import float5x4x2.txt -c First_set.conf -o First_set.h5
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 19
h5jam/h5unjam
• h5jamh5jam -u test_ub.txt -i test_ub.h5
• h5unjamh5unjam -i test_ub.h5 -o out_ub.txt -o out_ub.h5
HDF5User Block
HDF5User Block
unjamJam
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 20
h5ls
/Floats Group/Floats/DoubleArray Dataset {10, 5}/Floats/FloatArray Dataset {4, 3}/Floats/subs Group/IntArray Dataset {5, 6}
• Lists selected information about file objects in the specified format
For Example, h5ls -r SDS2.h5
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 21
gif2h5 / h52gif
gif2h5 – converts a GIF file into HDF5gif2h5 apollo17_earth.gif apollo17_earth.h5
h52gif – converts an HDF5 file into GIFh52gif apollo17_earth.h5 apollo17_earth2.gif -i /apollo17_earth.gif/Image0 -p
"/apollo17_earth.gif/Global Palette"
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 22
h5toh4 / h4toh5
h5toh4 -- Converts an HDF5 file to an HDF4 file h4toh5 -- Converts an HDF4 file to an HDF5 file
http://www.hdfgroup.org/h4toh5/
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 23
New tools
h5stat
h5mkgrp
h5copy
h5check
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 24
h5copy
Copies an object within a file/cross files
/
FloatArray
FloatsIntArray
/
FloatArray
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 25
h5copy -f shallow
/
i1
floatsintegers
64-biti2 f32
f2f1
/
floats
64-bitf32
f2f1
/
floats
64-bitf32-f s
hallow
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 26
h5copy -f soft
/
-f soft
dset_SL
/f1/f1
f1
/
dset_SL
/f1/f1
f1
/
dset_SL
/f1/f1
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 27
h5copy -f ref
/
-f ref
d1
dset_ref
d2
1895
763
/
d1
dset_ref
d2
679
1287
/
dset_ref
0
0
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 28
h5mkgrp
Creates groups
/
info data
test1 test2
h5mkgrp test.h5 /info /data /data/test1 /data/test2
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 29
h5stat
Prints statistics about HDF5 file• Header size• Raw data size• Attributes, datatypes, and etc.
Helps • To troubleshoot overhead in HDF5 files
• To choose specific object’s properties and storage strategies
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 30
h5check
A validation tool that verifies if an HDF5 file is encoded according to the HDF5 File Format Specification
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 31
Why h5check?
• Verify if the file is compliant with the File Format
• As a verification tool required by the application of HDF5 File Format to be an ANSI standard
• Serves as a watch dog that the HDF5 library implementation is compliant with the File Format
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 32
Example: a compliant file
• % h5check example1.h5• VALIDATING example1.h5• FOUND super block signature• VALIDATING the super block at 0...• VALIDATING the object header at 928...• VALIDATING the btree at 384...• FOUND btree signature.• VALIDATING the local heap at 96...• FOUND local heap signature.• …• Result: File is in compliance.
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 33
Example: a non-compliant file
h5check invalid2.h5
• FOUND super block signature• VALIDATING the super block at 0...• VALIDATING the object header at 928...• VALIDATING the btree at 384...• FOUND btree signature.• VALIDATING the SNOD at 1248...• FOUND SNOD signature.• VALIDATING the object header at 976...• check_sym(at 1248): Errors from check_obj_header()• decode_validate_messages(): Failure in type->decode().• H5O_sdspace_decode(): Bad version number in simple dataspace
message.• VALIDATING the local heap at 96...• FOUND local heap signature.• Main(): Errors from check_obj_header().• decode_validate_messages(): Failure in type->decode().• H5O_attr_decode(): Can't decode attribute dataspace.• H5O_sdspace_decode(): Bad version number in simple dataspace
message.• …• Result: File is not in compliance.
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 34
Implementation Status
Basic file format components
1.6 - DONE1.8 - IN PROGRESS
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 35
HDF-Java 2.4
http://www.hdfgroup.org/hdf-java-html
ftp://ftp.hdfgroup.org/HDF5/hdf-java
HDF-Java 2.4 Released !!
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 36
HDF-Java 2.4 Release
• Build with HDF4 2r2 and HDF5 1.6.6• New platforms
Mac intelLinux 64-bit AMD Solaris 64-bit
• 36 new enhancements and 44 bugs fixed• Fixed memory leak• Test suite• Enhanced documentation
11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 37
Memory leak in old version
while (true) { H5File f = new H5File
(fname, H5File.READ);
f.open(); f.close();}
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.