Status of WDC Climate and long-term archiving at DKRZ
description
Transcript of Status of WDC Climate and long-term archiving at DKRZ
Michael Lautenschlager, Hannes Thiemann, Frank Toussaint
WDC Climate / Max-Planck-Institute for Meteorology, Hamburg
Joachim Biercamp, Ulf Garternicht, Stephan Kindermann, Wolfgang Stahl
German Climate Computing Centre (DKRZ) Hamburg
CAS2K9September 13th – 16th, 2009 in Annecy, France
2
blizzard:blizzard:/work
/pf/scratch
/work/pf
/scratch
tape:/hpss/arch /hpss/doku /dxul/ut /dxul/utf /dxul/utd
tape:/hpss/arch /hpss/doku /dxul/ut /dxul/utf /dxul/utd
xtape:xtape:
ssh blizzard
(sftp xtape.dkrz.de)„get /hpss/arch/<prjid>/<myfile>“
pftp
HPSS(10 Pbyte /a )
HPSS(10 Pbyte /a )
GPFS(3 Pbyte)
GPFS(3 Pbyte)
IBM Power62 x Login250 x Compute150 TFlops peak
IBM Power62 x Login250 x Compute150 TFlops peak
StorageTek SilosTotal Capacity: 60000 Tapes Approx. 60 PB
(LTO and Titan)
Data production on IBM-P6: 50-60 PB/year
Limit for long-term archiving: 10 PB/year
◦ Required is a complete data catalogue entry in WDCC
(metadata) but decision procedure for long-term archive
transition is not finally decided (data storage policy).
Limit for field-based data access: 1 PB/year
◦ Oracle BLOB-tables are replaced by CERA container file
infrastructure which is developed by DKRZ/M&D
3
4
Oct. 2008
Mid of 2009:10 PB
5
Oct. 2008
Mid of 2009:400 TB
6
Data system (HPSS) (Information on DKRZ Webserver)
The DXUL/UniTree will be replaced by HPSS (High Performance
Storage System). The existing DXUL-administered data -
about 9 PetaByte – will be transferred.
6 robot-operated silos with 60.000 slots for T10000 A/B, LTO4,
9940B and 9840C magnetic cartridges provide a primary
capacity of 60 PetaByte with 75 tape drives.
The average bandwidth of the data server is at least 3
GigaByte/s while simultaneously reading and writing with
peak flow rate up to 5 GigaByte/s.
390 TB Oracle BLOB data transferred into CERA container files
7
9 PB DXUL/UniTree data have to be transfered to HPSS
without copying the data
◦ 9 PB DXUL data are stored in 25,000 cartridges and 25 * 10**6 files
◦ It was not feasible to run two systems in parallel for 3 -5 years which is
the estimated time for copying from DXUL/UniTree to HPSS at DKRZ
Challenges of physical movement from Powderhorn (Unitree)
into SL8500 (HPSS):
◦ Technical aspects
◦ Legal aspects
◦ Quality assurance
8
Challenges of physical movement from Powderhorn
(Unitree) into SL8500 (HPSS):
◦ Technical aspects: In principal it is possible to read UniTree cartridges
with HPSS but it has been tested with old systems and with less
complexity of name spaces (17 name spaces on 3 servers have to
consolidated into 1 HPSS name space)
◦ Legal aspects: An unexpected license problem appeared with the
proprietary UniTree library data format. Solution was to write data library
information after consilidation into one large text file (10 GB).
◦ Quality assurance: complete comparison of metadata and checksum
comparison of a subset of 1% of the data files
Transfer to HPSS has been successfully completed, the new
system is up and running with the old data.
9
10
3 of 6 StorageTek SL8500 silos under construction
Room for 10 000 magnetic cartridges in each silo
CERA-2 data model left unchanged
◦ Metadata model modifications are planned in
relation to the outcome of the EU-project
METAFOR and CIM (Common Information Model)
WDCC metadata are still residing in Oracle
database tables which build the searchable
data catalogue
11
12
Entry
Reference
Status
Distribution
Contact Coverage
Parameter
SpatialReferenceLocal Adm.
Data Access
Data Org
METAFOR / CIM:• Data provenance information• Searchable Earth system model description
Unchangedsince 1999
Field-based data access is changing from
Oracle BLOB data tables into CERA
container files for two reasons:
◦ Financial aspect: Oracle license costs for an
Internet accessible database system of the size of
PB are out of DKRZ‘s scope.
13
◦ Technical aspect: The BLOB data concept in the
range of TB and PB requires seamless data
transition between disk and tape in order to keep
the RDBMS restartable. This worked for Oracle
and UniTree but it could not be guaranteed for the
future neither by Oracle nor by HPSS.
◦ Requirement for BLOB data replacement: Transfer
to CERA container files has to be transparent for
CERA-2 and user data access.
14
15
Model variables
Mod
el R
un T
ime
2 D: small LOBs (180 KB)
3 D: large LOBs (3 MB)
Each columm is one data table in CERA-2
T2M Precip SLP2D variables . . Temp
Water vapour
3D variables . .
T1T2T3.......Tn...................Tend
CERA Container Files• are LOBs plus index for random data access
• are transparent for field-based data access in WDCC
• include basic security mechanisms of Oracle BLOBs
Motivated by long-term archive strategy and scientific
applications like CMIP5/AR5 the WDCC data access is
extended:
◦ CERA Container Files: transparent field-based data access from
tapes and disks (substitution of Oracle BLOB data tables)
◦ Oracle B-Files: transparent file-based data access from disk and
tapes
◦ Thredds Data Server: field-based data access from files on disks
(CMIP5/AR5)
◦ Intransparent data access: URLs provide links to data which are
not directly/transparently accessible by WDCC/CERA (e.g.
remote data archives)16
17
Appl. Server
Storage@DKRZTDS
(or the like)
LobServer
HPSS
CERA
DB Layer• What• Where• Who
• When• How
Midtier
Archive: files
Container: Lobs
Three major decisions are made in connetion with long-term
archiving in transistion to HLRE2 and HPSS:
Limitation of annual growth rates
◦ File archive: 10 PB/year
◦ CERA Container Files: 1 PB/year
Development of CERA Container File infrastructure with
emphasis on field-based data access from tapes
Integration of transparent file-based data access into
WDCC/CERA in addition to traditional field-based data access
18