How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and...
Transcript of How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and...
![Page 1: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/1.jpg)
B2STAGE
How to shift large amounts of data
Version 3
June 2014
1
www.eudat.eu | http://www.eudat.eu/b2stage B2STAGE Training
![Page 2: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/2.jpg)
B2STAGE is part of EUDAT...
a pan-European initiative building a sustainable
cross-disciplinary and cross-national data
infrastructure providing a set of shared services for
accessing and preserving research data
supporting multiple research
communities by working closely
with them to deliver these
technical services as part of the
EUDAT Collaborative Data
Infrastructure (CDI) www.eudat.eu | http://www.eudat.eu/b2stage B2STAGE Training
![Page 3: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/3.jpg)
A truly pan-European Infrastructure
general data centres
community centres
representing all the associated
community data centres
Research Communities
National Data centres
Technology providers
Offering permanence,
persistence, reliability
and long term
solutions
www.eudat.eu | http://www.eudat.eu/b2stage B2STAGE Training
![Page 4: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/4.jpg)
Where is B2SHARE in the EUDAT suite?
B2STAGE represents an extension of the B2SHARE and offers communities a light approach to ingest and replicate data. Data ingested through B2STAGE is registered with a Persistent Identifier (PID) using the same mechanism adopted by B2SAFE www.eudat.eu | http://www.eudat.eu/b2stage B2STAGE Training
![Page 5: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/5.jpg)
A reliable, efficient, lightweight and easy-to-use service to ship large amounts of research data between EUDAT storage resources and workspace areas of high-performance computing systems.
5
B2STAGE is... B2STAGE does...
B2Stage can be used to simply ingest community data onto EUDAT resources using a high performance protocol, like GridFTP.
www.eudat.eu www.eudat.eu/b2stage
![Page 6: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/6.jpg)
Why use B2STAGE?
Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs in the human body, seismic analyses of earthquakes at continental scale and
Researchers’ data and compute demands are rising fast
Efficient shipping of data to high performance computing (HPC) workspaces is essential especially in distributed computing, where resources are geographically dispersed
6 www.eudat.eu www.eudat.eu/b2stage
![Page 7: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/7.jpg)
Why use B2STAGE?
Facilitate transfer of large data collections from EUDAT storage resources to external HPC facilities.
Offers reliable, efficient, easy-to-use tools to manage data transfers.
Provides the means to re-ingest computational results back into the EUDAT infrastructure.
Ingests data sets onto EUDAT resources for long-term preservation.
7 www.eudat.eu www.eudat.eu/b2stage
![Page 8: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/8.jpg)
Who can use B2STAGE?
Researchers can transfer large data collections from EUDAT storage resources to HPC facilities for processing.
Community Managers can replicate community data through a lightweight service and ingest data sets to EUDAT storage resources for long term preservation.
8 www.eudat.eu www.eudat.eu/b2stage
![Page 9: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/9.jpg)
Why is B2STAGE unique?
The DSS is the only tool handling data transfer using PIDs.
Easy, reliable and fast solution for data ingestion and transfer onto and from EUDAT resources.
9 www.eudat.eu www.eudat.eu/b2stage
![Page 10: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/10.jpg)
How can you use B2STAGE?
10
For more information please email: [email protected]
EUDAT offers B2STAGE to all registered researchers and interested communities enabling them to make use of the service to stage data out of EUDAT, and ingest computational results back.
Access to remote HPC facilities should be negotiated and
arranged by individual users in parallel.
To help researchers to use the B2STAGE service, EUDAT offers documentation, educational material and a service helpdesk.
www.eudat.eu www.eudat.eu/b2stage
![Page 11: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/11.jpg)
B2STAGE User communities
VPH Community to ingest data onto EUDAT resources
Approximately 12TB will be ingested thought this service
NeuGRID and INCF are considering its adoption to replicate data
Collaboration with other e-infrastructures
VPH to transfer data across EUDAT, PRACE, EGI
11 www.eudat.eu www.eudat.eu/b2stage
![Page 12: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/12.jpg)
B2STAGE currently...
The current version of B2STAGE offers:
data staging functionalities to easily and efficiently ship data across EUDAT storage resources and HPC facilities;
a powerful mechanism to ingest data onto EUDAT resources;
a script to facilitate the staging, the ingestion and the retrieving of PID information of transferred data.
12 www.eudat.eu www.eudat.eu/b2stage
![Page 13: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/13.jpg)
Where does B2STAGE fit within EUDAT?
13
B2STAGE represents an extension of the B2SHARE and offers communities a light approach to ingest and replicate data. Data ingested through B2STAGE is registered with a Persistent Identifier (PID) using the same mechanism adopted by B2SAFE
www.eudat.eu www.eudat.eu/b2stage
![Page 14: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/14.jpg)
Future features...
Optimization of transfers on the basis of data location within the EUDAT infrastructure (under evaluation).
Improvement of user experience with the Data Staging Script (i.e. data path autocompletion, multi-pid parallel handling, etc.).
Foster the collaboration with EGI and PRACE to develop cross-infrastructure usage: the B2STAGE will be the main service to enable the
interoperability of these infrastructures.
14 www.eudat.eu www.eudat.eu/b2stage
![Page 15: How to shift large amounts of data · Why use B2STAGE? Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs](https://reader034.fdocuments.net/reader034/viewer/2022050219/5f6511e9ab5de768a63f9e72/html5/thumbnails/15.jpg)
Thanks
For more info: www.eudat.eu/b2stage www.eudat.eu | http://www.eudat.eu/b2stage B2STAGE Training