Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and...
Transcript of Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and...
![Page 1: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/1.jpg)
Cross Site Data Movement:The UK as a “Data Grid”
Jens Jensen, Mad Scientist
Scientific Computing Dept
UKRI-STFC
March 2019
![Page 2: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/2.jpg)
Context
• GridPP – UK grid for particle physics– STFC-funded; infrastructure spans ~20 sites in UK
– Connected into WLCG
• IRIS – STFC funded research
• This talk focuses on data and “plumbing”– Not compute
– Nor metadata, nor information systems
– Nor storage accounting
Note these slides will work best as powerpoint because they have some animations
![Page 3: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/3.jpg)
Context
Google map of WLCG sites (source: WLCG, resp. Google…)WLCG == Worldwide LHC Computing GridLHC == Large Hadron Collider Clearly weighted towards northern hemisphere but a global endeavour
![Page 4: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/4.jpg)
Context
Similar map of UK: GridPP(Source: WLCG, and Google for the map…)
But it’s not just the LHCLIGO, LSST, SKA, DUNE, T2K, …
WLCG is mostly HTC but other users need more HPC or big memory machines
![Page 5: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/5.jpg)
File Structure
Logical
Site1
Physical1 Physical2
Site2
Physical
![Page 6: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/6.jpg)
Dataset Structure (Rucio)
FileDataset“Container”0..* 0..*
DID(data
identifier)
scope
name
1
1
0..*
![Page 7: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/7.jpg)
Storage Architecture
“Grid”
Site1
Obj store
Tape
Site2
StoRM
Lustre
Site3
StoRM
HDFS
Site 4
DPM
RAID
Site5
dCache
RAID
Catalogue
![Page 8: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/8.jpg)
Storage Architecture
Grid
DataLake
Site1 Site2
DataLake
Site3
Site 4
Site5
Network proximity Geographic
proximity
![Page 9: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/9.jpg)
Copying Data
GOOD GOOD BAD
WORSE Less worse
![Page 10: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/10.jpg)
Transfer Protocols
• Source and destination need to share transfor protocol
• Parallel streams (e.g. GridFTP)
• Standards-based:– GridFTP (GFD.47)
– HTTP/WebDAV
– SRM (GFD.154)
• xroot
![Page 11: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/11.jpg)
Networks
perfSONAR monitors latency and bandwidth
Plot credits: Sam Skipsey, GridPP, U Glasgow
![Page 12: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/12.jpg)
Authentication, Authorisation, Delegation
• X.509 certificate authentication– In WLCG most users have individual certificates
(IGTF)
– Some GridPP communities generate on-the-fly (e.g. RCauth, Pathfinder)
• RBAC– Simple VO-defined roles through VOMS
• Token-based authorisation– JWT (RFC7519)
![Page 13: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/13.jpg)
gLite
CDMI
FTS
DPM
tools
performance
VO relevantmonitoring
DPMrequestsmonitoring
sonarrelevance
Small VOs
T2K
storagemeeting
tuning
singlemetadata filesystems
small filehandling
pNFS
HDFS
CEPH
MapReduce
EOS
T3?
T1?
hardware
procurement
vendor
GridPPT2s
CERN
CASTOR
collaboration
fsprobe
channelbonding
![Page 14: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/14.jpg)
Data Transfer Zone(ESNET’s “Science DMZ”)
SiteCSiteA
SiteB
DTN
DTN
DTN
network
Sites run Data Transfer Nodes connecting over JANETNodes are secured through the IGTF PKI (incl client authentication)
OPN
DTN
SiteD
![Page 15: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/15.jpg)
Firewalling DTNs
DTNstorage
1. Firewalled
DTNstorage
3. Firewall in parallel,inspecting packets
DTNstorage
4. DTN outside firewall
DTNstorage
2. Pass through
![Page 16: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/16.jpg)
Transfer Tools
• Globus Connect
• FTS
• Long history of low-level data toolsets and APIs– edg-*, lcg-*, gfal-*, globus-*
– davix-*
• davix is CERN’s WebDAV implementation
– Supports AWS S3, Azure
![Page 17: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/17.jpg)
File Transfer Service - FTS
• In numbers– 17 instances (across WLCG) support 20 VOs
– Move 20PB data in 26M transfers per week (~1EB/yr)
• Scheduler– Prioritisation
• Optimiser– Reorder based on throughput, success rate
– Also optimises #parallel streams
• Automatic retries
• Small file optimisation (= conn. reuse)
![Page 18: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/18.jpg)
• FTS distributes the majority of Large Hadron Collider data across the World LHC Computing Grid (WLCG) infrastructure.
• Developed at CERN.MySQL
Server
Bring online
REST
Server
Bring online
REST
Optimizer
Scheduler
Fts-Url-Copy
Gfal2
…
Multi-protocol support
Client Access
Staging from tapes
Optimizes parallel transfers
(HA Proxy)
MSG MSG
This slide from Andrea Manzi (CERN)
![Page 19: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/19.jpg)
Multiprotocol support: gfal2• FTP/GSIFTP, HTTP, XROOTD, SRM, S3, GCLOUD, ..
• TPC (3rdParty copy) or protocol translation (streaming)
28/02/19 FTS news and plans19This slide from Andrea Manzi (CERN)
![Page 20: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/20.jpg)
Turtles
• High Level data management (e.g. Rucio)– Replication policy
– Deletion policy
– User-facing APIs
• File transfer service
• Storage Elements
• Distributed File System, Object Stores, etc.
• Storage Fabric
![Page 21: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/21.jpg)
Scaling to Exabyte
0
1
2
3
Tier0: instrument,preprocess
Tier1: preprocess,global replication
Tier2: user analysis,regional replication,local cache
Tier3: end useranalysis
Not to be confused with data centre tiers!
![Page 22: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/22.jpg)
![Page 23: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/23.jpg)
![Page 24: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/24.jpg)
Optimising data/compute
• Generally sending compute to where data is
• “Federated storage” – cross site access– E.g. if a replica is missing
• Pilot jobs provide late binding of workload to job slot– Job slot is allocated to experiment but once the
job starts, it figures out what to do…
![Page 25: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/25.jpg)
(Other) Future Directions
• Better support for non-wizard users– (not CLI, federated id)
• More inter-turtle communication– Cache-aware data layers
– Make use of Redfish (DMTF)/Swordfish (SNIA)
• Accommodating WLCG evolutions– Cache only sites
– Further increasing “federated” storage (cross site access)
• More interfacing to other infrastructures
• Supporting IRIS (STFC funded researchers)– Then UKRI research communities?
![Page 26: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/26.jpg)
Conclusions
• Exascale: regimented data model
• Many-turtled approach– Individual turtles have been replaced over the
years
– Turtles work well, they have some independence and can talk to each other
![Page 27: Cross Site Data Movement: The UK as a “Data Grid” · orse. Transfer Protocols • Source and destination need to share transfor protocol • Parallel streams (e.g. GridFTP) •](https://reader035.fdocuments.net/reader035/viewer/2022081409/608890fc5e0f607b0729200b/html5/thumbnails/27.jpg)
References
• GridPP: www.gridpp.ac.uk
• WLCG: wlcg.web.cern.ch
• Rucio: rucio.cern.ch
• GridFTP: www.ogf.org/documents/GFD.47.pdf
• SRM: www.ogf.org/documents/GFD.129.pdf
• FTS: fts.web.cern.ch
• IGTF: www.igtf.net
• Redfish www.dmtf.org/standards/redfish
• Swordfish www.snia.org/tech_activities/standards/curr_standards/swordfish