Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed...
Transcript of Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed...
![Page 1: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/1.jpg)
Multi-VO Rucio
Andrew Lister
![Page 2: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/2.jpg)
OverviewRucio!
Multi-VO?
The Project
Development
Status
Usage
Rucio@RAL
Future work
![Page 3: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/3.jpg)
Rucio!• Distributed data management tool
• Developed to handle HEP domain volumes of data
• Originally built for ATLAS, now supporting CMS and other VOs
• Open source
![Page 4: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/4.jpg)
Multi-VO?A single instance capable of supporting smaller VOs
MICE
T2K
DLS
LSST
MICE
T2K
DLS
LSST
![Page 5: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/5.jpg)
The ProjectThe Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment
My project is a 6-month project in collaboration with ATLAS to develop Multi-VO functionality in Rucio.
IRIS Tasks:• Add Multi-VO support to Rucio, to support multiple experiments in the same instance:
– 2.1 Schema adaptation - Add VO field to all database tables.
– 2.2 Authentication and Base client - Add VO specific authentication
– 2.3 Functionality - Add VO field to all Rucio functions
Developed on an STFC Cloud machine (m1.large, 8GB RAM, 2VCPUs)
![Page 6: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/6.jpg)
DevelopmentSplit into 4 phases, each ending in a Pull Request:
– 1: Separate internal and external representations for RSE
– 2: Internal representations for Account and Scope
– 3: Database Schema updates (2.1)
– 4: Multi-VO features! (2.2 & 2.3)
No impact to Single-VO instances
![Page 7: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/7.jpg)
DevelopmentTravelled to CERN to kick start the project
Joined weekly meetings with developers via vc
Collaborated with core development team via Slack to make important design decisions
Implemented code to the agreed plan
Tested against unit tests, code review, and integration tests
![Page 8: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/8.jpg)
StatusIRIS Tasks:• Add Multi-VO support to Rucio, to support multiple experiments in the same
instance:
– 2.1 Schema adaptation - Add VO field to all database tables.
– 2.2 Authentication and Base client - Add VO specific authentication
– 2.3 Functionality - Add VO field to all Rucio functions
Miscellaneous:– Ensure all daemons work with Multi-VO (16/18)
Still to do: Auditor and c3po (Dark data and Copying popular files)
![Page 9: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/9.jpg)
Usage (Admin)
super_root
root accounts
users
Super_root creates VOs
New VOs have individual root
accounts for standard admin
(e.g. Adding users)
![Page 10: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/10.jpg)
Usage (User – Current)An example of using Rucio for a typical user
Authentication• Add Account, and credentials to “rucio.cfg” config file.
rucio ls user.jdoe:*List Files• Use Rucio CLI
rucio upload --pfn ~/a_file.txt --rse endpoint1 ...Upload Files• Use Rucio CLI
![Page 11: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/11.jpg)
Usage (User – Multi-VO)An example of using Multi-VO Rucio for a typical user
Authentication• Add Account, VO, and credentials to “rucio.cfg” config file.
rucio ls user.jdoe:*List Files• Use Rucio CLI
rucio upload --pfn ~/a_file.txt --rse endpoint1 ...Upload Files• Use Rucio CLI
![Page 12: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/12.jpg)
SKA
Australia
South Africa
Manchester
Cambridge
Queen Mary
Rucio
Rucio@RAL
Elastic
Search ActiveMQ
FTS Service
HA Proxy
rucio-ui
rucio-auth
rucio-daemons
rucio-server
Database
![Page 13: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/13.jpg)
MICERAL Users
Rucio@RALDLS
RAL
Users
T2K
Users
Japan
LSST
Users
Chile
Rucio
![Page 14: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/14.jpg)
Future work
Remaining daemons
Demo Multi-VO at RAL for integration tests
Multi-VO Rucio service running at RAL
Iterate permissions scheme as users join
![Page 15: Template: STFC UKRI Powerpoint - 16:9 widescreen (ppt)€¦ · The Project The Tier-1 is committed to providing a distributed data management service via Rucio to any supported experiment](https://reader035.fdocuments.net/reader035/viewer/2022081402/604bbb43d053853de71f6f23/html5/thumbnails/15.jpg)
Thanks