WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud...

WebFTSFile Transfer Web Interface for

FTS3 Andrea Manzi

On behalf of the FTS team

Workshop on Cloud Services for File Synchronisation and Sharing

Overview

The FTS3 service WebFTS features WebFTS cloud integrations

Dropbox CERNBox

Ongoing development Dropping X509 Data management operations

17/11/2014WebFTS: File Transfer Web Interface for FTS3 2

What is WebFTS?

Web based tool to transfer files between grid/cloud storages

Modular protocol support gsiftp, http(s), xrootd and srm Cloud extensions: dropbox, CERNBox

Development funded by

17/11/2014 3WebFTS: File Transfer Web Interface for FTS3

Provide access to leading technology

Based on FTS3FTS3 is the service responsible for distributing the majority of LHC data across the WLCG infrastructureLow level data movement service, responsible for moving sets of files from one site to another while allowing participating sites to control the network resource usageUsed by LHC VOs + many others VOs part of EGI ~20PB monthly transfer volume / ~2.2M files per

day (WLCG) http://dashb-fts-transfers.cern.ch/ui/


WebFTS architecture

WebFTS: File Transfer Web Interface for FTS3

BROWSERBROWSER

REST APIREST API REST API

GFAL2GFAL2

FTS3FTS3

DAVIXDAVIX GSIFTPGSIFTP DROPBOX

DROPBOX ……

WEBFTS.jsWEBFTS.js

17/11/2014 5

Security

Simpler access while keeping the same level of security X509 + Oauth for Dropbox Transparent delegation of credentials

Avoid storing or transferring any sensitive data Open access to all source code All sensitive information is used within the

browser and forgotten


Delegation

Delegation is needed to let WebFTS access the grid on users behalf Users make private key available to

browser Not available via browser API

VOMS extensions acquired by the service on users behalf

Why it’s important Gives the users a service which can access

the grid for them, from a browser, with full VOMS credentials

WebFTS: File Transfer Web Interface for FTS3 17/11/2014 7

Additional Features

Check-summing and file overwriting

Possibility to resubmit transfer jobs or only-failed files transfers.

Storage Endpoints Auto-completion For endpoints published on the BDII (EGI

and WLCG Information System) Support for LFC Registration

File catalog developed at CERN and used by EGI and WLCG


Success Stories

WebFTS has been successfully tested to transfer from/to:

EUDAT B2Stage ( iRODS DSI) Any gsiftp/webdav/xrootd aware grid storage

( DPM, dCache, Castor, EOS, Storm) HPC Titan @ Oak Ridge National Lab

(ongoing) https://www.olcf.ornl.gov/titan/

Under evaluation by LHCb


Landing page and Guided-tour


Credential delegation


Transfer interface


Job status interface


Extension for Dropbox

Nice way import/export data from the grid world Avoid the installation of new software and

uses what the user has already installed Zero development of clients Multiplatform is given for free

Integration with Oauth By delegating to FTS the right to interact

with dropbox on users behalf Achieved using web tech

Which requires the interactivity of a browser



Extension for Dropbox

Dropbox plugin

Server side the development of a plugin for the metadata management and I/O operations was needed: FTS REST integrates the plugin to perform

metadata management operations FTS3 server uses the plugin to perform

the transfers: GridFTP <-> dropbox Http(s) <-> dropbox


While Dropbox has been integrated via the implementation of a plugin for CERNBox we waited for the new version with EOS as backend ( CERNBox 2.0) We use EOS access via standard grid

protocols ( e.g. xrootd) We map user credentials to correct EOS

namespace The rest comes for free

CERNBox integration


WebFTS With CERNBox



How can we get rid of the delegation step?

1.An Identity Federation: eduGAIN To allow identity providers to authenticate users

at their own institute (SSO)2.A token translation service : STS

To ask the CA for a certificate for the users3.An “IOTA” Certification Authority

To grant the short lived certificate4.VOMS

To accept the new cert as a VO member

Ongoing developments:Access without X509

EDUGAIN


Built on existing federations and infrastructures

CERN participates in eduGAIN via SWITCHaai Many NRENs participate in eduGAIN too

17/11/2014 24

Security Token Service (STS)


• An EMI service• SAML in, X509/VOMS

out17/11/2014 25

“IOTA” CA


VOMS admin


Architecture

WebFTSWebFTS

CERN SSOCERN SSOIdPIdP

Cred

entia

lsAtt

ribut

es

Web

Redi

rect

WAY

F SAM

L

VOMSVOMSIdPIdPIdPIdPIdPIdP

GridStorageElement

GridStorageElement

X.509VOMS

STSSTS

IOTACA

IOTACA

SAM

L

X.50

9VO

MS

Slide adapted from Romain Wartel, GDB Sept 2014

28

Pros/Cons

X509-free access to the grid infrastructure With VOMS support Without modifying all the services

Federated single sign on One password to remember Numerous services potentially accessible

But we need Site acceptance


Not only Transfers..

FTS REST API have been extended to support data management operationsDelete Create/Remove foldersRename

Under integration in WebFTS


Ongoing developments:Data Management

Online service accessible: https://webfts.cern.ch try now! User certificate in your browser

User guide, F.A.Q: Online guided-tour http://fts3-service.web.cern.ch/

documentation/webfts

Official support & code [email protected] https://github.com/cern-it-sdc-id/webfts

Links


Questions?

[email protected]


WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud...

Documents

Transcript of WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud...