Bologna, 21 Marzo 2006 Riccardo Zappi / Luca Magnoni INFN-CNAF, Bologna.

Bologna, 21 Marzo 2006

Riccardo Zappi / Luca MagnoniINFN-CNAF, Bologna

INFN Storage WorkshopBologna, 21 Marzo

2006

Summary

1.What SRM is2.What StoRM is3.StoRM status4.StoRM next release 5.Tests done on StoRM

6.Conclusion


2006

What SRM is• SRMs are middleware components that manage

shared storage resources on the grid and provide management functionalities like:– Uniform access to heterogeneous types of

storage– File pinning– Disk space allocation and advanced disk space

reservation– Protocol negotiation– Life time management of files– Management of security

• SRMs implements a SRM interface specified by GSM-WG within GGF. (v.1.1, v.2.1.1 and v.3.0)


2006

Storage System

GPFS

Storage System

LUSTRE

Storage System

DCACHEdC

Storage System

CASTOR

SRM-dCACHE

Managent Access & Data Access to SE

Access Protocol

rfio

Access Protocol

dCapAccess Protocol

posix

Data Access

Management Access

SRM-Castor StoRMManagement Protocol

SRM v.1.1

Management Protocol

SRM v.2.1.1


2006

• GFAL– a low-level IO interface to Grid Storage– Storage Element Abstraction ( SRM v1)

• lcg_util– Provides both direct C library calls and CLI tools– Is a thin wrapper on top of GFAL SRM v1

• FTS– The FTS receive requests either through their Web Service

interface directly from the clients or indirectly through the Data Scheduler.

– The transfers managed by the FTS are all asynchronous, i.e. the client submits a transfer ’job’, which may contain a list of files (with proper source and destination qualifiers) to be transferred.

– Current version use SRM v1.

How SRM-service should be used?


What StoRM is


2006

What StoRM is

• StoRM is a Storage Resource Manager.

• It is exposed as web service.– Web service description (wsdl) of StoRM is compliant with

SRM specification version 2.1.1 (WS-I compliant).

• StoRM uses Cluster File System with ACL support.

• Current StoRM version is built on top of GPFS

• StoRM allows direct access (native posix call)

– StoRM works without server I/O too.

– Aggregation is performed by underlying file system (better than StoRM could do).


2006

StoRM is different …

Gli accessi allo storage avvengono con modalità Posix-like. L’interfaccia esposta NON è del File System, ma è un wrapping fatto attraverso qualche libreria.. (dcap, rfio, ecc.)

Gli accessi allo storage avvengono attraverso chiamate POSIX esposte dal file system sottostante. Nel caso di Parallel File System, si possono sfruttare in pieno le performance offerte!!

I metadati relativi ai file e allo storage (come il namespace, space management, ecc..) sono gestiti completamente a livello applicativo, replicando le funzionalità tipiche dei file system (performance? Scalabilita?)

I metadati relativi ai file e allo storage, (quando possibile), sono gestiti completamente dal file system sottostante. Si evita così di replicare le stesse funzionalità del file system a livello applicativo (performance! Scalabilita!)


2006

• Front end (FE) has responsibilities of :

– expose a web service interface – manage connection with authorized clients – store asynchronous request into data

base. – retrieve asynchronous request status. – co-operate with backend directly for

synchronous call. – co-operate with external authorization

service to enforce security policy on service.

– manage user authentication

• Data Base :– Store SRM request and status– Store application data

• Back end (BE) has responsibilities of :

– accomplish all synchronous (active) action.– get asynchronous request from data base. – accomplish all asynchronous action.– bind with underlying file system. – enforce authorization policy on files– manage SRM file and space metadata.

StoRM architecture


2006

StoRM Security Aspects

1. User perform srmPrepareToGet

2. StoRM verifies if the principal holds a valid proxy certificate and delegates the external policy decision point to validate the request.

3. StoRM then queries the Authorization Sources to verify if the user can perform the specified operation on the SURL

4. StoRM queries LCMAPS to obtain local user account corresponding to the grid identity of the requestor

5. Physical file name derives by SURL and user attributes (Virtual organization name space)

6. The file system wrapper enforces permissions by setting a new ACL on the physical file.

7. The user job can be executed into the worker node

8. The application can perform a standard POSIX call to access the file into/from the storage system.


2006

• Current release provides Command Line Clients.

• These Clients are intended for Test use only.

• FTS, lcg-util and GFAL are not usable with SRM v.2.1 yet.

• Interoperability Clients are going to be used with StoRM.

StoRM “Command Line” clients


StoRM : Status1) Current release

2) Next release


2006

StoRM status – Release• First release [v 1.0.0]

– End of November 2005 : base version• Christmas release [v 1.0.1]

– Xmas 2005 : patched version• Second release [v 1.0.2]

– End of January 2006 (CHEP version) : directory functions.

Current release provides these functionalities:- Data transfer : srmCopy, srmPtG, srmPtP, srmStatus<XXX>- Space Management : srmReserveSpace,

srmGetSpaceMetadata- Directory : srmLs, srmRm, srmMkDir, srmRmdir.

Documentation (installation and configuration) available.Web Site of project : User management to better support


2006

• Third release [v 1.1.0] • pre INFN storage workshop• Build completed ( Friday 17 March ..)

• Important improvements implementedNew clients– Interoperability (WSDL WS-I compliant, SRM v.2.1.1_modified)– Inline help improved– Parameter syntax checking Front End improved– WSDL WS-I compliant– Larger space reservation (2^64 byte)– Better explanation in case of failureBack End improved– Space Reservation : multiple utilization – “thread-safe” LCMAPS usage– performance tuning

StoRM status – Today


2006

StoRM status – fine Aprile/Maggio

• Fourth release [v 1.2.0] • Scheduled for May (check web site for news).

• Important improvements designed:Client improvement– More parameters available– Client-API on space management for Advance Reservation service.

Front End improvement– VOMS – GPBOX authorization service integration– Checking on malformed parameters– Information Service Integration

Back End improvement– Configuration file based on XML format (hierarchic and more expressive)– Authorization source integration (LFC and/or ad-hoc solution)– New functionalities : service monitoring, …

Web site and documentation – New documentation (installation and configuration) in different media (wiki).


StoRM : TEST


2006

StoRM TESTING – current version

• Third release [v 1.1.0] •Site A : Tier1 (22TB)•Site B : Bari (2TB)•Site C : ICTP-Trieste (30GB)•Site D : CNAF-Cert-SE (50GB)

• Tests•Functionality tests : Sito C e D•Stress tests : Sito A•Transfer tests : Siti A B


2006

Test resultsData transfer T1 to/from Bari

via srmCopy v.2.1.150 parallel srmCopy with• From SURL at CNAF• To SURL at BARI • 1GB file size, everyone

Bandwidth saturated.

Only 100Mb/s


StoRM : Releasing process

Group dedicated to

• Developing

• Documentation

• Testing and integration

• Prototyping


2006

Releasing process


2006

• StoRM service development involves many aspects (Development, Integration, Testing,

Prototyping, Dissemination).

Important progress done on each service component.

• StoRM will be in production version by June!

• We need help on Testing and Prototyping.

• Help on integration is welcome.

Conclusions


2006

• Alberto Forti (CNAF), from 01 April

• Luca Magnoni (CNAF), 100%

• Riccardo Zappi (CNAF), 100%

• Ezio Corso (ICTP), 100%

• Antonio Messina (ICTP), 50%

• Riccardo Murri (ICTP), 50%

Project managed by Antonia Ghiselli.

StoRM Develop Team


2006

Questions and Suggestions

?!

Bologna, 21 Marzo 2006 Riccardo Zappi / Luca Magnoni INFN-CNAF, Bologna.

Documents

Transcript of Bologna, 21 Marzo 2006 Riccardo Zappi / Luca Magnoni INFN-CNAF, Bologna.