30-31 Jan 2003 J G Jensen, RAL/WP5
Storage Elephant
Grid Access to Mass Storage
30-31 Jan 2003 J G Jensen, RAL/WP5
Overview
Control Info Data Xfer
Storage Element
MSS2MSS1
Grid clientsGrid clientsGrid clients
30-31 Jan 2003 J G Jensen, RAL/WP5
Interfaces
• Control• Pin/unpin files (reserve for reading)
• Create new file (reserve for writing)
• Modify metadata
• ...
• Information• Get amount of free space
• Protocols supported by the SE
• ...
• Data Transfer
30-31 Jan 2003 J G Jensen, RAL/WP5
• HTTPS– Reliable, well known protocol– Looking at extensions for Grid security
• Web services– Rapid development– Language independent (but some languages are more
independent than others…)
• GridFTP– Extension of well known FTP protocol, complicated– Achieves good transfer performance
• LDAP• WP3 information producer
– Currently provides only status of daemons• Worker nodes can access their local SE via NFS
– We will investigate using SlashGrid instead – RFIO is trivial to add
Protocols
30-31 Jan 2003 J G Jensen, RAL/WP5
Portability
• RPMs have been or can be built for RedHat 6.2, 6.3, 7.2, 7.3– Support for RedHat 8.x soon
• SE builds from source on both Debian x86 stable and testing and on other Linux.
• TODO: Test on Solaris - ?
30-31 Jan 2003 J G Jensen, RAL/WP5
Client App
API
SE HTTPlibrary
SSL socketlibrary
ApacheTomcat
Axis
RMANMAN
SE core
C Client
SE
Web ServicesClient
Axislibraries
JavaClients
The design of the SE follows a layered model with a central core handling all paths between client and MSS.
Core is flexible and extensible making
it easy to support new protocols, features and MSS
30-31 Jan 2003 J G Jensen, RAL/WP5
RManager
SE CORE
Handler1
Handler2
Handler3
Handler4
Responsereturnedto user
MSS
• Each handler performs a simple task.
• The sequence of handlers depends on type of request, the type of storage system, local configurations, etc.
• Rmanager (request manager) is responsible for running handlers
This architecture is extremely flexible and allows SE to easily meet diverse
requirements for both clients and backend MSS.
30-31 Jan 2003 J G Jensen, RAL/WP5
Interfaces to SE
Control Information Data Transfer
HTTPS
GridFTP
Web S.
Done Done
Nearly done
Planned
Planned
DoneDone Not planned
Not planned
Note: diagram shows existing interfaces, not implemented functionality
30-31 Jan 2003 J G Jensen, RAL/WP5
SE functionality
• For MSS with many features, the SE will be a thin layer
• For MSS with few features (or disk), SE will provide “missing” functionality F
unct
iona
lity
CASTOR ADS Disk
= SE
30-31 Jan 2003 J G Jensen, RAL/WP5
Security• Access control:
– Currently uses Andrew McNab’s GACL– Will also use VoMS (and CAS?)– Fine grained access control – files and dirs all
have ACLs but ACLs can also be shared between files/dirs
• GSI– Will use G-HTTPS– Currently uses Mike Jones’ patch to mod_ssl to
accept proxies
30-31 Jan 2003 J G Jensen, RAL/WP5
SRM Collaboration
• SRM – storage resource manager – collaboration with LBNL, Fermi, Jefferson, CERN
• Particle Physics Mass Storage System interface– Mostly control interface; data transfer done through
GridFTP
• “Old” SRM specification version 1.0 – IDL, web service..?– Some naughtiness: server callbacks
• New specification: interoperability through WSDL– Is WSDL sufficient for interoperability?– Complicated space reservation semantics
30-31 Jan 2003 J G Jensen, RAL/WP5
Non-PP Grids• There exist people who will use Grids for non-PP,
non-HEP projects!• Examples from DataGrid:
– Earth observations (WP9): will write handlers for their own MSS.
• EO applications have many (106) small (2-30 kB) files
– Medical Imaging (WP10): will do the same but WP10 have more challenging requirements
• Confidentiality – data must be enciphered when it goes out to the Grid
• Key will be held in a Grid database on trusted site
• CrossGrid is / will be based on DataGrid software
30-31 Jan 2003 J G Jensen, RAL/WP5
Deployment
• SEs are deployed and working at– CERN, talking to CASTOR using RFIO– RAL, talking to ADS and disks
• SE soon to be deployed for testing and development at INSA Lyon– INSA SE will talk to DICOM (medical images)
• Integration with testbed in March, tested by applications in May
30-31 Jan 2003 J G Jensen, RAL/WP5
Concluding Remarks
• Current interfaces are flexible andcan be expanded to support new commandsand features, or add support for new protocols
• Core is very flexible and it is easy to add new functionality
• We’re making the SE easier to install and maintain – will be installable by sysadmins rather than developers!
Top Related