The POOL and 3D Projects The LCG POOL and 3D Projects Dirk Duellmann CERN IT-DB Fermilab Computing...
-
Upload
hope-blake -
Category
Documents
-
view
226 -
download
0
Transcript of The POOL and 3D Projects The LCG POOL and 3D Projects Dirk Duellmann CERN IT-DB Fermilab Computing...
The POOL and 3D Projects
The LCG POOL and 3D ProjectsThe LCG POOL and 3D ProjectsThe LCG POOL and 3D ProjectsThe LCG POOL and 3D Projects
Dirk DuellmannDirk DuellmannCERN IT-DBCERN IT-DB
Fermilab Computing Seminar, Fermilab Computing Seminar, 21. October 200421. October 2004
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 22
OutlineOutline
• What is POOL? What is POOL? – POOL architecture and implementation choices– Integration into experiment s/w frameworks– Experience in Data Challenges– New developments this year
• Distributed Database DeploymentDistributed Database Deployment– Project scope– Constraints from application and deployment side– First ideas for a service architecture
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 33
What is POOL? What is POOL?
• A common A common Persistency FrameworkPersistency Framework for LHC physics applications for LHC physics applications– Pool Of persistent Objects for LHC
• Common effort from Common effort from LHC experimentsLHC experiments and CERN and CERN databasedatabase / / softwaresoftware groups groups – for defining its architecture and features – for the development of its components
• Part of the LHC Computing Grid (LCG)Part of the LHC Computing Grid (LCG)– One of the first Application Area Projects (started in June 2002)
• About 25 public releases, 120 internal releasesAbout 25 public releases, 120 internal releases– first public: December 2002 (0.3.0)– first fully functional release: May 2003 (1.0.0)– production releases: March – June 2004 (1.6 series)
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 44
POOL Objectives POOL Objectives
• To allow the To allow the multi-PB of experiment of experiment data and associated meta data to be to be stored in a distributed and Grid enabled fashion stored in a distributed and Grid enabled fashion – various types of data of different volumes
• Hybrid technology approach, combining, combining – C++ object streaming technology, [ ROOT I/O ] for the bulk data – Transactionally safe Relational Database (RDBMS) services, such as
MySQL or Oracle, for catalogs, collections and metadata
• In particular, it provides In particular, it provides – Persistency for C++ objects – Transparent navigation across file and technology boundaries– Integrated with an external File Catalog to keep track of the file physical
location, allowing files to be moved or replicated
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 55
LCG Component ArchitectureLCG Component Architecture
• POOL is a component based system following the LCG POOL is a component based system following the LCG Architecture BlueprintArchitecture Blueprint
• POOL provides a technology neutral API POOL provides a technology neutral API – Abstract component C++ interfaces – Insulates the experiment framework user code from several
concrete implementation details and technologies used today • POOL user code is not dependent on implementation libraries POOL user code is not dependent on implementation libraries
– No link time dependency on implementation packages (e.g. MySQL, Oracle, Root, Xerces-C..)
– Backend component implementations are loaded at runtime via the LCG- SEAL plug-in infrastructure
• Three major domains, weakly coupled, interacting via abstract Three major domains, weakly coupled, interacting via abstract interfacesinterfaces
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 66
POOL Component BreakdownPOOL Component Breakdown
POOL API
Storage Service
FileCatalog Collections
ROOT I/OStorage Svc
XMLCatalog
MySQLCatalog
Grid ReplicaCatalog
ExplicitCollection
ImplicitCollection
RelationalStorage Svc
• POOL is (mainly) a client-side packagePOOL is (mainly) a client-side package– Coupling to standard file, database and grid services. No
specialized POOL servers!
• Storage Manager Storage Manager – Streams transient C++ objects to/from disk – Resolves a logical object reference to a physical object – I/O via ROOT (rfio/dcache) or RDBMS
(Oracle/MySQL/SQLite)
• File Catalog File Catalog – Maintains consistent lists of accessible files
together with their unique identifiers (FileID), – Used to resolves the logical file reference (from a POOL
Object ID) to a physical file • Collections Collections
– Implements (large) containers for persistent objects (eg event collections) stored via POOL
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 77
POOL on the Grid POOL on the Grid
File Catalog
Collections
Meta Data
Replica Location Service
Meta Data Catalog
Grid Dataset Registry G
rid
Reso
urce
s
Exp
erim
en
t Fra
mew
ork
Use
r A
pplica
tion
LCG POOL Grid Middleware
RootI/O
Replica Manager
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 88
POOL off the Grid POOL off the Grid
File Catalog
Collections
Meta Data
XML or MySQLCatalogs
MySQL or SQLite
MySQL or RootIOCollections
Exp
erim
en
t Fra
mew
ork
Use
r A
pplica
tion
LCG POOL Disconnected Laptop
RootI/O
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 99
Navigational Access via TokensNavigational Access via Tokens
• Unique Object Identifier (OID) per object Unique Object Identifier (OID) per object (also called a POOL Token)(also called a POOL Token)– Direct access to any object in the distributed store
– Natural extension of the pointer concept
• Tokens allow to implement networks of persistent objects Tokens allow to implement networks of persistent objects (“associations”)(“associations”)– Association Cardinality: 1:1 , 1:n, n:m
• POOL tokens are not directly exposed to the end userPOOL tokens are not directly exposed to the end user– POOL provides templated “smart-pointer” pool:ref<T>
• Smart pointer type which calls back to POOL to open files / database Smart pointer type which calls back to POOL to open files / database connections and loads objects into memoryconnections and loads objects into memory– Physical information like file or hostnames are not exposed
FileID ContainerIDObjectN
o
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 1010
Physical and Logical Data ModelPhysical and Logical Data Model
• Important to clearly separate between the two models• Avoids polluting user code with unnecessary detail about file and host names• Leaving the physical storage open to optimise for performance or changing access patterns without affecting applications
File CatalogFile Catalog
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 1111
Storage ComponentsStorage Components
Data Cache StorageSvc
DiskStoragedatabasedatabaseDatabase
ContainerContainerContainer
ObjectsObjectsObjectsObjectsObjectsObjectsObjectsObjects
•Object Pointer
•Persistent Address
Technology
DB name
Cont.nameItem ID
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 1212
POOL Cache AccessPOOL Cache Access
Key
CacheMgr
Pointer
Ref<T>Ref<T>
Cache Manager
Persistency Service
TokenToken
Storage TechnologyStorage Technology
Object TypeObject Type
Persistent LocationPersistent Location
KeyKey ObjectObject TokenToken
…… …… ……
22 <pointer><pointer> <pointer><pointer>
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 1313Technology dependent
.h.h
ROOTCINTROOTCINT
CINT dictionary codeCINT dictionary code
DictionaryGeneration
CIN
T
dic
tio
nar
yC
INT
d
icti
on
ary
I/O I/O
Data I/O
Dictionary: Population/ConversionDictionary: Population/Conversion
GCC-XMLGCC-XML
LCG dictionary codeLCG dictionary code
.xml.xml
Code GeneratorCode Generator
LC
Gd
icti
on
ary
LC
Gd
icti
on
ary
Gat
eway
Gat
eway
Reflection
Oth
er
Cli
en
tsO
the
r C
lie
nts
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 1414
DevelopmentsThis YearDevelopmentsThis Year
• Move to ROOT4 (POOL2.0 Line)Move to ROOT4 (POOL2.0 Line)– To take advantage of automatic schema evolution and simplified streaming of STL
containers• Need to insure backward compatibility for POOL 1.x files
– Currently undergoing validation by the experiments• Will release two branches until POOL 2 is fully certified
• File Catalog deployment issuesFile Catalog deployment issues– DC productions showed some weaknesses of grid catalog implementations
• Several new/enhanced catalogs coming up• Changes in the experiment computing models need to be taken into account
– POOL tries to generalise from specific implementations and provides an open interface to accommodate upcoming components
• CollectionsCollections– Several implementations of POOL collections exist– Collection cataloguing has been added in response to experiment requests
• Similar to file catalogs• re-use of catalog implementation and commandline tools
– Experiment analysis models are still being concretized – Expect experience from concrete analysis challenges
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 1515
Why a Relational Abstraction Layer (RAL)?Why a Relational Abstraction Layer (RAL)?
• Goal: Vendor independence for the relational components of POOL, Goal: Vendor independence for the relational components of POOL, ConditionsDB and user codeConditionsDB and user code– Continuation of the component architecture as defined in the LCG
Blueprint– File catalog, collections and object storage run against all available
RDBMS plug-ins
• To reduced code maintenance effortTo reduced code maintenance effort– All RDBMS client components can use all supported back-ends– Bug fixes can be applied once centrally
• To minimise risk of vendor binding To minimise risk of vendor binding – Allows to add new RDBMS flavours later or use them in parallel and are
picked up by all RDBMS clients– RDBMS market is still in flux..
• To address the problem of distributing data in RDBMS of different To address the problem of distributing data in RDBMS of different flavoursflavours– Common mapping of application code to tables simplifies distribution of RDBMS
data in a generic application independent way
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 1616
Relational Access functionality• Database Schema Access and ManipulationDatabase Schema Access and Manipulation
– Describing existing and creating new tables– Support for primary, foreign keys and indices
• Formed by one or more table columns
• Data Manipulation LanguageData Manipulation Language– Insertion, update and deletion of table rows– Bulk insertions to minimise database server roundtrips
• QueriesQueries– Nested queries involving one or more tables– Ordering and limiting the result set– Control of client cache for the result set– Database cursors
• scalable iteration through large query results
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 1818
How does this fit into POOL?How does this fit into POOL?
uses
Abstract interface
Implementation
implements
Technology dependent plugin
FileCatalog Collection StorageSvc
Experiment framework
RelationalAccess Seal reflection
ObjectRelationalAccessRelationalCollection
RelationalCatalog
RelationalStorageSvc
MySQL Oracle SQLite
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 2020
RDBMS plug-ins in POOL
• Oracle 9i/10gOracle 9i/10g– Based on OCI– Supports Oracle instant client – Fully supports the POOL RAL interfaces– Available for the Linux platforms (win32 will follow)
• SQLiteSQLite– A light-weight embeddable SQL database engine– File-based (zero configuration, administration)– Available for the Linux and Win32 platforms
• MySQLMySQL– Implementation based on the MyODBC driver– Prototype released with POOL 1.8
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 2121
Object to Relational MappingObject to Relational Mapping
• How to map classes ↔ tables ?How to map classes ↔ tables ?– Both C++ and SQL allow to describe data
layout– But with very different constraints/aims
• no single unique mapping
• Need for fast object navigation an unique Need for fast object navigation an unique Object identity (persistent address)Object identity (persistent address)– requires unique index for addressable objects– part of mapping definition
• POOL stores mapping with the object dataPOOL stores mapping with the object data– including mapping versions
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 2222
A Mapping ExampleA Mapping Example
class A {class A { int x;int x; float y;float y; std::vector<double> v;std::vector<double> v; class B {class B { int i;int i; std::string s;std::string s; } b;} b;};};
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 2323
A Mapping ExampleA Mapping Example
.....
“Hi”32.2222
“Hello”31.4101
B_SB_IYXID
T_A
0.132
0.122
32.112
5.45241
4.131
12.221
0.1211
VPOSID
T_A_Vp.k. f.k. constraint
This is only one of the possible
mappings!
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 2424
Mapping Elements
• A complete mapping consists of A complete mapping consists of – A mapping version per object– A hierarchical tree of mapping elements per version
• Each mapping element containsEach mapping element contains– Element type (“Object”, “Primitive”, “Array”, “POOL
reference”, “Pointer”)– Database table and column names– C++ member name and type– Lower level associated mapping elements
• POOL stores these persistently in 3 (hidden) POOL stores these persistently in 3 (hidden) relational tablesrelational tables
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 2525
Generating a Mapping..
• Two use cases need to be supportedTwo use cases need to be supported1) Starting from existing table schema and data
• Give access to RDBMS data with minimal changes to existing data
• POOL generates default header and mapping from the DB schema
2) Starting from existing C++ header file
• Implement existing class with minimal changes to user C++code
• POOL generates default DB schema and mapping from the LCG dictionary entry
• In both cases the user can override a default mapping via an xml In both cases the user can override a default mapping via an xml steering filesteering file
• Select the C++ classes which are mapped
• Override default mapping rules (eg member names and types)
• Define the mapping version
– Mapping is stored in the database with a command line tool
– Will provide support for mapping copies and extraction from DB
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 2626
POOL File Catalog ModelPOOL File Catalog Model
• POOL adds system generated POOL adds system generated FileIDFileID to standard Grid m-n mapping to standard Grid m-n mapping – Stable inter-file reference
• POOL model includes file-level POOL model includes file-level meta-datameta-data for production catalog administration for production catalog administration – used to query large file catalogs– e.g. extract partial catalogs (fragments) based on production parameters
• Fragments can be shipped (+ referenced files) to other sites / decoupled production nodesFragments can be shipped (+ referenced files) to other sites / decoupled production nodes– POOL command line tools allow cross-catalog +cross-implementations operations– End-users can connect to several catalogs at once
• Different implementations can be mixed; Only one per context can be updated
Logical NamingLogical Naming
Object LookupObject Lookup
FileIDLFN1LFN1 PFN1PFN1
LFN2LFN2
LFNnLFNn
PFN2PFN2
PFNnPFNn
File metadataFile metadataeg jobid, owner…
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 2727
POOL Deployment in the Grid POOL Deployment in the Grid
• Coupling to Grid services Coupling to Grid services – Middleware based on the EDG-RLS; Service uses Oracle Application Server + DB– Connects POOL to all LCG files
• Local Replica Catalog (LRC) for GUID <-> PFN mapping for all local files• Replica Metadata Catalog (RMC) for file level meta-data and GUID <-> LFN• Replica Location Index (RLI) to find files at remote sites (not deployed in LCG)
Resulted in a single centralized catalog at CERN (scalability and availability concerns)
But Grid-decoupled modes also required by production use-casesBut Grid-decoupled modes also required by production use-cases
XML Catalog• Typically used as local file by a single user/process at a time
– no need for network – supports R/O operations via http; tested up to 50K entries
Native MySQL Catalog
• Shared catalog e.g. in a production LAN – handles multiple users and jobs (multi-threaded); tested up to 1M entries
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 2828
Use case: isolated systemUse case: isolated system
jobslookuplookup
input input filesfiles
registerregister
output filesoutput files
XML
EDG-RLS
Import
Import
No networkPublis
h
Publish
• The user extracts a set of interesting files and a catalog fragment The user extracts a set of interesting files and a catalog fragment describing them from a (central) Grid based catalog into a local XML describing them from a (central) Grid based catalog into a local XML catalogcatalog– Selection is performed based on file or collection descriptions
• After disconnecting from the Grid the user executes some standard jobs After disconnecting from the Grid the user executes some standard jobs navigating through the extracted data navigating through the extracted data – New output files are registered into the local XML catalog
• Once the new data is ready for publishing and the user is connected the Once the new data is ready for publishing and the user is connected the new catalog fragment is submitted to the Grid based catalognew catalog fragment is submitted to the Grid based catalog
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 2929
Use case: farm productionUse case: farm production
jobs
jobs
lx1.cern.chlx1.cern.ch
pc3.cern.chpc3.cern.ch
lookuplookup
lookuplookup
registerregister
registerregister
quality quality checkcheck
quality quality checkcheck
publish
publish
publishpublish
MySQL
XML
publish
publish
publish
publish
EDG
• A production job runs and creates files and their catalog entries in a local XML A production job runs and creates files and their catalog entries in a local XML file file
• During the production the catalog can be used to cleanup filesDuring the production the catalog can be used to cleanup files
• Once the data quality checks have been passed the production manager Once the data quality checks have been passed the production manager decides to publishes the production XML catalog fragment to the site database decides to publishes the production XML catalog fragment to the site database one and eventually to the Grid based catalogone and eventually to the Grid based catalog
D.D
uell
man
n, C
ER
N
Slide 30
CMS DC04CMS DC04
Demonstrate the capability of the CMS computing system to cope with a sustained rate of 25Hz for one month
Started in March 2004 based on the PCP04 pre-production (simulation)
Reconstruction phase including POOL output concluded in April 2004
Distributed end-user analysis based on this data is continuing
digitization
reconstruction
Total amount of data (TB)
24.5 4
Throughput (GB/day) 530 320Tot num of jobs (1k) 35 25Jobs/day 750 2200
D.D
uell
man
n, C
ER
N
Slide 31
CMS DC04 Data CatalogCMS DC04 Data Catalog
During DC04, the RLS was used both as a File Catalog and as a metadata catalog to store POOL file-level attributes
XML catalog fragments were published in the RLS
About 570k logical filenames, 2M physical filenames and 9 attributes/file
POOL MySQL Catalog used at some Tier1 (FNAL)
[ Oracle multi-master replication tested outside production ]
MySQL
D.D
uell
man
n, C
ER
N
Slide 32
CMS DC04 Problems CMS DC04 Problems
RLS backend showed significant performance problems in file-level meta-data handling
Queries and meta data model became concrete only during the data challenge GUID<->PFN queries 2 orders magnitude faster on POOL MySQL than RLS LRC-RMC cross queries 3 orders magnitude faster on POOL MySQL than RLS
Main causes: overhead of SOAP-RPC protocol missing support for bulk operations in EDG-RLS catalog implementation
Transaction support missing Failures during a sequence of inserts/updates require recovery “by hand”
Basic lookup / insert performance satisfactory
The POOL model for handling a cascade of file catalogs is still valid Good performance of POOL XML and MySQL backends proves this RLS backend problems being addressed now by IT-Grid Deployment Group
Good stability of the RLS service achieved!
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 3333
POOL SummaryPOOL Summary
• The LCG POOL project provides a hybrid store integrating The LCG POOL project provides a hybrid store integrating object streaming with RDBMS technologyobject streaming with RDBMS technology– POOL has been successfully integrated into three quite different LHC
experiments software frameworks– Successfully deployed as baseline persistency mechanism for CMS,
ATLAS and LHCb at the scale of ~400TB• POOL continues the LCG component approach by abstracting POOL continues the LCG component approach by abstracting
database access in a vendor neutral waydatabase access in a vendor neutral way– A Relational Abstraction Layer has been released and is being picked
up by several experiments– Minimised risk of vendor binding, simplified maintenance and data
distribution are the main motivations• POOL as a project is (slowly) migrating to a support and POOL as a project is (slowly) migrating to a support and
maintenance phase maintenance phase – Need keep remaining manpower focused in order to finish remaining
developments and to provide relevant support to user community
The POOL and 3D Projects
Distributed Database Deployment for Distributed Database Deployment for the LHC Experimentsthe LHC Experiments
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 3535
Why a LCG Database Deployment Project?Why a LCG Database Deployment Project? • LCG today provides an infrastructure for distributed access to file based LCG today provides an infrastructure for distributed access to file based
data and file replicationdata and file replication• Physics applications (and grid services) require a similar services for Physics applications (and grid services) require a similar services for
data stored in relational databasesdata stored in relational databases– Several applications and services already use RDBMS
– Several sites have already experience in providing RDBMS services
• Goals for common project as part of LCG Goals for common project as part of LCG – increase the availability and scalability of LCG and experiment components
– allow applications to access data in a consistent, location independent way
– allow to connect existing db services via data replication mechanisms
– simplify a shared deployment and administration of this infrastructure during 24 x 7 operation
• Need to bring service providers (site technology experts) closer to Need to bring service providers (site technology experts) closer to database users/developers to define a LCG database service database users/developers to define a LCG database service – Time frame: First deployment in 2005 data challenges (autumn ‘05)
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 3636
Project Non-GoalsProject Non-Goals
Store all database data• Experiments are free to deploy databases and distribute data
under their responsibility
Setup a single monolithic distributed database system• Given constraints like WAN connections one can not assume
that a single synchronously updated database work and provide sufficient availability.
Setup a single vendor system• Technology independence and a multi-vendor implementation
will be required to minimize the long term risks and to adapt to the different requirements/constraints on different tiers.
Impose a CERN centric infrastructure to participating sites• CERN is one equal partner of other LCG sites on each tier
Decide on an architecture, implementation, new services, policies• Produce a technical proposal for all of those to LCG PEB/GDB
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 3737
Situation on the Application SideSituation on the Application Side
• Databases are used by many applications in the physics production chainDatabases are used by many applications in the physics production chain
• Currently many of these applications are run centralizedCurrently many of these applications are run centralized– Several of these applications expect to move to a distributed model for scalability
and availability reasons
• This move can be simplified by a generic LCG database distribution This move can be simplified by a generic LCG database distribution infrastructure - but infrastructure - but still will not happen by magicstill will not happen by magic
• Choice of the supported database Choice of the supported database – Is often made by application developers– Not necessarily yet with the full deployment environment in mind
• Need to continue to make key applications vendor neutralNeed to continue to make key applications vendor neutral– DB abstraction layers exist or are being implemented in many foundation libraries
• OGSA-DAI, ODBC, JDBC, ROOT, POOL, … are steps in this direction• Degree of the abstraction achieved varies
– Still many applications which are only available for one vendor• Or have significant schema differences which forbid DB<->DB replications
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 3838
Database Services at LCG Sites TodayDatabase Services at LCG Sites Today
• Several sites provide Oracle production services for HEP Several sites provide Oracle production services for HEP and non-HEP applicationsand non-HEP applications– Deployment experience and procedures exists…– … but can not be changed easily without affecting other site
activities
• MySQL is very popular in the developer communityMySQL is very popular in the developer community– Used for some production purposes in LHC, though not at large
scales– Expected to deployable with limited db administration resources– So far no larger scale production service exists at LCG sites
• But several applications are bound to MySQL
• Expect a significant role for both database flavorsExpect a significant role for both database flavors– To implement different parts of the LCG infrastructure
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 3939
Local Database vs. Local CacheLocal Database vs. Local Cache
• FNAL experiments deploy a combination of http based FNAL experiments deploy a combination of http based database access with web proxy caches close to the clientdatabase access with web proxy caches close to the client– Performance gains
• reduced real database access for largely read-only data• reduced transfer overhead compared to low level SOAP RPC
based approaches – Deployment gains
• Web caches (eg squid) are much simpler to deploy than databases and could remove the need for a local database deployment on some tiers
• No vendor specific database libraries on the client side• “Firewall friendly” tunneling of requests through a single port
• Expect cache technology to play a significant role towards Expect cache technology to play a significant role towards the higher tiers which may not have the resources to run a the higher tiers which may not have the resources to run a reliable database servicereliable database service
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 4040
OracleOracle MySQLMySQL
RAL
APP
network
db file storagedb file storage
db & cacheservers
db & cacheservers
client s/wclient s/w
webcache
webcache
SQLite fileSQLite file
Application s/w stack and Distribution OptionsApplication s/w stack and Distribution Options
RAL = relational abstraction layer
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 4242
Tiers, Resources and Level of ServiceTiers, Resources and Level of Service
• Different requirements and service capabilities for different tiersDifferent requirements and service capabilities for different tiers– Tier1 Database Backbone
• High volume, often complete replication of RDBMS data• Can expect good network connection to other T1 sites• Asynchronous, possibly multi-master replication• Large scale central database service, local dba team
– Tier2 • Medium volume, often only sliced extraction of data • Asymmetric, possibly only uni-directional replication• Part time administration (shared with fabric administration)
– Tier3/4 (eg Laptop extraction)• Support fully disconnected operation • Low volume, sliced extraction from T1/T2
• Need to deploy several replication/distribution technologiesNeed to deploy several replication/distribution technologies– Each addressing specific parts of the distribution problem– But all together forming a consistent distribution model
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 4343
M
Starting Point for a Service Architecture?Starting Point for a Service Architecture?
O
O
O
M
T1- db back bone- all data replicated- reliable service
T2 - local db cache-subset data-only local service
T3/4
M
M
T0- autonomous reliable service
Oracle StreamsCross vendor extractMySQL FilesProxy Cache
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 4444
LCG 3D Project LCG 3D Project
WP1 -Data Inventory and Distribution RequirementsWP1 -Data Inventory and Distribution Requirements– Members are s/w providers from experiments and grid services
that use RDBMS data– Gather data properties (volume, ownership) requirements and
integrate the provided service into their software
WP2 - Database Service Definition and ImplementationWP2 - Database Service Definition and Implementation– Members are site technology and deployment experts– Propose a deployment implementation and common deployment
procedures
WP3 - Evaluation TasksWP3 - Evaluation Tasks– Short, well defined technology evaluations against the
requirements delivered by WP1– Evaluation are proposed by WP2 (evaluation plan) and typically
executed by the people proposing a technology for the service implementation and result in a short evaluation report
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 4545
Data InventoryData Inventory
• Collect and maintain a catalog of main RDBMS data typesCollect and maintain a catalog of main RDBMS data types– Select from catalog of well defined replication options
• which can be supported as part of the service– Conditions and Collection/Bookkeeping data are likely candidates
• Experiments and grid s/w providers fill a table for each data type which is Experiments and grid s/w providers fill a table for each data type which is candidate for storage and replication via the 3D servicecandidate for storage and replication via the 3D service– Basic storage properties
• Data description, expected volume on T0/1/2 in 2005 (and evolution)• Ownership model: read-only, single user update, single site update, concurrent
update– Replication/Caching properties
• Replication model: site local, all t1, sliced t1, all t2, sliced t2 …• Consistency/Latency: how quickly do changes need to reach other sites/tiers • Application constraints: DB vendor and DB version constraints
– Reliability and Availability requirements• Essential for whole grid operation, for site operation, for experiment
production, • Backup and Recovery policy
– acceptable time to recover, location of backup(s)
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 4646
Service Definition and ImplementationService Definition and Implementation
• DB Service DiscoveryDB Service Discovery– How does a job find a close by replica of the database it needs?– Need transparent (re)location of services - eg via a database replica catalog
• Connectivity, firewalls and connection constraintsConnectivity, firewalls and connection constraints• Access Control - authentication and authorizationAccess Control - authentication and authorization
– Integration between DB vendor and LCG security models• Installation and ConfigurationInstallation and Configuration
– Database server and client installation kits• Which database client bindings are required (C, C++, Java(JDBC), Perl, ..) ?
– Server and client version upgrades (eg security patches)• Are transparent upgrades required for critical services?
– Server administration procedures and tools• Need basic agreements to simplify shared administration• Monitoring and statistics gathering
• Backup and RecoveryBackup and Recovery– Backup policy templates, responsible site(s) for a particular data type – Acceptablelatency for recovery
• Bottom line: service effort should not be underestimated!Bottom line: service effort should not be underestimated!– We are rather close to LHC startup and can only afford to propose models that have a good
chance of working!– Do not just hope for good luck; These services will be a critical part of the experiments’
infrastructure and should be handled accordingly!
The POOL and 3D ProjectsThe POOL and 3D Projects D.Duellmann, CERND.Duellmann, CERN 4747
SummarySummary
• Together with the LHC experiments LCG will define and deploy a distributed Together with the LHC experiments LCG will define and deploy a distributed database service at Tier 0-2 sitesdatabase service at Tier 0-2 sites– Several potential experiment applications and grid services exist but need to be coupled to
the upcoming services• development work will be required on the 3D service and the application side
– Difference in available T0/1/2 manpower resources will result in different level of service • a multi-vendor environment has been requested to avoid of vendor coupling and to
support the existing s/w base
• The 3D project ( The 3D project ( http://lcg3d.cern.chhttp://lcg3d.cern.ch ) has been started in the LCG deployment area ) has been started in the LCG deployment area to coordinate this activityto coordinate this activity– Meetings in the different working groups are starting to define the key requirements and
verify/adapt the proposed model– Prototyping of reference implementations of the main model elements has started and
should soon be extended to a (small) multi-site test-bed
• Need to start pragmatic and simple to allow for first deployment in 2005Need to start pragmatic and simple to allow for first deployment in 2005– A 2005 service infrastructure can only draw from already existing resources– Requirements in some areas will only become clear during first deployment when the
computing models in this area firm up