Database Readiness Workshop Summary

Dirk Duellmann, CERN IT

For the LCG 3D project http://lcg3d.cern.ch

SC4 / pilot WLCG Service Workshop

11th February, Mumbai

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 2

Why a LCG Database Deployment

Project?

• LCG today provides an infrastructure for distributed access to file based data and file replication

• Physics applications (and grid services) require a similar services for data stored in relational databases

– Several applications and services already use RDBMS

– Several sites have already experience in providing RDBMS services

• Goals for common project as part of LCG

– increase the availability and scalability of LCG and experiment components

– allow applications to access data in a consistent, location independent way

– allow to connect existing db services via data replication mechanisms

– simplify a shared deployment and administration of this infrastructure during 24 x 7 operation

• Scope set by PEB

– Online - Offline - Tier sites

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 3

3D Participants and Responsibilities

• LCG 3D is a joint project between

– Service users: experiments and grid s/w projects

– Service providers: LCG tier sites including CERN

• Project itself has (as all projects) limited resources (2 FTE)

– Mainly coordinating requirement discussions, testbed and production configuration, setup and support

– Rely on experiments/projects to define and validate their application function and requirements

– Rely on sites for local implementation and deployment of testbed and production setup

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 4

LCG 3D Service Architecture

T2 - local db cache-subset data-only local service

MO

O

O

M

T1- db back bone- all data replicated- reliable service

T0- autonomous reliable service

Oracle Streamshttp cache (SQUID)Cross DB copy &MySQL/SQLight Files

O

Online DB-autonomous reliable service-Pit or CERN CC

F

S S

S S

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 5

Online-Offline Connection

A well-documented schema was reported at the last LCG3D Workshop

Artwork by Richard Hawkings

Slide : A. Vaniachine

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 6

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 7

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 8

LCG Database Deployment Plan

• After October ‘05 workshop a database deployment plan has been presented to LCG GDB and MB

– http://agenda.cern.ch/fullAgenda.php?ida=a057112

• Two production phases • March - Sept ‘06 : partial production service

– Production service (parallel to existing testbed)– H/W requirements defined by experiments/projects– Based on Oracle 10gR2– Subset of LCG tier 1 sites: ASCC, CERN, BNL, CNAF, GridKA, IN2P3, RAL

• Sept ‘06- onwards : full production service– Adjusted h/w requirements (defined at summer ‘06 workshop)

– Other tier 1 sites joined in: PIC, NIKHEF, NDG, TRIUMF

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 9

Proposed Tier 1 Hardware Setup

• Propose to setup for first 6 month

– 2/3 dual-cpu database nodes with 2GB or more• Setup as RAC cluster (preferably) per experiment• ATLAS: 3 nodes with 300GB storage (after mirroring) • LHCb: 2 nodes with 100GB storage (after mirroring) • Shared storage (eg FibreChannel) proposed to allow for clustering

– 2-3 dual-cpu Squid nodes with 1GB or more• Squid s/w packaged by CMS will be provided by 3D• 100GB storage per node• Need to clarify service responsibility (DB or admin team?)

• Target s/w release: Oracle 10gR2

– RedHat Enterprise Server to insure Oracle support

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 10

DB Readiness Workshop last Monday

• http://agenda.cern.ch/fullAgenda.php?ida=a058495

• Readiness of the production services at T0/T1

– status reports from tier 0 and tier 1 sites

– technical problems with the proposed setup (RAC clusters)?

– open questions from sites to experiments?

• Readiness of experiment (and grid) database applications

– Application list, code release, data model and deployment schedule

– Successful validation at T0 and (if required T1)?

– Any new deployment problems seen by experiment users which need a service change

• Review site/experiment milestones from the database project plan

– (Re-)align with other work plans - eg experiment challenges, SC4

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 11

T0 Database Service Evolution

• Until summer 2005

– Solaris based shared Physics DB cluster (2-nodes for HA)• Low CPU power, hard to extend, shared by all experiments

– (many) linux disk servers as DB servers• High maintenance load, no resource sharing, no redundancy

• Now consolidation on extensible database clusters

– No sharing across experiments

– Higher quality building blocks • Midrange PCs (RedHat ES)• FibreChannel attached disk arrays

• As of last month - all LHC services moved

Slide : Maria Girone

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 13

CERN Hardware evolution for 2006

Current State

ALICE ATLAS CMS LHCb Grid 3D Non-LHC Validation

- 2-node offline

2-node 2-node 2-node - - 2x2-node

2-node online test

Pilot on disk server

Proposed structure in Q2 2006 2-node

4-node 4-node 4-node 4--node

2-node 2-node (PDB replacement

)

2-node valid/tes

t

2-node valid/te

st

2-node valid/test

2-node pilot

Compass??

Online?

• Linear ramp-up budgeted for hardware resources in 2006-2008

• Planning next major service extension for Q3 this year

Slide : Maria Girone

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 15

Tier 0 preps

• Database Service extension going according to schedule

– Resource prioritization and extension planning needs experiment involvement (and real deployment experience)

– Significant lead time for h/w orders - need experiment / project requests early!

• Also Streams and Frontier setups proceeding well

– New downstream capture proposal under test - seems promising to avoid some couplings observed in the test bed during site problems

• Need production setup for Database Monitoring (Oracle Grid Control 10gR2)

– Tier 1s may use another local grid control instance

– Two agents reporting into common 3D and local Grid Control

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 18

Tier 1 Progress

• Sites largely on schedule for a service start end of March

– h/w either installed already (BNL, CNAF, IN2P3) or expect delivery of order shortly (GridKA, RAL)

– Some problems with Oracle Clusters technology encountered and solved!

– Active participation from sites - DBA community building up• First DBA meeting focusing on RAC installation, setup and monitoring hosted by Rutherford scheduled for second half of March

• Need to involve remaining Tier 1 sites now

– Establishing contact to PIC, NIKHEF, NSG, TRIUMF to follow workshops, email and meetings

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 19

Service Issues

• Oracle Issues

– X.509(proxy) certificates - will they be supported by Oracle?

– s/w and support licenses for Tier 1

– instant client distribution within LCG

– With commercial Oracle contact (IT-DES group) and IT license officer

• Application Server support

• During initial phase (March-Sept) CMS proposed to support tomcat/frontier/squid setup

• Will discuss other experiments requirements

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 20

Databases in Middleware & Castor

• Took place already for services used in SC3

– Existing setups at the sites

– Existing experience with SC workloads -> extrapolate to real production

• LFC, FTS - Tier 0 and above

– Low volume, but high availability requirements

– CERN: Run on 2-node Oracle cluster; outside single box Oracle or MySQL

• CASTOR 2 - CERN and some T1 sites

– Need to understand scaling up to LHC production rates

– CERN: Run on 3 Oracle servers

• Currently not driving the requirements for the database service

• Need to consolidate databases configs and procedures

– may reduce effort/diversity at CERN and Tier 1 sites

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 21

LCG Application s/w Status

• COOL and POOL have released versions based on CORAL

– Includes re-try and failover required for reliable db service use• These features need be tested for experiment

– POOL includes production version FroNTier plug-in• Control of SQUID caching may still be required to implement more

realistic caching policies

– These releases (or bug fixes) are target for 2006 deployment

• LCG s/w expected to be stable by end of February for distributed deployment as part of SC4 or experiment challenges

• Caveats:

– COOL still has important functionality items on the development plan for this year

– Conditions schema stability will need careful planning for COOL and FroNTier

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 22

Experiment Applications Status

• Conditions - Driving the database service size at T0 and T1

– EventTAGs (may become significant - need replication tests and concrete experiment deployment models)

• Framework integration and DB workload generators exist

– successfully tested in various COOL and POOL/FroNTier tests

– T0 performance and replication tests (T0->T1) looks ok

• Conditions: Online -> Offline replication only starting now

– May need additional emphasis for online tests to avoid surprises

– CMS and ATLAS are executing online test plans

• Progress in defining concrete conditions data models

– CMS showed most complete picture (for Magnet Test)

– Still quite some uncertainty about volumes, numbers of clients

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 23

Test Status : 3D testbed

• Replication test in progress

– Offline->T1: • COOL ATLAS : Stefan Stonjek (CERN, RAL, Oxford)• COOL LHCb : Marco Clemencic (CERN, RAL, GridKA?)• FroNtier CMS : Lee Lueking (CERN and several t1/t2 sites) • ARDA AMGA: Birger Koblitz (CERN->CERN) • AMI : Solveig Albrandt (IN2P3->CERN)

– Online->offline: • CMS Conditions : Saima Iqbal (functional testing)• ATLAS : (Gancho Dimitrov) Server setup, networking config with pit network

• LHCb : planning with LHCb online

• Coordination during weekly 3D meetings

– Status: successful functional test - ramping up volume/load

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 24

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 26

Summary

• Database Production Service and Schedule defined (unchanged since GDB/MB approval)

– Phase 1 - end of March: ASCC, BNL, CERN, CNAF, IN2P3, RAL

– Full deployment - end of September: PIC, NIKHEF, NDG, TRIUMF

• Consolidation with grid service oracle setups

• Setup progressing on schedule at tier 0 and 1 sites

• Application performance tests progressing

• First larger scale conditions replication tests with promising results for streams and frontier technologies

• Concrete conditions data models still missing for key detectors

SC4 / pilot WLCG Service Workshop, 11th February, Mumbai

Dirk Duellmann 27

“My Conclusions”

• There is little reason to believe that a distributed database service will move into stable production any quicker than any of the other grid services

• Should start now with larger scale production operation to resolve the unavoidable deployment issues

• Need the cooperation of experiments and sites to make sure that concrete requests can be quickly validated against a concrete distributed service