WLCG Status Ian Bird Overview Board 3 rd December 2010 Project report: Brief status update: preparation for and experience with HI Resources usage Tier 0 for information EGI/EMI follow up from last OB Future actions Middleware review Data management Virtualisation SIR analysis Follow up on action from last meeting Planning 2011,12,13 Considerations in view of prospects for data Agenda for today Following request at last OB: https://espace.cern.ch/WLCG-Overview- Board/Shared%20Documents/Forms/AllItems.aspx https://espace.cern.ch/WLCG-Overview- Board/Shared%20Documents/Forms/AllItems.aspx Contains material/pointers to places where WLCG is mentioned/thanked/ Useful information In September just before the LHCC meeting: Learned of intention for ALICE and CMS to run at much higher rates: ALICE 12.5 MB/evt * 100 Hz (1,250 MB/s) 100 TB/day ATLAS 2.2 MB/evt * Hz ( MB/s) TB/day CMS 12 MB/evt * 150 Hz (1,800 MB/s) NB: large event size due to no zero suppression for HI 2010, and no trigger 150 TB/day LHCb No HI data taking Preparations for Heavy Ion run 12.5 MB/evt * 200 Hz (2,500 MB/s) 200 TB/day - Doubling of ALICE rate -(same integrated data volume as planned) - New large CMS request only arriving now - Doubling of ALICE rate -(same integrated data volume as planned) - New large CMS request only arriving now ALICE No complete export/processing during the heavy-ion run Export to be completed in the months after HI During the HI run: ~20% of the RAW rate (hence ~200MB/s) ~20% RAW reconstruction (but NO ESD recording) ATLAS Complete RAW export to all Tier 1 as in pp running Some processing: some ESD+DESD(+AOD) to tape and export NB: (ESD+DESD)/RAW is ~ 6 (Very large ESD and DESD) Assumption: at CERN express stream only the rest is done at the Tier1s CMS Complete RAW export to 1 site (FNAL) Full processing at CERN Reminder: Following data loss incident in May: Realised risk to ALICE data during the weeks following data taking when there would only be a single copy Mitigation was that we would expand ALICE disk pool to keep 2 copies at CERN until data had been exported to Tier 1s Had not planned this for CMS as expectation was that all HI data would be copied to FNAL Heavy Ion processing Castor pools were expanded for ALICE + CMS To ensure 2 full copies at CERN until data can be migrated to Tier 1s CMS: able since already had 2011 resources available Planned tests for technical stop Required upgrade of Castor and individual and combined experiment testing at full expected rates Test had to be brought forward urgently as the technical stop was changed Many people were at CHEP conference However, tests were successful, but only very limited combined testing was possible Heavy Ion preparations CERN IT Department CH-1211 Genve 23 Switzerlandt Internet Services DSS 7 Castor Tape subsystem > 5GB/s to tape this Sunday Data written to tape last month (1 k GiB = 1 TB = 1 tape) Heavy Ion run going on smoothly at impressive data rates: Oct 2009 to April 2010: 0.7 PB/month (23 1TB tapes /day) May 2010 to October: 1.9 PB/month (62 1TB tapes / day) November (Heavy Ion run) : ~ 3-4PB ? (120 1TB tapes / day) ALICE+CMS HI test HI Data taking CERN IT Department CH-1211 Genve 23 Switzerlandt Internet Services DSS 8 Update on tape writing Last Friday we exceeded 220 TB (tapes) in a single day which is the new HEP world record. Castor name-server DB problem during this day CERN IT Department CH-1211 Genve 23 Switzerlandt Internet Services DSS 9 Castor Disk Usage Disk Servers (Gbytes/s) last month Per experiment early Nov ALICE ATLAS CMS CERN IT Department CH-1211 Genve 23 Switzerlandt Internet Services DSS 10 Summary Tier 0 for HI System behaving very well, can accept and manage the very high data rates Wrote ~3.4 PB to tape in November Peak rates: 220 TB/day to tape; 5.9 GB/s data in pp running ~ 2PB/month Tape drives used very efficiently (this had been a major concern); because: Castor upgrade improved scheduling Large files are being written (up to 40 GB) Using only ~50 drives even at peak rates (worried that we would need many more); write speeds close to native drive speed (average of 80 MB/s) EXPERIMENT EXPERIENCES Pb+Pb at s=2.76TeV ALICE ALICE ALICE LHCC17/11/10 16 CMS HI Test As advised we ran a IO test with ALICE during a technical stop during the proton-proton run. By the second half of the test we got to 1GB/s out of the T0export buffer to tape This test did not include re- reconstruction Technical stop was moved with short notice to during CHEP, but generally successful Overview Board03/12/10 17 Tier-0 in Heavy Ions Prompt Reconstruction of Heavy Ion is currently able to keep up with data taking Good machine live time this week Should be able to finish the run with 100% prompt reconstruction Prompt Reco from Previous fill Completing Overview Board03/12/10 18 Reconstruction in Heavy Ion Reconstruction from earlier in the run Clean triggers Overview Board03/12/10 19 IO Castor IO holding up well IO tracks with CPU Averaging about 1GB/s to tape. Peaks while writing are 2GB/s as expected. Read rates from disk peak at ~5GB/s IO was optimized in CMSSW_3_8 and makes this Heavy Ion processing possible First Heavy Ion Collisions 20 First collisions on Sunday, Nov. 7, at 00:30 No stable beams Pixel detector was switched off, SCT in standby voltage Setting up of the machine started on November 1 First Heavy Ion Stable Beam Collisions 21 1 st stable beams (1 bunch colliding) Monday, Nov. 8, ~11:20 4 colliding bunches: Tuesday, 01:01 Now routinely with 121 colliding bunches 22 Submitted to Phys.Rev.Letters last Thursday Accepted on Friday, 10 hours later Fall re-processing finished Fall re-processing May re-processing Group production 40,000 jobs Xpress line 970,000 jobs bulk 900,000,000 events Aug 30 Release 16 tag deadline Sept 1-24 Release validation Sept 24 Conditions data deadline Extra week for Inner Detector Alignment Oct 8-14 Express Line re-processing Oct 29 Nov.21 Bulk re-processing Nov Crashed job (very few!) recovery All production Simulation Reconstruction Re-Processing group production User Analysis Month view 50,000 jobs Year view 20,000 jobs Year view User Analysis Fully occupying job slots in Tier 1 and Tier 2 ATLAS data movement Aim for an average rate of MB/s into Castor. Tier 0 event processing is keeping up, but just barely. It could get backed up if the LHC continues to deliver at a higher duty cycle Tested a backup plan to do prompt processing at T1 in order to alleviate T0 backlog Raw HI data is distributed like PP data to Tier1s Derived data, produced at T0 is delivered on request to special HI centres (BNL, Weizman and Cracow) ATLAS HI LHCb 2010 reprocessing Started 27/11, 45% done in 5 days 28 CPU years No. of jobs 2010 reprocessing efficiency % job success rate CPU/Wall clock efficiency Pile-up: LHCb running at higher pileup than design Larger event size and larger processing times are putting a strain on disk and CPU resources Use of Tier 2 envisaged for reconstruction in 2011 Currently does not limit physics reach, but will not be able to go much beyond current 2 kHz HLT rate with existing/planned resources LHCb concerns CREAM CE: More instances available; all experiments have tested; beginning to be more used Automated gathering installed capacity Tool available and in use Tier 0/1 OK for overall information; not complete for experiment dependent capacities Tier 2 reporting being verified (led by Tier 1s) Data management prototypes Progress presented in GDB Review in January Multi-user pilot jobs Software (glexec, SCAS, ARGUS) available and largely deployed Experiment framework reviews done and followed up Testing and use has been on hold during data taking Has not been a (reported) problem Milestones Whole node scheduling is: Allocating an entire machine to the experiment (not just a job slot) to allow the experiment to optimise use of all the cores (potentially with a mix of jobs) Responsibility for full utilization is then the experiments Informally requested by experiments in July workshop Eventually (Oct) agreed by computing coordinators that this is a real request from the experiments Pere Mato charged with setting up a wg to: Track commissioning of multi-core jobs in the experiments; mediate between experiments and sites on specific problems found, understand realistic requirements for eventual deployment Understand changes needed in job submission chain from user interface to batch system configurations Prepare roadmap for deployment of end-end submission Propose needed updates for accounting and monitoring Status: Requirement & Mandate agreed by 3 experiments Proposing wg members Process to be agreed in MB next week Timescale: WG to conclude by May 2011 Whole node scheduling Now Tier 1s and Tier 2s start to be fully occupied; as planned with reprocessing, analysis, and simulation loads Resource usage Tier 1 disk and CPU usage - Oct Tier 1 use - OctCPU use/pledgeDisk use/pledge ALICE ATLAS CMS LHCb Overall NB: Assumed effic factors 0.85 for CPU 0.70 for disk NB: Assumed effic factors 0.85 for CPU 0.70 for disk At the October RRB the message heard by the funding agencies was that there are plenty of resources Several delegates picked up on this and made it clear that it would be difficult to provide additional resources without first filling what we have But we are already at that point We are surely still in a ramp-up as the luminosity increases See later on planning for 2012 and later Resources We have had a very successful first year of data taking, processing, analysis However, we have not yet seen the situation where resource contention or over loading has been a problem With the potentially large amounts of data foreseen for 2011 etc this is likely to happen Must be prepared to react and address these issues Need to be pro-active in being ready for this: Is site monitoring providing all it can? Do sites get alarms for such conditions (over load )? What happens graceful failure or hang/timeout/? Urgently need to start activities to address this Already planning a study for the Tier 0 Some concerns End Nov was deadline for proposals for remote Tier 0 (extended from Oct 31) Have received 23!! Analysis ongoing Status of Tier 0 planning Intend to urgently review with experiments What middleware functions are essential to them now? What should effort be put into? E.g. how best/better to support pilot jobs; can we simplify information system; etc. cvmfs replacing software installation Data management Actions from demonstrator projects will be reviewed in January Clear that the various efforts on remote file access/location are coalescing Caching mechanisms needed What will help experiments most in the next 2 years Virtualisation CERN tested completed for the moment have a lxcloud pilot Hepix wg agreed mechanisms to allow trust of images Future developments Discussions ongoing on WLCG agreeing an MoU with EGI as a Virtual Research Community (perhaps as a seed for a HEP community) Agreed that wording can be found to work around the problem of blanket acceptance of policy Not really clear what the overall benefit to WLCG might be EGI Operations still very quiet Areas where EGI-Inspire is not providing funding, which are thus not covered Staged rollout of software does not work i.e. no sites are test-deploying any of the updates So far for WLCG there has been no real problem EGI Positive points: Provides effort for important support topics such as data management (dCache, FTS,..), etc. Agreed to have a direct channel for WLCG needs into EMI Less positive: Primary goal of EMI is harmonisation between gLite/ARC/UNICORE This is ~irrelevant for WLCG Release plan of EMI is of concern Back to the idea of yearly major releases WLCG needs a continuous support for releases as we have been doing The EMI-0, EMI-1 are starting to cause deviation of effort (building, certification) from what we need We now need to re-focus on the core middleware that we will need in the next few years EMI will need to adapt to support this EMI Very successful support of the first year of LHC data taking HI testing successful and HI run going well unprecedented data rates are being managed efficiently Resource use has grown in last months Tier 1 and 2 resources well used However must prepare for full loading and potential resource contention Concern over 2012 planning need to foresee additional resources if LHC runs in 2012 See later talk Various plans for future improvements ongoing Summary