No Slide Title

Between LHC and the Grid - Aspects of Operating the LHC Experiments

T. Camporesi, C.Clement, C. Garabatos Cuadrado, L. Malgeri, T. Pauly, R. Jacobsson 1

ExperimentsALICEATLASCMSLHCb

LHC Accelerator*

Physics Analysis*Already opened by Stefano Radaelli:http://indico.cern.ch/conferenceDisplay.py?confId=87720R. JacobssonCERNThe SceneExperiments have never in the past been so tightly connected to the accelerator

Whats so special about LHC?

Stored energy 2 x 360 MJ and fragile detectorsExperiment protection Operational communication and procedures Reliability requires direct communication interfaces

High interaction rate and large events sizeFast and reliable readout, storage and transfer to offline processingFast feedback from Data Quality checking

High intensity proton collider and sensitive detectorsMonitoring and understanding/analyzing/optimizing experimental conditionsLuminosity determination

Long-term detector stability and aging due to radiationAutomatic Calibrations

Many years of 24h operation with few people and non-expertsOperating the whole detector from one consoleUnderstandable high-level tools for diagnostics, alarms and data monitoringHomogeneity in the systemShifter training

These lectures will describe largely how we have addressed and solved these2R. JacobssonCERNProtecting Experiment and DataAt all levels we need tools to predict, prevent, and limit non-bb exposureFramework to combine information and understand correlations between background, beam characteristics and machine settings Dedicated archiving and analysis toolsWe need to look at the entire LHC machine

3Instantaneous damageBeam InterlockBackground.Trigger rates.....Poor data quality...Single event upsets...Accelerated aging.Long-term damage.....Online monitoringAccumulated doseandLuminosityBackgroundBeam characteristicsMachine settingsHalo/beam-gas/.....scraping....Beam incident

R. JacobssonCERNOperational Interfaces4

R. JacobssonCERNReadout Architecture5

R. JacobssonCERNRO Control/Event Management6Readout Supervisor LHC acceleratorBeam Phase and Intensity MonitorSubdetectorsHigh Level Trigger FarmL0 triggerConfigurationDBBookkeepingDBRS Event BankMulti Event RequestsBunch currentsFilling schemeClock/orbit,UTC, LHC InfoFilling SchemeLumi ScanHW and run parametersRun statisticsOnline LuminosityDetector statusL0 DecisionRO ElectronicsTrigger ThrottleTiming/Synch.ctrlFE ElectronicsTiming/Synch.ctrlTiming/Synch.ctrlReadout Supervisor is the intelligent readout controller The Orchestra DirectorCompletely based on Field Programmable Gate Logic devicesPool of RSs for autonomous stand-alone running of any subdetector

R. JacobssonCERNOnline Run Performance and Data Quality7Detector Performance, Readout Performance and Data QualityHistogram collected from all systemsMonitoring Farm spying on event streams at best effortAlso produces histograms from an online reconstruction at best effortHistogram analysis Automatic checks and alarmsHistogram inspected by Data Manager Shifter

CalibHistogram Handling(ECS)HISTODB

AutomaticHistogram AnalysisInteractive PresenterR. JacobssonCERNSketchy Offline Data Flow8Online35 kB/evt@2kHz = 70 MB/s or 2 GB file(~60 kevts) / 30sOfflineTier-0 / 1Tier-2Storage(CASTOR)ReconstructionSimulationStrippingAnalysisHLT Bulk StreamExpress Stream5 HzStorage(CASTOR)ReconstructionCalibrationAlignmentQCOffline Control Room

20h/file, 20 kB/evtBookkepingData and Production ManagementData Quality CheckingTest JobsQCRun InfoData QualityR. JacobssonCERNFill ProcedureTypical Physics fill cycle

Machine Development (MD) periodsSome scheduled days/weeks the LHC will be operated not to produce physics data (Stable Beam) but to study machine physics and improve performanceLHCb will not take data, but safety should be ensured. A shift crew will be needed still, but with lower activities.

9

R. JacobssonCERN

Calib

Readout control has two aspects:Control of data transferMEP Packing Destination assignment for event building and HLTLoad balancingPartitioning for parallel activitiesManagement of event types and associated destinations/processingPhysics triggersCalibration triggersLuminosity triggersNon-zero suppressed dataLuminosity scans (Vernier scan)

Driven and managed by the LHCb Timing and Fast Control SystemResponsible for distributing timing, trigger and synchronous and asynchronous information to entire readout systemFPGA based master: Readout SupervisorAlso performs rate control and generates all types of auto-triggers and calibration sequences

Centralized Readout Control10R. JacobssonCERNAccelerator vs Beam Modes11

R. JacobssonCERNBeam Modes for PhysicsIn LHCb we synthesize Machine/Beam modes into 9 Internal LHC StatesINJECTION, RAMP, PHYS_ADJUST, PHYSICS, ADJUST, DUMPEOF (End-Of-Fill), NO_BEAM, MD (Machine Development)12

VELO allowed INSoftware handshakes between LHC and experiments INJECTION, ADJUST and BEAM DUMPR. JacobssonCERNLHC/LHCb Operational ProcedureFill sequence for physics fill in LHCb language:

NO_BEAM(Injection Permit = FALSE, Any state of LHCb, Internal clock)

INJECTION(Injection Permit = TRUE, VELO out, External clock)

RAMP(Injection Permit = FALSE)

PHYS_ADJUST

PHYSICS (VELO in)

(ADJUST) (VELO out)

DUMP (VELO out, to be changed)

EOF (Internal Clock, Calibrations)

13Handshake for InjectionHandshake for Dump (Only when directly from PHYSICS)Handshake for AdjustR. JacobssonCERNMode Handshake (Ex. Injection)14PROBLEMWARNINGREADYPREPARESTANDBYIMMINENTVETOREADYOKLHC:NO_BEAMINJECTIONLHCb:STANDBYConfirmVETOExperiment Control SystemGET READYREADY General rule: Get ready in ~5 minutesConfirmRAMPR. JacobssonCERNNo Beam System Tests15Real-Time scheme to validate High Level Trigger, data flow and offline processing=Be ready to receive, process and analyze 7 million events in the first hour of collisions

LHCbMEP RequestsInjectorSimulated events

Replacing detector with injection of 108 accepted simulated events real-time in Online system at HLT rate (2 kHz)

Also allows testing new HLT versions with minimum bias events

R. JacobssonCERNCommissioning LHCb1616Two years of intense work 2006 2008 with the aim to:

Operate the detector AND people as a unit with common toolsBring all components (sub-detectors and service systems) to operational state.Define, implement and validate the tools and procedures needed to run the detector as a wholeOrganise the activities to reach the ready state in time

Understand and calibrate the detectorTest pulses, radioactive sourcesCosmicsLHC injection testsFirst days with beam

Operate with two shiftersOperating the whole detector from one consoleUnderstandable high-level tools for diagnostics, alarms and data monitoringHomogeneity in the systemShifter trainingOn-call Experts for all sub-systems and sub-detectors

Reach operational efficiencyStarting (