Introduction to COMA (and Metadata)

Introduction to COMA(and Metadata)

Elizabeth Gallas

Oxford

ATLAS Software TutorialMarch 11-15, 2013

March 2013 E.Gallas - COMA 2

Outline What is “Metadata” ?

Challenges in ATLAS

Overview of User Applications Subsystem specific applications (many) Dedicated Metadata repositories (3)

AMI COMA (main topic today) TAGs

Show COMA Portal (grid certificate required on your browser !) COMA Period Menu and Period Reports Trigger metadata in COMA

And Trigger-related content in COMA Reports Other COMA Report highlights

Conclusions: Caveats, Status, Plans


What is Metadata ?Metadata definition: Concisely:

“data about data” More precisely:

“data used to describe the context, content or structure of data”

Structural or DescriptiveMetadata: used extensively in ATLAS …In fact: No process doesn’t use metadata

“Descriptive” examples: Dataset name, Run Number,

Channel number in some detector, TWiki Name, Trigger Names, dates/times, DQ Defect, …

“Structural” examples: Number of runs or events or files,

data volume, structure of compound objects, …

Usage examples: Upstream: data taking with the

correct calibrations … Downstream: user finding Events

of interest … or Luminosity for an event sample

Metadata challenges: Size/Scope of ATLAS data …

Volume/Diversity of metadata Data/metadata: have grown

organically as the experiment evolved

Following evolution in Run1 Trying to anticipate changes for Run2

Offer a coherent / integrated view to physicists while Optimizing access / consonance for processing and analysis


Subsystem specific: driven by subsystem specific needs Trigger: wide variety of tools and interfaces Geometry DB: Detector Description Browser Conditions DB:

RunQuery (in-depth Run info from Conditions DB) ATLAS WEB DQ COOL Tag Browser

Lumi Data Summary Reports (Luminosity, Beam) GRLs (Good Run List xml)

And the Luminosity calculator Beam Spot Summary GANGA and PAthena Panda / monitor DQ2 Client ... (not a complete list !)

Dedicated Metadata Catalogs (3) AMI – Datasets, processing … other metadata

And the AMI Suite of services COMA – Run/LB level Conditions and configuration metadata

Plus Conditions DB management metadata TAGs (and TAG Catalog) – event level metadata

iELSSI and Suite of TAG Services

ATLAS User Application Overview

Fundamental areas for every analysis !See other talks on these topics.


COMA IntroductionThe COMA Project: TWiki: ConditionsMetadata Originally: built to support other systems. Evolved: into a standalone system with its own interfaces. Components:

Relational Database (Oracle) Copied, refined, reduced, derived content from many sources Unique content (not found elsewhere)

Data Periods, Derived/Aggregated quantities Unique Interfaces (Reports and Browsers)

COMA unique content and structure Ability to aggregate quantities over user-defined temporal periods Integration with AMI, TAG information and services General philosophy: minimize typing … navigate w/ links btwn reports

Beyond COMA: COMA is part of general effort to consolidate/relate ATLAS Metadata

Strong ties with AMI and TAG DB Any COMA information can be made available via “pyAMI”

COMA data/links now found in many ATLAS systems: AMI, TAGs, DataQuality, RunQuery, Muon alignment, Conditions DB tools Many links from ATLAS TWiki, physics pages, and personal pages


COMA Interfaces Portalhttps://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html

Most popular

COMA Period

Other Reports

(shade: grey) … operational … little current/active development

Conditions DB

expert tool


COMA: ATLAS Data Periods … + aggregating new content

A Data Period is a set of ATLAS Runs grouped for a purpose Defined by Data Preparation Coordinators Used in ATLAS data processing, assessment, and selection … Each Period uniquely defined with a combination of

Project name (i.e. ‘data10_7TeV’) Period name (i.e. ‘C1’, ‘C2’, ‘C’, ‘AllYear’ …)

Before 2011, Data Periods were Described on TWiki page

https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods Stored in a file based system

Edited by hand by Data Prep Coordination (experts) Structure evolved over 2010 with experience

This experience valuable to decide/define long term solution

In 2011: Data Periods moved into COMA Coordination/Effort: Data Prep, AMI, COMA experts This made all aspects of Period definitions available programmatically

via pyAMI Since then, COMA content has grown in many areas

Allows for more details reports and information to other systems Enables aggregation of LB-wise information by Run, … Period.

Painful to maintain,AFS dependent,

Error prone

Simple to enter, check integrity,

more robust, available


https://atlas-tagservices.cern.ch/RBR/rBR_Period_Report.php

Period Menu Purpose: Shows an overview of all Data Prep

defined Periods Offer links: Period Description Reports

containing verbal description and aggregated info about their Runs.

Choose the Period of interest: By Year

e.g. all ‘2011’ Or for ‘all years’

By Project e.g. ‘data12_8TeV’

By Beam Energy or Type e.g. ‘7TeV’

By specific Period or Group Click on the project and then

Period of interest

General feature of COMA Reports

“highlighted” link opens expanding

sections

Help, DocLinks


highlight links:

show / hide

period members

Members of

data12_8TeV.A

are A1-A8

Links: to COMA,RunQuery,

AMI Container production

Header:

Input criteriaLinks in Table column headers:

Short description of column

Note: some columns

removed using the

“customize report”

feature (not shown)

Hover on link:

Indicates what

will happen


Links: COMA Run, Coma Run Trigger,

RunQuery, and Lumi Data Summary

COMA Magnet States.

LB-wise transitio

ns

in COMA Run Report

COMA Multi-Run Report for Period 8TeV.C2

Purpose: quick overview of any set of Runs

COMA SMK and CRTrig Reports: Show all

Triggers in the Menu. CRTrig Report adds

Lumi per trigger and which trigs are “active”


“Event”: detector output during a single particle bunch crossing“Lots”: LHC max particle bunch crossing rate is 31.6 MHz“Fewer”: a few hundred events per second“Trigger” is a multi-component selection filter for events:

ATLAS detector hardware/electronics Many subsystems … TDAQ

ATLAS software: HLT Release Mostly C++ algorithms collected in a specific ATLAS

Software Release executed by the HLT (2nd,3rd trigger levels)

Trigger Menu: defines ~500 to 1000 Triggers Every distinct Menu is assigned a unique integer ID

SMK: Super Master Key Configurable input to the Trigger hardware and software

Specifies what logic or algorithms to execute, including configurable parameters (eg: thresholds)

Assigns each trigger to one/more output Streams Menu (SMK) is FIXED during each Run (not incl. prescales) Each trigger: 3 levels of pass OR fail

Each Event either passes or fails each Trigger Prescales: Blind filter applied by TDAQ when above Trigger

logic does not sufficiently reduce event output rate Prescales can change during a Run (on LB boundary) Integer identifiers are assigned to sets of prescales

Level 1 and HLT Prescale Keys

“Metadata”viewof the Trigger

“Trigger”

“Fewer” but more interesting Events

“Lots” of “Events”

Level 1

HLT: L2

HLT: EF

HLT PT

Event is recorded for offline physics analysis if it passes at least one trigger (and its prescale)


Trigger Configuration Metadata in COMA: SMK Trigger Chains: EF chain, L2 Chain, L1 Item

Names, Versions, Bit Assignments, Streams, ReRun LVL1, HLT Prescale Keys:

EF, L2, L1 prescales EF, L2 Passthrough

Details behind Trigger Configuration and what is stored event-wise: need tools from the Trigger Experts Understanding trigger execution and info storage

Algorithms, cuts, multiplicities, bunch groups Dead-time veto, BCID / Train / Lumi dependence Trigger objects related to trigger decisions HLT algorithm Error codes Trigger EDM and the Trigger Decision Tool How to work with Chain Groups (Trigger ‘OR’s)

See the trigger related talks in Software Tutorials: https://indico.cern.ch/conferenceDisplay.py?confId=212225

Trigger Metadata: just the tip of the iceberg

COMA: Stores this metadata.Combines it w / Period,Run,Lumi

data to provide unique reports(overview usage over many Runs)


Every Run has a Trigger Menu: Every analysis is based on one/more triggers

Critical to understand it … so many tools … different levels of detail

From Trigger experts (see Trigger related tutorials) http://atlas-runquery.cern.ch (RunQuery portal) https://atlas-trigconf.cern.ch/ (TrigConf portal) Trigger Tool (java based) AtlCoolTrigger.py (command line based)

From COMA (show collected/derived trigger metadata) Run Report: evolution of states of system over a Run (in LB ranges)

PrescaleKeys, Fills, StableBeam, Ready, Lumi, Magnets(w/links) RunTrig Report: Run-wise Trigger chain “activity” flags

1. Prescale 2. Passthrough 3. RerunClearly shows which configured triggers are NOT active in a Run

Many reports display: “Aggregate prescale” (L1*L2*EF PS) by Trigger

or the Range of AggPS by Trigger, Run, Period, … New content: prescale corrected Lumi by trigger

By LB, Run, SMK, Period (or other temporal criteria)


COMA Single-Run Report

RunQuery Links

TrigConf Links

COMA Links


Show all Triggers in Menu for a Run Shows which are “active”

Via Prescale, PT, Rerun … and which are disabled

Show range of AggPS, and PS Corrected LumiCaveats: Lumi in COMA is only during Stable Beam Ready conditions. PS Corrected Lumi by trigger do NOT include deadtime or DQ !!! Always an upper limit !

COMA Run Trig Report

March 2013 E.Gallas - COMA 16Jan 2013 E.Gallas- Metadata 16

2.Configuration Section:Shows where this element is configured:

Super Master Key(s) Project (Summary)

3. Period Evolution: Shows chain/item bit, version evolutionfor EF_g20_loose chains during Period Runs

4. Activation Section:Shows Runs where this chain is ”active”

Via prescale Via pass through Via rerun

EF_g20_loose

1.

2.

4.

3.

March 2013 E.Gallas - COMA 17Jan 2013 E.Gallas- Metadata 17

COMA Chain Wildcard Reports

L1_2EM*_MU*over all periods

EF_*ZEE*

The NEW version of these reports add integrated prescale corrected Lumi, PS ranges, and Run/Date ranges


Some Caveats Metadata is derived from the data

But occasionally does not contain all the details Generally noted in the documentation when known

One important caveat for users! Luminosity in COMA is During Stable Beam and ATLAS Ready flag condition

When that Lumi is tagged by experts in COOL Some runs are special, intended only for experts This is why a few runs are missing Luminosity in COMA

Prescale Corrected Lumi by trigger Does NOT include dead time, Data quality, … !!!

Always an upper limit !

COMA Lumi is intended to guide analysisBut is not the final luminosity with all corrections

Luminosity for any specific analysis: Must, in the end, use official LumiCalc tools !


Conclusions Understanding ATLAS Metadata is important:

To use subsystem specific tools and Understand how that information relates to the big picture

3 systems dedicated to collecting ATLAS Metadata (AMI, COMA, TAGs) They contain:

information found in other systems Plus unique data not found (or not easily found) elsewhere There-by: unique interfaces and tools

COMA contains Run/LB level Conditions/Configuration Metadata Quick overview of Periods, Runs, active streams, beam conditions, … Overview of configuration / activation of specific triggers

How they evolved over Run 1 … when was my trigger first prescaled ? Offer Aggregation of various quantities across Periods, Runs

COMA Plans: Adding event counts: Stream, Trigger Enhance aspects of MC metadata (LS1) Enhance: Content, Functionality, Usability

Comments/Questions on Metadata: [email protected]


General Comments: tutorials, applications, tools ATLAS: a complex, multi-purpose experiment General guidelines for analysis exist … but only take you so far

Every group, analysis, and user is different. Having a broad understanding of the LATEST tools available will

facilitate your analysis and deepen your understanding of data Software tutorials (~ one week-long, ~ every 3 months):

Overview of ATLAS Computing for users Geared toward new users But also useful for others

Advertise a parade of interfaces and tools Note: They are constantly evolving

So even experienced users look at latest tutorials Keep up to speed with latest trends, new tools Consider alternate tools to improve their analysis

One week is a limited time to take it all in Many details in the slides should be explored independently

Most important !!! Don’t be afraid to explore, compare, question, comment, rate Please provide feedback (to the appropriate e-groups) !!!

Introduction to COMA (and Metadata)

Documents

Transcript of Introduction to COMA (and Metadata)