Geant4 physics validation on the GRID
-
Upload
george-lestaris -
Category
Science
-
view
278 -
download
2
Transcript of Geant4 physics validation on the GRID
George Lestaris supervisors: W. Pokorski & A. Ribon
Geant4 physics validation on the GRID
SFT meeting - 28/04/2014
Geant4 physics validation on the GRID
Introduction to the problem
• Geant4 is a simulation framework
• Physics validation is form of regression testing but:
• simulation data can be compared with respect to real (reference) data (from papers, test beams, etc)
• Monte Carlo simulation does contain by its nature the concept of randomness
• need to run many simulation events to get data that could be statistically close to previous versions
2
Geant4 physics validation on the GRID
Previous work• Solution tailored for Simplified calorimeter
• a Geant4 application that simulates the showers that are produced inside simplified geometries of LHC calorimeters
• DIANE used to run jobs on GRID resources
• not very flexible:
• unable to easily restart single jobs
• scheduling among different GRID sites was round-robin
• not maintained/supported anymore
3
Geant4 physics validation on the GRID
Previous work
• Process required manual intervention in different stages..
• Error-prone actions
• Minor problems
• Slow access to stored data: mis-index MySQL database
• Slow plotting: a new ROOT session per graph
4
Geant4 physics validation on the GRID
Previous work
5
Worker
Step 1Build and store
binaries to CVMFS
Step 2Setup DIANE Master
Step 3Submit GANGA Jobs
Step 4Run analysis of results
CVMFS
MySQL DB
GANGA Iterface
WLCG
LxBatch
WorkerWorkerWorker...
DIANE Master
Web application
AFS
Geant4 physics validation on the GRID
Motivation• Extensible system to
support multiple tests
• Use of a better supported WMS
• Automations to reduce the room for user errors during jobs submission and running
• Efficient storage and plotting
6
Creation of DIANE master
Preparation of AFS directory
Submission of pilot jobs
Results merging, analysis and
storage
...
Old
Submission of production
...
New
Results become available in the web application
Geant4 physics validation on the GRID
g4.Val
7
Geant4 physics validation on the GRID
Architecture• Component-based architecture
• Core component contains essential classes and modules, referenced by all other components
8
Production side: • Production: Submission
of jobs. The job wrapper that runs the tests, collects stores and merges results.
• Client: Communication with the Web API.
Presentation side: • DB: Abstract interface to
the database
• Analysis: Runs lightweight analysis tasks (e.g.: plotting) over the results.
• Web: Hosts the UI and the results submission API.
Geant4 physics validation on the GRID
Architecture
9
Core
API Client
Production
Web
DB
Analysisspeaks API
DIRACserver EOS MySQLRedis
queue
Geant4 physics validation on the GRID
User roles (actors)
Test manager: person that imports a test in the system
• not necessarily the developer of the test but someone who understands how to run it
Production manager: submits the productions (sets of jobs) for tests
End user: that looks into the final results (web UI) by running on-demand analysis (plotting) tasks
10
Geant4 physics validation on the GRID
End user
11
http://g4-val.cern.ch/
Tests list Analysis plugins
Geant4 physics validation on the GRID12
Filters
Tabs
Results UI
Geant4 physics validation on the GRID
Results UI components
13
Version selection
Filters can be lists
of values
Geant4 physics validation on the GRID
Results UI components
14
Bookmarkable pages
Different analysis (plotting) plugins
Geant4 physics validation on the GRID
Production manager
• Submission with a recipe
• predefined recipe that submits a production by changing the Geant4 version
• Command line tool to interactively create custom production
• Python API
• if need custom input sandbox, environment, dependancies
15
Geant4 physics validation on the GRID
Recipe submission
$> source /cvmfs/geant4.cern.ch/grid/etc/login.sh# prompt to type the certificate password$> g4val-prod-recipe -t sc -v 10.0.ref03 -r test# some settings.........[g4val.production.submitter.dirac INFO] 2014-04-25 16:38:43 CEST - Submitted job with reference: Id = 64849[g4val.production.submitter.dirac INFO] 2014-04-25 16:38:43 CEST - 8 jobs were successfully submitted!
16
Test code Version Recipe
Geant4 physics validation on the GRID
DIRAC
The DIRAC (Distributed Infrastructure with Remote Agent Control) project is a complete Grid solution for a community of users needing access to distributed computing resources. DIRAC forms a layer between a particular community and various compute resources to allow optimized, transparent and reliable usage.
17
Geant4 physics validation on the GRID
DIRAC (in English)
• Developed for the LHCb experiment initially
• Today a collaboration with many users around the world
• Geant4 uses a small subset of its functionality
• Pilot job approach as well
• Scheduling is done opportunistically, i.e.: the site that runs the most pilot jobs runs the most DIRAC jobs
• Able to restart/reset/reschedule jobs
• Web UI to list, filter, monitor and manage jobs
18
Geant4 physics validation on the GRID
Test manager
Needs to define the test model
• what are the variables that can uniquely identify a test run?
• what are the variables, extracted from the test results, that are interesting to present?
19
Geant4 physics validation on the GRID
Test modelAttributes and parameters: variables that characterise a test instance (execution)
Test run: assignment of values to all the test parameters
• A test run is unique
• Parameters are indexing list of runs
Observables: variables that describe results
• Can be tuples, list of numbers, or pointers to (ROOT) files
• A test observable is also having its attributes and parameters
20
Geant4 physics validation on the GRID
Test model
21
str codestr namestr description
Teststr codestr nameint typemixed default_valuebool allow_null
Attribute
bool use_referenceParameterstr code
str namestr description
Observable*
1
*
* 1
1
Geant4 physics validation on the GRID
Test runningA lot of extensions points available to override system default behaviours for a test:
• Test running: production hooks
• add dependencies (libraries from CernVM-FS)
• environment (variables)
• macro file generation (template)
• macro file is used as an input to a Geant4 application
22
Geant4 physics validation on the GRID
Results storage
• Run splitting and merging:
• big (long) runs can be split to smaller jobs (e.g.: reducing number of events)
• Results storage: production hooks
• raw results harvesting if necessary
• format of registered results (observables)
23
Geant4 physics validation on the GRID
Results presentation
On-demand analysis
• analysis plugins are independent of a test, i.e.: can be reused
• mapping between what observables can be plotted with which analysis plugin
24
Geant4 physics validation on the GRID
Technical details (backup slides)
25
Geant4 physics validation on the GRID
Production: Job splitting and submission
• Large test runs can be split to smaller jobs with reduced number of events
• Each job appends its results to the run results
• List of jobs is submitted with the submitter interface
• Current implementation with DIRAC, but can be any
• Job scheduling is done by the WMS
26
Geant4 physics validation on the GRID
Production: Job wrapper
g4.Val gets involved again when the job arrives to a worker node (WN)
• Job wrapper is stored in CernVM-FS
• Job is a simple JSON file with its input sandbox
27
Geant4 physics validation on the GRID
Production: Job wrapper1. Input sandbox checking
2. Macro file generation given the run parameters
3. Dependency resolution per test for WNs platform
4. Starts the Geant4 process
5. Transfers results to EOS when done
6. Appends job results to the run results if run was split
7. Registers the results to the database with the API client component
28
Geant4 physics validation on the GRID
Database component
Database schema is generated upon test import
• table for storing the runs
• one table for each observable
• attributes are fields and parameters make up the table indexes
29
Geant4 physics validation on the GRID
Web component technologies
30
Geant4 physics validation on the GRID
Analysis component• Web component submits
analysis task to redis queue
• Analysis workers are processes that execute these tasks
• Disposable in case of crashes
• Analysis load balancer starts new workers if the queue has many waiting jobs
31
AJAXServer
Web interface
User Browser
HTTP
Request
Request
Request
...
Workers
Database
Geant4 physics validation on the GRID
Examples (backup slides)
32
Geant4 physics validation on the GRID
Simplified calorimeter• Big runs, split to many jobs
• Raw results as ROOT files (merged with TFileMerger)
• Registered results are value-error tuples for the six observables
• Using many runs in one graph as beam energy is the X-axis parameter
• Plotted with HighCharts JavaScript library (client side)
• or ROOT TGraphError (server side)
33
Geant4 physics validation on the GRID
Simplified calorimeter
34
http://bit.ly/1lZQef0
Geant4 physics validation on the GRID
Geant4 test test30
• Quick runs, no need of splitting
• One pilot job can run tens of test30 jobs
• Raw results are ROOT files
• Registered results are these ROOT files URLs (EOS supports natively XRootD)
• Each TObject is an observable
• Plotted with ROOT TObject superimpose plugin
35
Geant4 physics validation on the GRID
Geant4 test test30
36
http://bit.ly/1lXtSLc
Geant4 physics validation on the GRID
Example of test’s modelfrom g4val.model import Test, Attribute, Parametertest = Test( code="sc", name="Simplified calorimeter", description="...")# Parameterstest.add_attribute(Parameter( code="physics_list", type=Attribute.STRING, type_size=100, name="Physics list", use_reference=False))test.add_attribute(Parameter( code="calorimeter", type=Attribute.STRING, type_size=100, name="Calorimeter"))test.add_attribute(Parameter( code="beam_particle", type=Attribute.STRING, type_size=5, name="Beam projectile", default_value="pi-"))test.add_attribute(Parameter( code="beam_energy", type=Attribute.FLOAT, name="Beam energy"))# Attributestest.add_attribute(Attribute( code="events_amt", type=Attribute.SHORT, name="Number of events", default_value=5000))
37
Geant4 physics validation on the GRID
Example of macro generation hookdef get_job_macro(job): return """#verbose1#events%(events_amt)d#particle%(beam_particle)s#material%(material)s#energy(MeV)%(beam_energy)s#generator%(generator)s#run#exit""" % job
38
• From test test30
• Test parameters are:
• Number of events
• Beam particle
• Beam energy
• Material
• and Generator
• job is inheriting Python dict
Geant4 physics validation on the GRID
Conclusion
g4.Val is a flexible, extensible system for Geant4 physics validation in our group
• Introduces a Geant4 test formalisation
• Uses DIRAC for job submission and running on GRID and potentially other batch resources
• Efficient storage of results
• On-demand analysis tasks through an highly interactive intuitive web interface
39