Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene...

Recent improvements in HLRmon, an accounting portal suitable for

national Grids

Enrico Fattibene (speaker), Andrea Cristofori,Luciano Gaido, Paolo Veronesi

INFN - Italyhlrmon<at>lists.infn.it

WORLDCOMP GCA’11 – Las Vegas, July 18th, 2011

Outline

• gLite Grid infrastructure

• Grid accounting issues

• Accounting layout in the Italian Grid Infrastructure (IGI)

• HLRmon overview

• HLRmon main report section

• Storage accounting prototype

• Outlook and conclusions

2HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011

gLite Grid infrastructure

HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011 3

CECE

WMSWMS

BDIIBDII

Accounting ServiceAccounting Service

Grid site A

SESE

CECE

Grid site B

SESE

• The Computing Element (CE) is the service representing a computing resource• The Storage Element (SE) is the storage unit at Grid site level• CEs and SEs send information about the available resources and their status to the Berkley

Database Information Index (BDII)• Grid user can submit jobs:

through the Workload Management System (WMS) that retrieves information on the available resources from the BDII and sends the job to the appropriate CE

directly to the CE

• The accounting service is responsible for the collection of computing and storage information

Grid user

Grid accounting issues

• Resource usage accounting in Grid is challenging due to the distributed nature of the infrastructure and

the heterogeneous nature of the resources accessed necessary for its stakeholders, such as site administrators,

Virtual Organization (VO) managers, Grid managers, etc.

• Users need to access accounting information at different levels of aggregation an exhaustive and easy to use graphical interface is

fundamental many views are needed for different stakeholders


Accounting layout in IGI


• Distributed Grid Accounting System (DGAS) sensors collect accounting information at Grid site level

• Site data are sent to site or multisite Home Location Registers (HLRs)• A top-level HLR receives Usage Records (URs) from all the Grid HLRs • HLRmon retrieves data from the top level HLR and presents them through a web interface.

Aggregated data are also sent to the European Grid Infrastructure (EGI) Accounting Portal


HLRmon introduction

• Developed and funded in the framework of the INFN Grid project

• Deployed in the Italian production Grid since 2007• Choosen as the accounting portal by gLite-based

National Grid Infrastructures (NGIs) and projects IGI (Italy) HellasGrid (Greece) D-Grid (Germany) W-eNMR project Cybersar project


HLRmon structure• HLRmon collector retrieves URs from the desired DGAS HLR server

• Data are aggregated, stored in a local MySQL database and presented through a php-based web portal charts highly customizable by the users tabular reports available also in xls format

• A x.509 personal certificate is needed to access the portal

• The web interface includes: a main report section providing usage information aggregated in many

different ways a section reporting on resource usage by VOs, aggregated by the

corresponding scientific disciplines a section dedicated to the Worldwide LHC Computing Grid (WLCG)

sites and VOs (alice, atlas, cms, lhcb) a restricted section providing accounting per-user information, visible

only by registered and authorized users

8

Main section• Different sets of charts

Number of executed jobs CPU Time (CPT) and Wall Clock Time (WCT) row and normalized, expressed in

time (days) o the normalization is based on the KiloSpecInt2000 benchmark

Job efficiency in terms of (CPT / WCT) ratio

• Different keys for data aggregation and selection Grid site VO, group and role

o groups and roles can be granted to perform specific tasks within a VO Certification Authority (CA) and Registration Authority (RA)

o that released the x.509 personal certificate used by VO member to authenticate him/herself to the Grid

Job type (Grid or local)

HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011

Main section overview


10

Main section charts

Normalized CPTexpressed in KSI2K-dayper VO and per VO per day

Job efficiency (CPT/WCT) per site per day and per VO per day

Normalized CPTexpressed in KSI2K-dayper site and per site per day


VO / groups / roles menuVOComplete list of VOs with at least 1 job into the HLRmon DB

VO groupsFor each VO, the VO groups with at least 1 job are listed

VO rolesFor each VO group, the VO roles with at least 1 job are listed


VO groups / roles charts

Job number per VO groupJob number per VO group per day

Job number per VO roleJob number per VO role per day


CA / RA menu

CAThe CA list depends on the HLRmon server configuration.

RAThe IGI HLRmon server is configured to aggregate data per RA. For the INFN CA, the RA information is extracted from the L= field of the DN.


CA / RA charts

Job number per CAJob number per CA per day

Job number per RAJob number per RA per day


Storage accounting approach• Storage accounting system integrated in DGAS under development• Non standard schema has been used to generate storage URs

no standard schema exists we join the Open Grid Forum (OGF) working group on the storage UR

definition

• The information on the used space is collected from the BDII once a day specific sensors can be developed to retrieve more accurate data

o making the system independent from the BDII

• Storage URs are stored on the HLR server as the computing URs• HLRmon is able to retrieve storage information directly from the

desired HLR


Storage report

• HLRmon shows the daily disk consumption in terms of used space

• Data are aggregated per site, SE and Storage Area (SA)• Used space will be compared to the total available space (at

SA, SE and site level)• The plotting of these data allowed to detect wrong values

published in the BDII sites have been notified in order to correct the values


Storage charts

Used space per siteUsed space per site per day

Used space per SAUsed per SA per day


Outlook

• The storage accounting prototype will evolve in order to adopt a standard UR when available we will follow this discussion into the OGF UR working group

• HLRmon will be made independent from the underlying accounting system accounting services adopted in EGI are evolving to transport their

data through ActiveMQ messaging system HLRmon data collector will be improved to retrieve accounting

data from a common interface, where EGI accounting services will publish their URs


Conclusions• HLRmon

is developed to meet requirements from different kind of users is successfully deployed as accounting portal for IGI and other Grid

projects/NGIs provides a complete section on computing accounting reports offers a fine granularity of data aggregation is close to provide a complete storage accounting section can be adopted to visualize data taken from other accounting systems

• Documents, instance list, installation guide, video tour, etc.:http://www.italiangrid.org/grid_operations/tools/accounting_portal/HLRmon


http://www.italiangrid.org/grid_operations/tools/accounting_portal/HLRmon

Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene...

Documents

Transcript of Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene...