Xrootd Monitoring andControl

Harsh AroraCERN

Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module

Collecting Xrootd Parameters

MonalisaService

ExternalStorage

XrootdSystem/

Network Info

Sends monitoringdata

Control XrootdServices

Proposed Architecture to Monitoring & Control the Xrootd in ALICE

Long HistoryDB

MonALISA

Network Monitoring

ApMon

XROOTDSERVER

SystemMonitoring

MonaLisaXrootd Repository

Aggregated Data

Alerts

Actions

Iosif Legrand June 2012

ControlModule

Control /

Update

MonALISA

Network Monitoring

ApMon

XROOTDSERVER

SystemMonitoring

ControlModule

MonALISA

Network Monitoring

ApMon

XROOTDSERVER

SystemMonitoring

ControlModule

Run a MonALISA service on each host running an XROOTD server ( a new dedicated group )

Control the xrootd server Start / Stop / Update Dynamically Change conf parameters

Collect Monitoring information from the xrootd servers using the new & improved monitoring functionality from xrootd

Perform full network measurements and tests (RTT , available bandwidth, topology )

Monitor the storage system used by xrootd servers Create a dedicated MonALISA repository

5Iosif Legrand June 2012

Proposed Architecture to Monitoring & Control the Xrootd in ALICE

Active Available Bandwidth measurements between all the ALICE grid sites (2)

Iosif Legrand October 20106

6 Iosif Legrand June 2012

ALICE measurements between all VO BoxesImportant to have similar Data for Xrootd servers

7

Advantages of Using a dedicated MonALISA group to monitor and control all the XROOTD servers

Easy to maintain and update a critical service for Offline computing

Significantly improve the monitoring information and will

help to better understand the way storage system are used in the ALICE Computing model (Distributed Storage for Data)

Control the xrootd servers and can dynamically configure systems based on how they are used.

Monitor the true network connectivity Monitor the real storage used by xrootd servers Monitor the connection and transfer per client / job Can really help in developing optimization procedures for

complex data flows

Iosif Legrand June 2012

% of CPU Used by Monalisa

ParametersCollection and Verification

% of time spent by CPU in User Mode

Monalisa Host Monitoring Reported by Xrootd

ParametersOpen_files Srv_rd_mbytes

Srv_wr_files

Parameters(Cont.)Srv_rd_files Srv_wr_mbytes

Space_free

Average Load over last 15 min.Load15

Ambiguous Parameters

Connected Clients Number of Jobs Requiring a Thread

Dedicated MLRepository

Detailed Monitoring per Client

Monitoring per Client Patch to xrootd-3.1.0

Client Monitoring Example

Num

ber

of

Byte

s

Additional Information provided by monitoring each client from the xrootd

server Global Map of Data Transfers Debug individual clients Understand better the effect of long RTT

Preparing Correlations Plots

Bad xrootd Server Detection High CPU Usr with Low Disk Throughput -ve correlation b/w Traffic and Disk

IOUtil(%) High Disk IOUtil(%) with Less Read Write

Rate – Probably Damaged Disk High Load Value with Less number of

clients

Monalisa on Each Server

Local CPU Monitoring Info Understanding Network Topology Traffic Information Monitoring directly the storage system Control

Future Plan

Control via Monalisa

Start/Stop xrootd server

Upgrade

Problems

Latest version of xrootd-3.2.1 Incompatible with xrootd-ftsofs-1.1.0

Thank You