Atlas Grid Status - part 1

Atlas Grid Status - part 1

Jennifer SchopfJennifer Schopf

ANLANL

U.S. ATLAS Physics and Computing Advisory Panel ReviewU.S. ATLAS Physics and Computing Advisory Panel Review

Argonne National LaboratoryArgonne National LaboratoryOct 30, 2001Oct 30, 2001

Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 2

Globus/Atlas Interactions

GRAPPA/Gardner

Grid DataAccess/Malon

Magda/Wenaus

GridView and other MonitoringDe, Schopf, Yu

Condor (G)

GRAM

GSI

MDS/GIIS/GRIS

GridFTP

Replica Cat

Replica MgrPacMan/YoussefPackaging


Globus ToolkitTM

Core protocols and servicesCore protocols and services Grid Security Infrastructure

Grid Resource Access & Management

MDS information service & monitoring

GridFTP data access & transfer

Data Grid technologiesData Grid technologies Replica catalog, replica management service

Reliable file transfer

Defacto-standard for Grid projects:Defacto-standard for Grid projects: GriPhyN, PPDG, NEES, EU DataGrid, ESG, Fusion Collaboratory, DISCOM,

NASA IPG, NSF TeraGrid, DOE Science Grid, EU DataGrid, UK Grid Center, U.S.

GRIDS Center, Access Grid, GridPort, MPICH-G2, Condor-G, GrADS, and others


Globus Status

New releaseNew release In alpha4 now, beta before SC ‘01 (Nov 15), 4Q01 release

New packaging: enables modular binary and source distributions

GRAM 1.5 (job submission): enhanced robustness

MDS-2.1 (information service): security, better performance, etc.

GridFTP- secure large file transfer

Replica Mgt- Data management, catalogs for replicas

Continuing workContinuing work Community Authorization Services

Reliable file transfer

Java, other “Commodity Grid” toolkits


ATLAS GriPhyN/iVDGL

GriPhyN funded Fall 2000, $11.9M/5 yearsGriPhyN funded Fall 2000, $11.9M/5 years

iVDGL funded Fall 2001, 5 years, $13.65M/5 yeariVDGL funded Fall 2001, 5 years, $13.65M/5 year

Both involve ATLAS, CMS, Ligo and SDSSBoth involve ATLAS, CMS, Ligo and SDSS

ATLAS support: 2.5 FTE IU, 1.5 FTE BU, $331K/5yr IU HWATLAS support: 2.5 FTE IU, 1.5 FTE BU, $331K/5yr IU HW

R. Gardner is the ATLAS lead, J. Schopf is the CS liaisonR. Gardner is the ATLAS lead, J. Schopf is the CS liaison

GriPhyN proposal emphasizes virtual data requirements, GriPhyN proposal emphasizes virtual data requirements,

collaboration between experimentscollaboration between experiments

iVDGL proposal emphasizes testbed and infrastructure iVDGL proposal emphasizes testbed and infrastructure

issues across experimentissues across experiment


GriPhyN

Principal ATLAS GriPhyN/iVDGL deliverables:Principal ATLAS GriPhyN/iVDGL deliverables: 2001: Testbed with GriPhyN VDT 1.0, packaged with PacMan

2002: Serving DC1 data

2003: Dataset re-creation/Data signature

Additional efforts:Additional efforts: Monitoring: Dantong Yu, BNL, and J. Schopf,ANL, joint with PPDG

GRAPPA: Rob Gardner and Randy Bramley, IU


GriPhyN Testbed Issues

GriPhyN is defining VDT 1.oGriPhyN is defining VDT 1.o SW install for GriPhyN/PPDG, compatible with EDG as well

Globus 2.0 beta when it’s released (10/30) GSI, GridFTP, MDS, repl, cat stuff, etc (Gram 1.5)

GDMP 2.0 (supports flat files)

Condor 6.3.1 (also Condor-G, Dagman)

Extra tools for ATLASExtra tools for ATLAS objectivity 6.1

Magda

Still need to resolve CA issue between EDG and US test Still need to resolve CA issue between EDG and US test

sitessites


GriPhyN ATLAS Goal 1Serving DC1 Data (July-Dec 2002)

Limited reconstruction analysis job using grid job Limited reconstruction analysis job using grid job

submission interfacesubmission interface

Serving the data results form DC1Serving the data results form DC1 As part of DC1, data must be tagged with meta data for ease of

access

Minimal keywords would be sufficient

Magda already implements portions of this

Job submission with minimal smartsJob submission with minimal smarts Extend GRAPPA work

Move compute resources to data sites


GriPhyN ATLAS Goal 2Dataset Re-creation (Jan-Sept 2003)

Goal: be able to re-create a data fileGoal: be able to re-create a data file

Need to evaluate what parameters need to be kept track ofNeed to evaluate what parameters need to be kept track of

Need to evaluate data needed for full data signatureNeed to evaluate data needed for full data signature

Need to develop a metric for evaluating success - what is Need to develop a metric for evaluating success - what is

good enough?good enough?


GRAPPA: Grid Access Portal for Physics Applications

Provide a point of access to ATLAS grid resourcesProvide a point of access to ATLAS grid resources IU (Physics, CS), Northwestern (ECE), ANL (CS), BU (Physics),

Provide a simple interface for physicists to submit and Provide a simple interface for physicists to submit and

monitor jobs on the Gridmonitor jobs on the Grid Web-based as well as script-based

Ability to “replay”

Compatible with both ATLSim and Athena architectureCompatible with both ATLSim and Athena architecture

Adaptable and/or “extensible” to new developments in Grid Adaptable and/or “extensible” to new developments in Grid

software, Athena, etcsoftware, Athena, etc


GRAPPA Components

User InterfaceUser Interface

Job submissionJob submission

MonitoringMonitoring

BookkeepingBookkeeping

Resource selectionResource selection


Current Status

Simple prototype that allows users to submit an Athena job Simple prototype that allows users to submit an Athena job

from a web interface to the Condor pool on the Atlas IU from a web interface to the Condor pool on the Atlas IU

cluster (via Globus)cluster (via Globus)

Next stepsNext steps Adding more Athena functionality to the interface (e.g., user's

defined libraries)

Experiment with other job launch mechanisms Condor-G and DAGMAN description language Web Services Flow Language as a more general workflow description Explore interfaces for multiple metadata and replica catalog systems.

http://lexus.physics.indiana.edu/~griphyn/grappa/http://lexus.physics.indiana.edu/~griphyn/grappa/


Grid-Enabled Data Access in Athena

David Malon, ANLDavid Malon, ANL

Integrate Grid Data Access techniques (Globus replica Integrate Grid Data Access techniques (Globus replica

catalog and/or GDMP) into the Athena event selection catalog and/or GDMP) into the Athena event selection

modulemodule

When a file is needed, a check is done to see if the file is When a file is needed, a check is done to see if the file is

locallocal if not, use Grid Data Access tools to make it local


Grid-Enabled Data Access in Athena

Athena and GlobusAthena and Globus Search Globus replica catalog and select

Transfer file using the protocol associated with the location object in

the catalog (gsiftp, https/globus-url-copy from a remote

gass_server, ...)

Athena and GDMPAthena and GDMP Work to date uses GDMP 1.2.2, which had not yet incorporated the

Globus replica catalog

Uses the GDMP import/export catalogs

Supports certain subscription-based approaches

Automatically updates Objectivity/DB internal catalog when

Objectivity database files are transferred


Status

Paper presented at CHEPPaper presented at CHEP

Prototype up and running on ANL systemsPrototype up and running on ANL systems GDMP-based prototype work done between CERN and Milan

Next Steps:Next Steps: GDMP will use Globus data replica work

Use GDMP to copy files

Requirements and design work for interfaces between Athena data

producers and metadata catalogs and replica management services

(e.g., Magda)

Metadata work – extending the Athena Event Selector properties to

allow for data signature/virtual data


Monitoring

Joint working group set up between PPDG and GriPhyN to Joint working group set up between PPDG and GriPhyN to

investigate monitoring issuesinvestigate monitoring issues

Led by J. Schopf and D. YuLed by J. Schopf and D. Yu

Monitoring currently defined very broadly:Monitoring currently defined very broadly: Is this router configured correctly?

Has the application finished using that file yet?

What information do I need to determine where to run my

application?


GridView (Kaushik De)

Tool to show the status of the 8 test bed machines on the Tool to show the status of the 8 test bed machines on the

webweb

Uses Globus GRAM to query sites every 30 minsUses Globus GRAM to query sites every 30 mins

Hostname, Uptime, Idletime, # users, and Load averageHostname, Uptime, Idletime, # users, and Load average

Next steps includeNext steps include Integration with Globus information service (MDS)

Visualization as part of cross PPDG/GriPhyN monitoring work


Application Level Monitoring (Taylor)

Initial testbed to incorporate GRID monitoring capabilities Initial testbed to incorporate GRID monitoring capabilities

into Athenainto Athena Collaboration with Valerie Taylor, David Quarrie, and others

Very long ramp-up due to difficulty of “outsiders” running an Very long ramp-up due to difficulty of “outsiders” running an

Athena applicationAthena application

Working on developing an Auditor for AthenaWorking on developing an Auditor for Athena

http://www.ece.nwu.edu/EXTERNAL/vtaylorlab/http://www.ece.nwu.edu/EXTERNAL/vtaylorlab/

prophesy.htmlprophesy.html


Current Status

Mailing list has been set upMailing list has been set up [email protected]

Defining usage casesDefining usage cases Sensors

Predictors

Archiving

Will gather requirements and look at extending Globus Will gather requirements and look at extending Globus

MDS as a common framework to meet these requirementsMDS as a common framework to meet these requirements


Summary

Globus Globus

GriPhyN/iVDGLGriPhyN/iVDGL

GrappaGrappa

Grid-Enabled Data AccessGrid-Enabled Data Access

Monitoring and VisualizationMonitoring and Visualization

Atlas Grid Status - part 1

Documents

Transcript of Atlas Grid Status - part 1