International Grid Communities Dr. Carl Kesselman [email protected] Information Sciences Institute...
-
Upload
megan-skinner -
Category
Documents
-
view
215 -
download
0
Transcript of International Grid Communities Dr. Carl Kesselman [email protected] Information Sciences Institute...
International Grid Communities
Dr. Carl [email protected]
Information Sciences InstituteUniversity of Southern California
The Grid Problem
Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations
Enabling International Cooperation International cooperation valuable, because
– Scale of Grid problem is large– Expertise on both sides of Atlantic & Pacific– Important international applications– Cost of noncooperation can be high
Useful cooperation will not just happen but must be explicitly encouraged– Substantial testbed & application projects, jointly
sponsored by EU, US, others– Transatlantic ‘Terabit’ Testbed, etc.– International Virtual Data Grid Laboratory
Grid Forum
IETF like body to codify standard practice Two meetings held so far, next in April European Grid forum established to
address Europe specific issues
Layered Grid Architecture(By Analogy to Internet Architecture)
Application
Fabric“Controlling things locally”: Access to, & control of, resources
Connectivity“Talking to things”: communication (Internet protocols) & security
Resource“Sharing single resources”: negotiating access, controlling use
Collective“Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services
InternetTransport
Application
Link
Inte
rnet P
roto
col
Arch
itectu
re
The Grid Physics Network
Petabyte-scale computational environment for data intensive science– CMS and Atlas Projects of the Large
Hadron Collider– Laser Interferometer Gravitational-
Wave Observatory– Sloan Digital Sky Survey (200 million
objects each with ~100 attributes)
Data Grids
Integrate data archives into a distributed data management and analysis “Grid”
More than storage & network, also e.g.– Caching and mirroring to exploit locality
– Intelligent scheduling to determine appropriate replica, site for (re)computation, etc.
– Coordinated resource management for performance guarantees
– Embedded security, policy, agent technologies for effective distributed analysis
Virtual Data Grids Only raw data must exist
– Dynamic data production Large extent and scale
– national or worldwide, multiple distance scales
– large numbers of resources Sophisticated new services
– Coordinated use of remote resources Transparency in data-handling and processing
– Optimize for cost, time, policy constraints, …
Grid Communities & Applications:Data Grids for High Energy Physics
Tier2 Centre ~1 TIPS
Online System
Offline Processor Farm
~20 TIPS
CERN Computer Centre
FermiLab ~4 TIPSFrance Regional Centre
Italy Regional Centre
Germany Regional Centre
InstituteInstituteInstituteInstitute ~0.25TIPS
Physicist workstations
~100 MBytes/sec
~100 MBytes/sec
~622 Mbits/sec
~1 MBytes/sec
There is a “bunch crossing” every 25 nsecs.
There are 100 “triggers” per second
Each triggered event is ~1 MByte in size
Physicists work on analysis “channels”.
Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server
Physics data cache
~PBytes/sec
~622 Mbits/sec or Air Freight (deprecated)
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
Caltech ~1 TIPS
~622 Mbits/sec
Tier 0Tier 0
Tier 1Tier 1
Tier 2Tier 2
Tier 4Tier 4
1 TIPS is approximately 25,000
SpecInt95 equivalents
Image courtesy Harvey Newman, Caltech
Virtual Data ToolsRequest Planning and Scheduling
Tools
Request Execution Management
Tools
Transforms
Distributed resources(code, storage,computers, and network)
Resource Management
Services
Resource Management
Services
Security and Policy Services
Security and Policy Services
Other Grid
Services
Other Grid
Services
Interactive User Tools
Production Team
Individual Investigator Other Users
Raw data source
GriPhyn Architecture
?
Major Archive Facilities
Network caches & regional centers
Local sites
GriPhyn Usage Scenario
iVDGL International Virtual-Data Grid Laboratory
– A place to conduct Data Grid tests at scale
– Concrete manifestation of world-wide grid activity
– Continuing activity that will drive Grid awareness
– A basis for further funding Scale of effort
– For national, intl scale Data Grid tests, operations
– Computationally and data intensive computing
– Fast networks Who
– Initially US-UK-EU; Japan, Australia
– Other world regions later
– Discussions w/ Russia, China, Pakistan, India, South America
Structure of the iVDGL
Grid Operations Center Virtual-Data Grid Infrastructure (common middleware and services)
International Experiments
Education and Outreach
Grid Technology Development DataGrid Laboratory Users
DataGrid Laboratory
Resource Providers Resource Centers (Storage and Compute)
Experiment Resources (Storage and Compute)
Outreach Centers (Storage and Compute)
Compute Platform Storage Platform
iVDGL Monitoring Interface
iVDGL Mgmt. Interface
iVDGL Control Interface
Local Management Interface Interface
iGLS
Experiment Scheduler
Health and Status Monitoring
iVDGL Configuration Information
Access Control and Policy Services
iGOCExperiment
Management
Experiment Data Collection
Application Experiments
iVDGL Architecture
iVDGL Map Circa 2003-2004
Tier0/1 facility
Tier2 facility
10 Gbps link
2.5 Gbps link
622 Mbps link
Other link
Tier3 facility
iVDGL as a Laboratory Grid Exercises
– “Easy”, intra-experiment tests first (10-20%, national, transatlantic)
– “Harder” wide-scale tests later (50-100% of all resources)
Local control of resources vitally important– Experiments, politics demand it
Strong interest from other disciplines– HEP + NP experiments
– Virtual Observatory (VO) community in Europe/US
– Gravity wave community in Europe/US/(Japan?)
– Earthquake engineering
– Bioinformatics
– Computer scientists (wide scale tests)
Conclusions
Application communities for major Grid experiments are international– More communities then those mentioned
International testbeds are coming Wires are only part of the solution Common middleware archecture enabling
technology