Netzwerkunterstützung für Grid Computing (“Grid networking, ned Gridworking!“)
High-Performance Grid Computing and Research Networking
description
Transcript of High-Performance Grid Computing and Research Networking
1
High-Performance Grid Computing and High-Performance Grid Computing and Research NetworkingResearch Networking
Presented by Selim Kalayci
Instructor: S. Masoud Sadjadihttp://www.cs.fiu.edu/~sadjadi/Teaching/
sadjadi At cs Dot fiu Dot edu
Grid ComputingGrid Computing
2
Acknowledgements The content of many of the slides in this
lecture notes have been adopted from the online resources prepared previously by the people listed below. Many thanks!
Henri Casanova Principles of High Performance Computing http://navet.ics.hawaii.edu/~casanova [email protected]
Ian Foster Presentations&Tutorials from
www.globus.org
3
Agenda Grid Computing Grid Middleware - Globus Security in Globus Data Management Execution Management Monitoring Metaschedulers - Gridway
4
Multiple Computers Adding CPUs to a single computer
becomes very expensive How about multiple computers
together? Linux Clusters (60% of Top-500 list)
Blue/Gene: 30K computers
5
Beyond the machine room? Need more capacity than available at (most) single sites
Everyone would like a 10K-node 100GHz cluster Very expensive (cooling, power) More economical to have multiple sites
Need to locate available resources now Data/Instruments are inherently distributed
Campus
Machine Room
Nation
6
Grid Computing A dynamic multi-institutional network of computers that come
together to share resources for the purpose of coordinated problem solving.
resource
application
institutional boundaryAchieved through:
1. Open general-purpose protocols
2. Standard interfaces
7
Layers in Grid
8
A Grid Checklist coordinates resources that are not subject to centralized
control …
… using standard, open, general-purpose protocols and interfaces …
… to deliver nontrivial qualities of service.
Virtual Organizations Group of individuals or institutions defined by sharing
rules to share the resources of “Grid” for a common goal. Example: Application service providers, storage service
providers, databases, crisis management team, consultants.
9
How is a grid different? Grids focus on site autonomy
Grids involve heterogeneity
Grids involve more resources than just computers and networks
Grids focus on the user
10
Agenda Grid Computing Grid Middleware - Globus Security in Globus Data Management Execution Management Monitoring Metaschedulers - Gridway
11
Grid Infrastructure Distributed management
Of physical resources Of software services Of communities and their policies
Unified treatment Build on Web services framework Use WS-RF, WS-Notification (or
WS-Transfer/Man) to represent/access state
Common management abstractions & interfaces
12
Globus is Open Source Grid Infrastructure
Implement key Web services standards State, notification, security, …
Software for Grid infrastructure Service-enable new & existing resources E.g., GRAM on computer, GridFTP on storage
system, custom application services Uniform abstractions & mechanisms
Tools to build applications that exploit Grid infrastructure Registries, security, data management, …
Enabler of a rich tool & service ecosystem
13
GLOBUS TOOLKIT 4 – GT4
Open source toolkit developed by The Globus Alliance that allows us to build Grid applications.
Organized as a collection of loosely coupled components.
Consists of services, programming libraries, and development tools.
14
GT Domain Areas Core runtime
Infrastructure for building new services Security
Apply uniform policy across distinct systems Execution management
Provision, deploy, & manage services Data management
Discover, transfer, & access large data Monitoring
Discover & monitor dynamic services
15
GT4 Components
16
WSRF & WS-Notification Naming and bindings (basis for virtualization)
Every resource can be uniquely referenced, and has one or more associated services for interacting with it
Lifecycle (basis for fault resilient state mgmt) Resources created by services following factory pattern Resources destroyed immediately or scheduled
Information model (basis for monitoring, discovery) Resource properties associated with resources Operations for querying and setting this info Asynchronous notification of changes to properties
Service groups (basis for registries, collective svcs) Group membership rules & membership management
Base Fault type
17
Agenda Grid Computing Grid Middleware - Globus Security in Globus Data Management Execution Management Monitoring Metaschedulers - Gridway
18
Security Services Forms the underlying communication
medium for all the services Secure Authentication and
Authorization Single Sign-on
User need not explicitly authenticate himself every time a service is requested
Uniform Credentials Ex: GSI (Globus Security Infrastructure)
19
Grid Security Infrastructure - GSI
Grid Security Infrastructure (GSI)
Use GSI as a standard mechanism for bridging disparate security mechanisms
Doesn’t solve trust problem, but now things talk same protocol and understand each other’s identity credentials
Basic support for delegation, policy distribution Translate from other mechanisms to/from GSI
as needed Convert from GSI identity to local identity for
authorization
20
Grid Security Infrastructure - GSI
Grid Security Infrastructure (GSI)
Based on standard PKI technologies CAs allow one-way, light-weight trust relationships (not
just site-to-site) SSL protocol or WS-Security for authentication,
message protection X.509 Certificates for asserting identity
for users, services, hosts, etc. Proxy Certificates
GSI extension to X.509 certificates for delegation, single sign-on
21
Gridmap file A gridmap file at each site maps the grid id of a user to
a local id The grid id of the user is his/her subject in the grid
user certificate The local id is site-specific; multiple grid ids can be mapped to a single local id
Usually a local id exists for each VO participating in that grid effort
The local ids are then used to implement site specific policies
Priorities etc.
22
Gridmap file entry The gridmap-file is maintained by the site
administrator Each entry maps a Grid DN (distinguished
name of the user; subject name) to local user names
##Distinguished Name Local username#
“/DC=org/DC=doegrids/OU=People/CN=Laukik Chitnis 712960” ivdgl“/DC=org/DC=doegrids/OU=People/CN=Richard Cavanaugh 710220” grid3“/DC=org/DC=doegrids/OU=People/CN=JangUk In 712961” ivdgl“/DC=org/DC=doegrids/OU=People/CN=Jorge Rodriguez 690211” osg
23
How to create and use an Identity (1)
Run the below command to generate a personal grid identity certificate.
grid-cert-request
This will create the following files in $HOME/.globus
usercert_request.pem (request to sign certificate)userkey.pem (private key - encrypted)usercert.pem (public key - signed)
24
How to create and use an Identity (2)
After you have created the request then you need to mail it to the local certificate authority:
cat $HOME/.globus/usercert_request.pem | mail [email protected] (or [email protected])
Then the CA will mail you back a signed certificate which you will want to put into $HOME/.globus/usercert.pem
(it can take up to a day for the CA to process the request)
25
Commands to log in / logout grid-proxy-init
This "logs you into" the globus system.
grid-proxy-info Use this to see your status.
grid-proxy-destroy Use this to log out.
A proxy is like a temporary ticket to use the Grid, default in the above case being 12 hours.
Once this is done, you should be able to run “grid jobs” globus-job-run site-name command
26
Agenda Grid Computing Grid Middleware - Globus Security in Globus Data Management Execution Management Monitoring Metaschedulers - Gridway
27
GT4 Data Management Stage/move large data to/from nodes
GridFTP, Reliable File Transfer (RFT) Alone, and integrated with GRAM
Locate data of interest Replica Location Service (RLS)
Replicate data for performance/reliability Distributed Replication Service (DRS)
Provide access to diverse data sources File systems, parallel file systems, hierarchical
storage: GridFTP Databases: OGSA DAI
28
GridFTP
What is GridFTP?
A secure, robust, fast, efficient, standards based, widely accepted data transfer protocol
A Protocol Multiple independent implementations can interoperate
This works. Both the Condor Project at Uwisand Fermi Lab have home grown servers that work with ours.
Lots of people have developed clients independent of the Globus Project.
We also supply a reference implementation: Server Client tools (globus-url-copy) Development Libraries
29
Globus-url-copy GridFTP-compliant client from the Globus team Copy files from one URL to another URL
One URL is usually a gsiftp:// URL Another URL is usually a file:/ URL To move a file from remote GridFTP-enabled server to local machine
% globus-url-copy gsiftp://gcb.fiu.edu/tmp/jt file:/home/skala001/jt
To put file onto server reverse URLs % globus-url-copy file:/home/skala001/jt
gsiftp://gcb.fiu.edu/tmp/jt Monitor performance using –vb flag % globus-url-copy -vb gsiftp://gcb.fiu.edu/tmp/jt
file:/home/skala001/jt
30
Reliable File Transfer - RFT WSRF compliant Fault-tolerant, High- performance
data transfer service Soft state. Notifications/Query
Reliability on top of high performance provided by GridFTP. Fire and Forget. Integrated Automatic Failure Recovery.
Network level failures. System level failures etc.
Essentially a Data transfer scheduler with FIFO as a Queue Policy.
31
RFT
RFT Service
RFT Client
SOAP Messages
Notifications(Optional)
DataChannel
Protocol Interpreter
MasterDSI
DataChannel
SlaveDSI
IPCReceiver
IPC Link
MasterDSI
Protocol Interpreter
Data Channel
IPCReceiver
SlaveDSI
Data Channel
IPC Link
GridFTP Server GridFTP Server
32
Agenda Grid Computing Grid Middleware - Globus Security in Globus Data Management Execution Management Monitoring Metaschedulers - Gridway
33
Execution Management Common WS interface to schedulers
Unix, Condor, LSF, PBS, SGE, … More generally: interface for process
execution management Lay down execution environment Stage data Monitor & manage lifecycle Kill it, clean up
A basis for application-driven provisioning
34
Grid Job Management GoalsProvide a service to securely: Create an environment for a job Stage files to/from environment Cause execution of job process(es)
Via various local resource managers Monitor execution Signal important state changes to client Enable client access to output files
Streaming access during execution
35
GRAM GRAM:Globus Resource Allocation and
Management GRAM is a Globus Toolkit component
For Grid job management GRAM is a unifying remote interface to
Resource Managers Yet preserves local site security/control
GRAM is for stateful job control Reliable operation Asynchronous monitoring and control Remote credential management File staging via RFT and GridFTP
36
GRAMservices
GT4 Java Container
GRAMservices
Delegation
RFT FileTransfer
Transferrequest
GridFTPRemote storage element(s)
Localscheduler
Userjob
Compute element
GridFTP
sudo
GRAMadapter
FTPcontrol
Local job control
Delegate
FTP data
Cli
ent Job
functions
Delegate
Service host(s) and compute element(s)
GT4 WS GRAM Architecture
SEGJob events
37
GRAMservices
GT4 Java Container
GRAMservices
Delegation
RFT FileTransfer
Transferrequest
GridFTPRemote storage element(s)
Localscheduler
Userjob
Compute element
GridFTP
sudo
GRAMadapter
FTPcontrol
Local job control
Delegate
FTP data
Cli
ent Job
functions
Delegate
Service host(s) and compute element(s)
GT4 WS GRAM Architecture
SEGJob events
Delegated credential can be:Made available to the application
38
GRAMservices
GT4 Java Container
GRAMservices
Delegation
RFT FileTransfer
Transferrequest
GridFTPRemote storage element(s)
Localscheduler
Userjob
Compute element
GridFTP
sudo
GRAMadapter
FTPcontrol
Local job control
Delegate
FTP data
Cli
ent Job
functions
Delegate
Service host(s) and compute element(s)
GT4 WS GRAM Architecture
SEGJob events
Delegated credential can be:Used to authenticate with RFT
39
GRAMservices
GT4 Java Container
GRAMservices
Delegation
RFT FileTransfer
Transferrequest
GridFTPRemote storage element(s)
Localscheduler
Userjob
Compute element
GridFTP
sudo
GRAMadapter
FTPcontrol
Local job control
Delegate
FTP data
Cli
ent Job
functions
Delegate
Service host(s) and compute element(s)
GT4 WS GRAM Architecture
SEGJob events
Delegated credential can be:Used to authenticate with GridFTP
40
A Simple Example Command example:% globusrun-ws -submit -c /bin/date
Submitting job...Done.Job ID: uuid:002a6ab8-6036-11d9-bae6-0002a5ad41e5Termination time: 01/07/2005 22:55 GMTCurrent job state: ActiveCurrent job state: CleanUpCurrent job state: DoneDestroying job...Done.
A successful submission will create a new ManagedJob resource with its own unique EPR for messaging
Use –o option to create the EPR file% globusrun-ws -submit –o job.epr -c /bin/date
41
A Simple Example(2) To see the output, use –s (stream) option% globusrun-ws -submit –s -c /bin/date
Termination time: 06/14/2007 18:07 GMTCurrent job state: ActiveCurrent job state: CleanUp-HoldWed Jun 13 14:07:54 EDT 2007Current job state: CleanUpCurrent job state: DoneDestroying job...Done.Cleaning up any delegated credentials...Done.
If you want to send the output to a file, use –so option% globusrun-ws -submit –s –so job.out -c /bin/date
…% cat job.out
Wed Jun 13 14:07:54 EDT 2007
42
A Simple Example(3)Submitting your job to different schedulers Fork% globusrun-ws -submit -Ft Fork -s -c
/bin/date(Actually, the default is Fork. So, you can skip it in this case.)
SGE% globusrun-ws -submit -Ft SGE -s -c
/bin/hostname
43
Batch Job Submissions% globusrun-ws -submit -batch -o job_epr -c
/bin/sleep 50Submitting job...Done.Job ID: uuid:f9544174-60c5-11d9-97e3-0002a5ad41e5Termination time: 01/08/2005 16:05 GMT
% globusrun-ws -status -j job_eprCurrent job state: Active
% globusrun-ws -status -j job_eprCurrent job state: Done
% globusrun-ws -kill -j job_eprRequesting original job description...Done.Destroying job...Done.
44
Complete Factory Contact Override default EPR
Select a different host/service Use “contact” shorthand for convenience
Relies on proprietary knowledge of EPR format!
Command example:
% globusrun-ws -submit –F gcb.fiu.edu\-c /bin/date
45
Read RSL from File Command:% globusrun-ws -submit -f touch.xml
Contents of touch.xml file:<job> <executable>/bin/touch</executable> <argument>touched_it</argument></job>
46
Resource Specification Language (RSL) RSL is the language used by the clients to submit
a job. All job submission requests are described in RSL,
including the executable file and arguments. You can specify the type and capabilities of
resources to execute your job. You can also coordinate Stage-in and Stage-out
operations through RSL.
47
Common/useful options globusrun-ws -J
Perform delegation as necessary for job globusrun-ws -S
Perform delegation as necessary for job’s file staging
globusrun-ws -s Stream stdout/err during job execution to the
terminal globusrun-ws -self
Useful for testing, when you have started the service using your credentials instead of host credentials
48
Staging job<job><executable>/bin/echo</executable><directory>/tmp</directory><argument>Hello</argument><stdout>job.out</stdout><stderr>job.err</stderr><fileStageOut> <transfer> <sourceUrl>file:///tmp/job.out</sourceUrl> <destinationUrl> gsiftp://host.domain:2811/tmp/stage.out </destinationUrl> </transfer></fileStageOut>
</job>
49
RSL Variable Enables late binding of values
Values resolved by GRAM service
System-specific variables ${GLOBUS_USER_HOME} ${GLOBUS_LOCATION} ${GLOBUS_SCRATCH_DIR}
Alternative directory that is shared with compute node
Typically providing more space than user’s HOME dir
50
RSL Variable Example<job><executable>/bin/echo</executable><argument>HOME is ${GLOBUS_USER_HOME}</argument><argument>SCRATCH = ${GLOBUS_SCRATCH_DIR}</argument><argument>GL is ${GLOBUS_LOCATION}</argument><stdout>${GLOBUS_USER_HOME}/echo.stdout</stdout><stderr>${GLOBUS_USER_HOME}/echo.stderr</stderr>
</job>
!!!/tmp/rslExample
51
GRAM Commands Run a job using:
% globus-job-run localhost /bin/date Submit to Fork:
% globus-job-run localhost/jobmanager-fork /bin/date
Submit a batch job using:% globus-job-submit localhost /bin/sleep 50
globus-job-status globus-job-get-output globus-job-cancel
52
Running a Script in GRAM Add this script to file “job”
#! /bin/csh -fecho "Hello World from ";$GLOBUS_LOCATION/bin/globus-hostnameecho arg 1 = $1echo arg 2 = $2echo -n "sum is " echo "$1+$2" | /usr/bin/bc –l
Change the permissions for “job”% chmod +x job
Run the job% globus-job-run localhost ./job 5 6
You should getHello World fromgcb.fiu.eduarg 1 = 5arg 2 = 6sum is 11
!!!/tmp/job
53
Agenda Grid Computing Grid Middleware - Globus Security in Globus Data Management Execution Management Monitoring Metaschedulers - Gridway
54
What is MDS4? Grid-level monitoring system used most often
for resource selection and error notification Aid user/agent to identify host(s) on which to run an
application Make sure that they are up and running correctly
Uses standard interfaces to provide publishing of data, discovery, and data access, including subscription/notification
WS-ResourceProperties, WS-BaseNotification, WS-ServiceGroup
Functions as an hourglass to provide a common interface to lower-level monitoring tools
55
MDS4 Components Information providers
Monitoring is a part of every WSRF service Non-WS services are also be used
Higher level services Index Service – a way to aggregate data Trigger Service – a way to be notified of changes Both built on common aggregator framework
Clients WebMDS
All of the tool are schema-agnostic, but interoperability needs a well-understood common language
56
Information Providers GT4 information providers collect
information from some system and make it accessible as WSRF resource properties
Growing number of information providers Ganglia, CluMon, Nagios SGE, LSF, OpenPBS, PBSPro, Torque
Many opportunities to build additional ones E.g., network monitoring, storage systems,
various sensors
57
Information Providers Data sources for the higher-level services Some are built into services
Any WSRF-compliant service publishes some data automatically
WS-RF gives us standard Query/Subscribe/Notify interfaces
GT4 services: ServiceMetaDataInfo element includes start time, version, and service type name
Most of them also publish additional useful information as resource properties
58
Information Providers:GT4 Services Reliable File Transfer Service (RFT)
Service status data, number of active transfers, transfer status, information about the resource running the service
Community Authorization Service (CAS) Identifies the VO served by the service instance
Replica Location Service (RLS) Note: not a WS Location of replicas on physical storage
systems (based on user registrations) for later queries
59
Information Providers (2) Other sources of data
Any executables Other (non-WS) services Interface to another archive or data
store File scraping
Just need to produce a valid XML document
60
Information Providers:Cluster and Queue Data
Interfaces to Hawkeye, Ganglia, CluMon, Nagios Basic host data (name, ID), processor information,
memory size, OS name and version, file system data, processor load data
Some condor/cluster specific data This can also be done for sub-clusters, not just at
the host level Interfaces to PBS, Torque, LSF
Queue information, number of CPUs available and free, job count information, some memory statistics and host info for head node of cluster
61
Higher-Level Services Index Service
Caching registry Trigger Service
Warn on error conditions
All of these have common needs, and are built on a common framework
62
MDS4 Index Service Index Service is both registry and cache
Datatype and data provider info, like a registry (UDDI) Last value of data, like a cache
Subscribes to information providers In memory default approach
DB backing store currently being discussed to allow for very large indexes
Can be set up for a site or set of sites, a specific set of project data, or for user-specific data only
Can be a multi-rooted hierarchy No *global* index
63
Container-wide Index Each GT4 container has a local index Collects information about services in that
container Each service registers to container index when
correctly configured
64
VO-wide indexes Local indexes can be registered to VO wide indexes Configfile at resource container or at VO index –
contains URL for resource or VO index
65
MDS4 Trigger Service Subscribe to a set of resource
properties Evaluate that data against a set of
pre-configured conditions (triggers) When a condition matches, action
occurs Email is sent to pre-defined address Website updated
66
Information models Each information sources publishes
information in XML according to some schema.
Some times the author of the information source or the grid resource defines that schema.
Some collaborative efforts to define common schemas–for example GLUE for compute information
Schema typically written in XSD, but not required
67
GLUE schema Grid Laboratory Uniform Environment Schema developed by DataTAG for
EU/USA interoperability. Modelled in UML Implementations
XML version for MDS Information collected from various cluster
monitoring systems Also: LDAP and SQL versions (used by older
versions of MDS and other monitoring systems).
68
MDS user interfaces
General purpose UIs Web browser based interface -
WebMDS Command line tools
Specialized clients Brokers
69
WebMDS
Web-based interface to display monitoring information
Easily extensible for new data using XSLT
70
MDS4 - Command Line Xpath Queries to query the Index Service To see all collected in the Index Service
wsrf-query -s \ https://gcb.fiu.edu:8443/wsrf/services/DefaultIndexService
To see the number of free nodes: wsrf-query -s
https://gcb.fiu.edu:8443/wsrf/services/DefaultIndexService "number(//*/glue:GLUECE//glue:ComputingElement/glue:State/@glue:FreeCPUs)"
To see how many jobs are currently running: wsrf-query -s
https://gcb.fiu.edu:8443/wsrf/services/DefaultIndexService "number(//*[local-name()='GLUECE']//glue:ComputingElement//glue:State/@glue:TotalJobs)"
71
Configuring GRAM to use a cluster monitoring system
GRAM extracts and publishes cluster information from either Ganglia or Hawkeye
$GLOBUS_LOCATION/etc/globus_wsrf_mds_usefulrp/gluerp.xml
<defaultProvider> tag specifies whether to use Ganglia or Hawkeye or none.
Uncomment appropriate example supplied in the configfile
72
Agenda Grid Computing Grid Middleware - Globus Security in Globus Data Management Execution Management Monitoring Metaschedulers - Gridway
73
Grid Meta-Scheduler Local Schedulers is not fit for Grid environment
Meta-scheduler(s) should interact with lower-level schedulers for scheduling decisions
Resources (Computational, Data, Network, etc.) and Jobs are other entities, Meta-Scheduler should be aware of and interact with
Meta-Scheduler uses existing Grid services
74
GridWay Lightweight metascheduler on top of GT 2.4 –
4.x Properties:
Support of GGF DRMA standard API for job submission and management
Support for JSDL Simple scheduling mechanisms but extensible Interoperability between different grid infrastructures
and middlewares (Globus, EGEE, UNICORE…) Allows job dependencies (workflow) Supports job migration/adaptive execution (Grid- and
application-initiated)
75
GridWay Architecture
RequestManager
DispatchManager
InformationManager
ExecutionManager
TransferManager
SchedulerScheduler
GridWay Core
Jobpool
Hostpool
GRAMRFT MDS
Resource
GRAMRFT MDS DRMAA Library CLIJob control operations
Matchmaking, execution and
migration
Execution of jobs on LRM
Performance Monitor
76
GridWay Modules Request Manager Interfaces with client
commands Dispatch Manager Performs job scheduling Information Manager Resource Monitoring and
data gathering Execution Manager Executes job stages Performance Monitor Evaluates the job
performance
77
Scheduling Strategy Dispatch manager wakes up at every scheduling
interval Uses Resource Selector to select the host(s) to
submit the job Resource Selector interfaces with Grid Information
Services, such as MDS Resource Selector returns a candidate list of hosts
to submit the job by using a policy script You can implement your own policy script, so it is
extensible Dispatch Manager then submits the job to the
Execution Manager
78
GridWay Commands gwd - start the daemon gwhost - information about
resources gwps - information about jobs gwuser - information about users gwsubmit - submits job gwkill - cancels a job