Multiple Middleware co-existence: another aspect of Grid Interoperation

Roberto Barbera1,2 Marco Fargetta1

Emidio Giorgio2

1Departement of Phisycs and Astronomy, University of Catania, Italy2 Istituto Nazionale di Fisica Nucleare, Sezione di Catania

{Roberto.Barbera, Marco.Fargetta, Emidio.Giorgio}@ct.infn.it

Abstract

Since the birth of computational and data grids, manymiddlewares have been developed and deployed. Currently,they are used in a multitude of isolated e-Infrastructures anda hot topic in grid technology in the last year or so has beenthat of middleware interoperability/interoperation.

In this paper we present a new approach to grid inter-operation based on the so called middleware co-existence.Following this approach, different middlewares are de-ployed on the same infrastructure and allow the same usersto access and/or share the resources, with well defined pol-icy, regardless the middleware they want to use.

Although this approach has been used for educationalpurposes in a training infrastructure, it is so general thatit could be used everywhere the interoperation is a concernfor the users and/or site managers of an e-Infrastructure.

1. Introduction

The dream of a universal Grid, based on a single infras-tructure with only one middleware deployed, seems to beunlikely to be turned into reality due to both technical andpolitical issues. However, existing infrastructures havingdifferent middlewares, need to communicate (to be ”inter-operable”) between each other in order to provide more re-sources to users.

Since the birth of the grid [5], many middlewares havebeen developed to support this paradigm. Although thesemiddlewares aim to implement almost the same functional-ities, they have many differences in relation to key aspectssuch as security, application integration, communicationmechanism, etc. However, only some middlewares havethe code maturity to be deployed on real grid infrastructure.These include, but are not limited to, Condor[15], gLite[2],Globus[4], OMII-UK[11], UNICORE[7] and some others.

In the last year or so there is a big effort put in placeto allow different middlewares to interoperate between each

other. On this purpose the Open Grid Forum (OGF) [12] hasdeveloped several standards that, if adopted in every mid-dleware, should allow a complete interoperability. How-ever, the adoption of these standard is quite limited, becausethey have a big impact on the architecture and features of themiddleware, so currently the interoperation is indeed verysmall.

Currently, when a grid infrastructure is deployed a givenmiddleware is chosen and then this infrastructure can beused only by clients supporting that specific middleware.Users who want to access the infrastructure need to learnhow the clients work, and this can require a lot of time. Ifusers knowledge background is about a different middle-ware, they need to learn everything again wasting a hugeamount of time.

In this scenario, if two or more grid infrastructure plan tomerge, in order to increase the amount of resources shared,all of them should deploy the same middleware and this isa clear brake in the expansion of grids in the world. Thisproblem could be solved by an effective adoption of stan-dards, but standards cannot take in account all of the tinyfeatures of each middleware, they usually arrange just a sub-set of specifications that have to be implemented in the var-ious middlewares. As a result, even if every middlewarewould comply with standards, a user willing to exploit aspecial feature offered by one particular middleware wouldneed to use the client developed for that middleware andcould not use a common grid infrastructure by accessing itwith clients of a different middleware.

In this paper we focus on a different approach tothe interoperation based on the so called middleware co-existence. In this approach, different middlewares do notneed to communicate between each other in order to havetwo grid infrastructures based on them to be able to merge.Computing resources installed in the two infrastructures areall made available to all the users independently of the par-ticular middleware they want to use. The concept is shownin figure 1.

Following the approach of middleware co-existence, the

Third IEEE International Conference on e-Science and Grid Computing

0-7695-3064-8/07 $25.00 © 2007 IEEEDOI 10.1109/e-Science.2007.17

577

Third IEEE International Conference on e-Science and Grid Computing

0-7695-3064-8/07 $25.00 © 2007 IEEEDOI 10.1109/e-Science.2007.17

577

Figure 1. Multi Middleware infrastructure

integration of two (or more) different grid infrastructuresis much easier and all the users can exploit their previousknowledge base. A working example of middleware co-existence is currently under development in the context ofthe EU funded ICEAGE project [8]. The focus of the projectis to deploy a training infrastructure where different middle-wares can be installed to be demonstrated and taught duringsummer schools and other training events. Although, train-ing and education are the main focuses of this infrastruc-ture, the idea of a Multi-Middleware grid solves the prob-lems highlighted above and seems to be a viable approachfor the integration of different infrastructures, as shown inthe following sections.

The paper is organised as follows: next section presentsan overview of the GILDA t-Infrastructure on top of whichwe have developed the Multi Middleware infrastructure.Section 3 reports on the activity performed to integrate theadditional middlewares in the t-Infrastructure. Then, in sec-tion 4, we present several issues that in future implemen-tation of middleware should be taken into account in orderto simplify the integration activity. An overview of relatedworks is presented in section 5. In section 6, we present thefuture activity to be carried out for the interoperation in thecontext of the ICEAGE project and beyond. Conclusionsare drawn in section 7.

2. Background technologies: the GILDA in-frastructure

The GILDA (Grid Infn Laboratory for Dissemination Ac-tivities) project was started in 2004, by the Italian NationalInstitute of Nuclear Physics (INFN), in the context of boththe Italian INFN Grid project [9], and the European EGEEproject [1]. The purpose of GILDA was to create a gridtestbed entirely dedicated to training and dissemination ac-tivities. GILDA initially consisted of a full grid infrastruc-ture, based on gLite [2] (formerly LCG) middleware, plusother facilities, as a dedicated Certification Authority and aVirtual Organization. This infrastructure, mostly managedon a best effort basis, has been capable to serve the trainingneeds of EGEE and several european grid related projects,supporting more than 200 training events1.

Presently, GILDA is made up of about 20 sites, spreadall over the world, and it is a valid gateway to the grid notonly for final users who want to test their applications, butalso for site administrators willing to experience the prob-lems can arise during the setup and management of a gridinfrastructure.

The fact of being totally decoupled from any productionservice, and substantially indipendent from the needs of re-liability typical of a production environment, has made pos-sible to use GILDA as a laboratory for applications with

1As of July, 31th, 2007

578578

particular requirements and as a testbed where to test newfeatures of the middleware.

This characteristic of openness has been fully exploitedwhen, in the context of the ICEAGE project, the need fora training infrastructure supporting more than one middle-ware during grid schools and beyond arose. The main pur-pose of ICEAGE is, in fact, to advance grid education, andthis purpose is not bound to any specific technology ormiddleware. Therefore, an infrastructure for training, notsingle-middleware based, as most of the existing ones, wasrequired. Although it would have been enough to deployseparately the different chosen middlewares, for practicalpurposes the path of middleware co-existence has been fol-lowed. As we will see, this choice has indeed simplifieduser authentication and authorization on the training infras-tructure, besides the clear benefit of reducing the number ofinvolved machines.

This work has turned GILDA in the first grid infrastruc-ture in the world where different middlewares co-exist andshare the same computational resources.

3. Multiple Middleware Integration

3.1. Middleware selection

The setup of a Multi-Middleware Grid in a productionenvironment can be painful because every element has towork properly and the new components should not alteratethose already deployed. In order to limit problems, onlyfew, well established, middlewares having good documen-tation and support can be considered for the co-existence.Moreover, the architecture of these middlewares should notinfluence or be influenced by any other middlewares. Con-sequently, in order to build a multi-middleware infrastruc-ture a preliminary step is to select the middlewares to beintegrated and analyse thoroughly the features of each one.

Inside the ICEAGE project, where this multi middle-ware architecture has been developed, the initial middle-ware was gLite because the relying infrastructure was theGILDA training infrastructure, that was originally based ongLite, as discussed in section 2. Therefore, additional mid-dlewares should not modify any gLite components, sinceGILDA is used by several other projects. So, integrity wasa strict requirement. Furthermore, to reduce the numberof components, we decided that those already existing andworking should have been reused, whenever and whereverpossible. The general strategy pursued was then to changethe least possible.

Besides gLite, the middlewares chosen for the installa-tion were Globus GT4 [3] and OMII-UK [11]. The choiceof these middleware was due both to technical and diffusionreasons. The diffusion reasons are related to the relevance

of these middlewares in Europe and other parts of the worldin the context of many grid projects and infrastructures.

The technical reasons deal with the architectural differ-ences of the various middlewares and the possibility to in-tegrate every component into a single common element. Inparticular, the elements of the infrastructure that could havebeen clearly reused are the security infrastructure and thelocal clusters. In fact, all of them, and obviously gLite, sup-port Torque/MAUI as local scheduler, and this has giventhe idea to share the local clusters, which in other wordsdoes mean to share what is usually the biggest part of aninfrastructure. Moreover, GT4 and gLite both use X.509certificates and OMII-UK makes use of a keystore able toimport X509 certificates so the same grid security infras-tructure can be shared by the three different middlewares.

3.2. Multi-middleware integration

The first issue faced has regarded the security model.gLite and Globus have common basis for it, having gLitebeen initially adopted the GSI model developed by Globusfor user authentication. This model foresees the use of dig-itally signed certificates and proxies for the mutual authen-tication of users and hosts. Still based on X.509 certifi-cates and proxies, this model has been extended in gLitewith the adoption of the Virtual Organisation MembershipService (VOMS) which issues fully X.509 compatible dig-itally signed extensions to proxies. These extensions bringadditional information on the users which is needed for themapping of the users on different levels of authorisation.The fact that a voms proxy is fully compatible with a X.509proxy ensures that a VOMS proxy can be accepted as au-thentication credential even on resources deploying GT4.

A VOMS server contains information about the users au-thorised to access the grid infrastructure, the VOs they be-long to, the roles they can perform inside the VO, and thesubgroups inside the VO they can be part of. When a usercreates a proxy, with the command voms-proxy-init,he/she has to specify the VO and, if he/she wishes, the groupand role. If the user assigned privileges match the ones re-quested, a proxy is created and extended with the informa-tion returned by the VOMS server, which are called attributecertificates (AC).

Therefore, the responsibility of authentication and autho-risation of users is largely based on the AC extensions re-turned by the VOMS server and included in the proxy usedfor the authentication. In fact, when a job “lands” on gLiteresources two modules (LCAS and LCMAPS) at site level,check (LCAS) that the user belongs to a supported VO andthen, via a sort of ACL, provide for the remote user map-ping to a local account (LCMAPS). Of course, the choiceis influenced by the different roles a user has as well as thegroup he/she belongs to.

579579

Differently, in GT4 the authentication/authorisationmechanism is based on a special file, the grid-mapfile,that can be defined at site level, or by a reciprocal agree-ment. This file contains the list of grid users authorised toaccess the local site and the relative local users. Grid usersare identified by the subject of their certificates, so to accessthe infrastructure the user needs a valid certificate issued bya trusted Certification Authority. After the mapping of thecertificate to a local user, the submitter can execute his/herjob with the authority of the local user. Therefore, in GT4the system administrator defines the policies for authenti-cation/authorisation of the users at site level, modifying thegrid-mapfile, while in gLite this is obtained manip-ulating access control lists, against which the AC presentinside the VOMS proxy are matched.

OMII-UK uses a different approach to handle certificatesfor the authentication/authorisation of the users. The com-munication between the user client and the server is basedon the WS-Security[10], so the server can authenticate theuser by the X.509 certificate used in the communication.On the other hand, the authorisation mechanism is definedat service level and each service can grant different au-thorisation policies to the same user. However, there is aservice supporting the accounting and many services canuse this account mechanism to authorise the user. Whenthe user wants to access an OMII-UK based infrastructurehe/she has to create an account contacting the account ser-vice and providing the personal credentials for the identi-fication. Then the administrator has to accept or reject theaccount and decide the privileges to grant to the user.

The security approach used by the above middlewares isvery different and their unification in a single mechanism isquite difficult. Therefore, in the proposed architecture theeffort has been done on the harmonisation between thesesystems in order to limit the user problems related to themanagement of multiple certificates and tools. However,the activity of harmonisation was limited only to the in-stallation/configuration, without modification of the sourcecode, because the goal was to have co-existence of standardmiddlewares “out-of-the-box”.

For Globus the target was to use the same certificates, thesame local users, and the same authorisation policy definedfor gLite. The certificate model follows the X.509 standardso they can be exchanged between Globus and gLite withoutany problems. VOMS extension were only used for gLite.

At the end, the solution was to set up scripts that peri-odically contacts the VOMS and downloads the list of allthe users authorised to access the site. This list is thenused to update the grid-mapfile routinely, as neededby Globus. In this way, gLite and Globus share the samepolicy. It is worth noting that the same X.509 credentialsare used to access both the gLite and Globus interface onthe local batch scheduler, and even the same set of local ac-

counts where remote users are mapped is shared by the twomiddlewares.

The OMII-UK model is different from that of gLite andit was impossible to share the same policy with gLite, sothe integration was only limited to the fact of reusing a sin-gle certificate for authentication. Also, differently from theGlobus/gLite approach, users have to be accepted manuallyby the administrators of OMII-UK resources, and even sin-gle changes in the set of supported users have to be appliedmanually.

Besides the security mechanism, a second focus of themulti-middleware integration activity was the sharing of theinfrastructure in order to allow users to use the same re-sources with all the middlewares. This means that the clus-ter manager handling the local resources has to be config-ured in a way that it can accept jobs submitted with all thedifferent middlewares. In our infrastructure the local sched-uler is Torque/MAUI and there are several queues where thejobs can be submitted. Torque/MAUI has the advantage tobe very well supported by gLite, Globus and OMII-UK soit has been enough to define new queues for the additionalmiddlewares in order to use the same resources. To definenew queues was not really necessary but it simplified thedebugging, and also allowed for a better definition of re-sources scheduling and usage accounting.

4. Middleware requirements for co-existence

The integration activity described so far has been limitedonly to few components of the architecture. Moreover, forthose components has been adopted a “light” integration,where “light” means that the target of the integration is nota complete interoperation between the middlewares but justtheir co-existence on the same resources.

However, performing the integration activity allowed toidentify several points where the various middlewares couldbe improved for an even better co-existence. The main is-sues are related to the support of existing security mecha-nism and to the user interface integrity, since both elementshave a direct impact on the user activity.

The first one, security, has been partially analysed insection 3.2 where the solution used in the proposed infras-tructure was presented. However, there are several miss-ing features that should be implemented in the middlewaresin order to have a really uniform access. As an example,the OMII-UK approach shows that just the use of a com-mon certificate model is not enough to share the user poli-cies among different middlewares. VOMS based authenti-cation is not currently available in OMII-UK and the factthat user and host certificates required to access the infras-tructure have been manually stored in the keystore, such asto reproduce the same authentication policy supported byother middlewares, was only a workaround. Since the man-

580580

ual replication of user policies in two different architecturesand the preservation of their equivalence is a very heavyand error prone task, this approach cannot be considered ac-ceptable for a large production grid although it was possiblein a relatively little educational grid such as GILDA whereshort-lived user certificates2 are mostly used. Consequently,the co-existence, as the interoperation, to be supported atall sites and at all levels requires that different middlewaresshould be able to exchange users information. This has tobe standardised and the protocol should be dynamic so thatnew information can be added only in one place and auto-matically exported everywhere.

The second issue, related to the user interface, came outduring the process of installation of the clients of the dif-ferent middlewares on the same physical machine, the userinterface. When a middleware is installed and/or used, theuser needs to specify the value for some environment vari-ables. These usually conflict when different middleware aredeployed on the same infrastructure. An example of con-flict is the use, in the user interface, of the same environ-ment variable GLOBUS TCP PORT RANGE by gLite andGlobus but with different syntaxes (both require two val-ues but gLite uses a space to separate the values whereasGlobus requires a comma).

In order to overcome this, after the installation of thedifferent clients needed by the different middlewares, sev-eral shell scripts switching between the various environ-ments have been developed and deployed on the user in-terface. Therefore, when a user has to access the infras-tructure, he/she has to decide the middleware to use andthen call the proper script in order to set the environmentand avoid conflicts. This is graphically shown in figure 2.graphically represents.

This extra work could have been easily avoided if eachmiddleware had been self contained and had not make anyreferences to the user environment. The procedure of pack-aging and deploying of the middlewares should be changedto be as more self-consistent as possible, in order to sim-plify its installation both in a multi-middleware context andin a normal stand-alone configuration.

5. Related works

Considering the relevance of the interoperation prob-lem for current grid infrastructures, the OGF has openeda group, named Grid Interoperation Now CommunityGroup(GIN-CG)[6], to support the improvement of the in-teroperation among different grid infrastructures. The focusof GIN-CG is to plan and implement interoperation provid-ing feedback to standardisation groups.

2Certificate are often released just for the time of a training course thatgenerally go from few days to several weeks.

The activity of GIN-CG is allocated to some sub-groups,each of them focusing on a specific topic. Among thesesub-groups the most relevant to the activity of this paperare: GIN-info and GIN-jobs.

GIN-info aims to produce a Super Information Indexable to collect data from all the grid systems and offer themin a consistent way. They have identified three types ofinformation index currently used in production Grids andtried to define a higher level index containing informationfrom those. However, they use a translation approach tocopy data from the lower information index to the newhigher index, when the information schema are different.As a result, this translation produces lower quality informa-tion because some attribute can differ and the correspondingdata can be lost during the translation. Therefore, the GINSuper Information Index could be a good starting point forthe information index of the multi middleware infrastruc-ture even though it is in an initial level of development andextra work needs to be done to increase the quality of thepublished information.

The aim of GIN-jobs is to define a set of environmentvariables that a job can use regardless the middleware usedto submit it and/or the infrastructure where it is submittedto. This standardisation is very important to develop mid-dleware independent jobs. However, for the multi middle-ware approach it is important to have also a standard for theuser environment in order to define how the job has to besubmitted regardless the grid client used by the user.

Finally, several projects are developing special clientsallowing users to submit their jobs to different infrastruc-tures. A relevant example of these clients is the P-GRADEportal[13]. P-GRADE is a web based service for the de-velopment, execution and monitoring of workflows on vari-ous grid infrastructures. Although P-GRADE allows user toaccess resources through different infrastructure, similarlyto the multi middleware, there are several differences be-tween the two approaches. The most important is that on P-GRADE users access resources through the portal and havea limited the control to the underlying middleware handlingthe resources. Differently, on a multi middleware infras-tructure users have to choose the client and consequentlythe middleware handling the allocation. In fact, the aimof multi middleware is not to make the process transpar-ent to the user but rather to allow the access to resources ofdifferent infrastructures without changing the way the userexploits them.

6. Future plans

There are many directions in which this multi-middleware infrastructure can and will be ex-tended/improved. The first and most obvious onewould be to stress the co-existence aspect including other

581581

gLitePATH=$PATH:/opt/glite/bin

LD_LIBRARY_PATH...GLOBUS_LOCATION...

....

OMIIPATH=$PATH:/opt/OMII/bin

LD_LIBRARY_PATH.......

GT4PATH=$PATH:/opt/GT4LD_LIBRARY_PATH...

GLOBUS_LOCATION.......

user spacesource glite-env.sh

source omii.sh

source gt4.sh

Super User Interface

gLite resources OMII resources GT4 resources

load environment

load environment

load environment

Access Access Access

Figure 2. Multi Middleware infrastructure

widely used grid technologies such as Condor [15] andUnicore [7]3.

Another improvement could be represented by the in-crease of the number of shared components and, wheneverand wherever possible, the enhancement of their interop-eration, with the final goal of reducing the software redun-dancy among the various middlewares and consequently thesize of the software to install on each element of the infras-tructure. For instance, the whole security stack could beshared, since it is based on the X.509 standard, and VOMScould be a common basis for fine grained authorisation inall middlewares. Also the data management aspect mustbe tackled: the final goal would be to make accessible dataindependently from the middleware used to store it. Thecentralised mechanism for authentication already set up forjob submission, and the presence of standards as StorageResource Management [14] are encouraging starting pointsin this direction.

Finally, the biggest challenge is undoubtedly to producenew components able to interact with the different middle-wares in order to support the activity of the user in a moretransparent way.

In this context, the first component to develop should bea Super Information Index capable of collecting informa-tion by different middlewares. Each middleware publishes

3The integration of Unicore is currently analysed

a specific set of information with a specific format. Actu-ally, each middleware produces its set of information forthe site, so two different middlewares may have redundan-cies or show different aspects of the site. As result, a goodmerging of the information in a higher level information in-dex is a very important activity. This Super InformationIndex can be developed starting from the GIN-info activityhighlighted in section 5.

Furthermore, a centralised component, acting as a SuperResource Broker between users and resources, able to prop-erly address job requests, should also be developed. Thiscomponent should always show a single interface to users,independently of the middleware used, so implementing therequired transparency. Although many software allow usersto submit to different infrastructure, such as Condor[15],this super broker should work with existing grid clients.In fact, it should be an additional component of the Gridinfrastructure able to search the resources needed by theusers and provide their entry-points. Therefore, the clienthas to access the resources through the entry-points, as cur-rently done. However, such a component is well beyondthe scope of the ICEAGE project but it would be essentialfor a production grid service supporting the middleware co-existence architecture.

582582

7. Conclusions

The next step in the evolution of grids is the interoper-ation between the many infrastructures developed and de-ployed in the past years. This step is necessary to make thegrid the mainstream environment for all the contexts requir-ing a huge amount of resources to perform computation orstore data, such as in many public and private research cen-tres. Without interoperation the number of resources willremain limited to those available inside a given infrastruc-ture and the overhead of a grid, compared to a simpler newgeneration HPC system, would be difficult to justify.

However, the absence of widely adopted standards andthe large number of different middlewares existing todaymake very difficult to find a single solution to the interoper-ation problem.

In this paper we presented a different approach to in-teroperation carried out in the context of the EU fundedICEAGE project and based on the co-existence of manymiddlewares on the same infrastructure.

The problems found to be solved to reach the full co-existence concern the installation, configuration, and har-monisation of different middlewares in a single site. Infact, these middlewares should share a common policy forusers and resources, such that if a user is granted to accessa resource with a given middleware he/she has to be alsogranted with the other middlewares. The middlewares shar-ing the same computational and storage resources shouldalso publish data on a common Super Information System.If a resource is kept busy by a middleware it should not ac-cessible by the others.

Although it has been introduced in an educational con-text, where the target was only to show how to use differentmiddlewares to access grid resources, the multi-middlewareco-existence could be considered as a practicable and valu-able approach to solve some of interoperation issues in theshort term.

Acknowledgments

This work was supported in part by the EU founded In-ternational Collaboration to Extend and Advance Grid Ed-ucation (ICEAGE) Project and in part by the Italian IstitutoNazionale di Fisica Nucleare (INFN).

References

[1] EGEE Project. http://www.eu-egee.org/.[2] EGEE Project. gLite. http://www.glite.org.[3] I. Foster. Globus Toolkit Version 4: Software for Service-

Oriented Systems. In Network and Parallel Computing,volume 3779 of LNCS, pages 2–13, Beijing, China, 2005.Springer-Verlag.

[4] I. Foster and C. Kesselman. Globus: A Toolkit-Based GridArchitecture. In The Grid: Blueprint for a Future Com-puting Infrastructure, pages 259–278. Morgan Kaufmann,1998.

[5] I. Foster, C. Kesselman, and S. Tuecke. The Anatomy ofthe Grid: Enabling Scalable Virtual Organizations. Inter-national Journal of High Performance Computing Applica-tions, 15(3):200–222, May 2001.

[6] Grid Interoperation Now Community Group. https://forge.gridforum.org/sf/projects/gin.

[7] V. Huber. UNICORE: A Grid Computing Environment forDistributed and Parallel Computing. In V. Malyshkin, editor,Parallel Computing Technologies : 6th International Con-ference, volume 2127 of Lecture Notes in Computer Science(LNCS), Novosibirsk, Russia, September 2001. Springer-Verlag Heidelberg.

[8] ICEAGE: the International Collaboration to Extend and Ad-vance Grid Education. http://www.iceage-eu.org.

[9] INFN Grid. http://grid.infn.it.[10] K. Lawrence, C. Kaler, A. Nadalin, R. Monzillo, and

P. Hallam-Baker. Web Services Security: SOAP MessageSecurity 1.1 (WS-Security 2004). OASIS, http://www.oasis-open.org, February 2006.

[11] OMII-UK. Open Middleware Infrastructure Institure UK.http://omii.ac.uk.

[12] Open Grid Forum. http://www.ogf.org.[13] P-GRADE portal. http://www.lpds.sztaki.hu/

pgportal.[14] Storage Resource Management. http://sdm.lbl.

gov/srm/.[15] D. Thain, T. Tannenbaum, and M. Livny. Distributed com-

puting in practice: the Condor experience. Concurrency andComputation: Practice and Experience, 17(2-4):323–356,2005.

583583