Managed by

A Lightweight Platform for Integration of Resource Limited Devices into Pervasive

Grids

Stavros Isaiadis and Vladimir Getov

University of Westminsterhttp://www.coregrid.net

s.isaiadis@wmin.ac.uk, v.s.getov@wmin.ac.uk

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 2

• Current Grid Systems• Motivation• Project Overview• Research Issues• Operational Overview• Conclusion

Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 3

Current Grid Systems

• Grids were initially developed to support large (mainly) scientific data-centric applications

• Resources were in abundance and Grids orchestrated sharing in large scale and/or harnessing of idle resources

• Current Grid systems assume:– Plenty of resources– Relatively reliable and static networks

• Traditionally, Grid systems are not concerned with the platform’s footprint: platform is “heavy” and cannot be deployed in resource limited devices

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 4

Grids and “Small” Devices

• But networks have evolved. The “edge” of networks increasingly consists of resource limited and/or mobile devices

• The Grid, however, hasn’t yet allowed the integration of these devices

• While access to the Grid and job submission is generally supported (e.g. through portals), contribution of resources is very difficult

• Resource constraints make installation of Grid middleware prohibiting

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 5

• Current Grid Systems

• Motivation• Project Overview• Research Issues• Operational Overview• Conclusion

Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 6

But Why Integrate Resource Limited Devices?

• Future Grid systems should be truly pervasive and ubiquitous

• Mobile and pervasive computing attract a big part of current research and industry funds and hold a big share of the market pie

• Small and/or mobile devices increasingly offer functionality not available in traditional Grid nodes –multimedia equipment, sensors, global positioning systems etc.

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 7

• Current Grid Systems• Motivation

• Project Overview• Research Issues• Operational Overview• Conclusion

Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 8

A Lightweight Grid Platform

A lightweight platform will provide the foundations for our project:

• Should be based on the CoreGRID component model

• Will consist of a small well defined set of core components

• Extra functionality through pluggable components

• Small footprint and communication overhead so that it won’t drain resources

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 9

• Current Grid Systems• Motivation• Project Overview

• Research Issues• Operational Overview• Conclusion

Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 10

Dealing with Device Dynamicity

• Increased dynamicity is not acceptable in complex and heavily loaded Grid systems

• Solution: – Hide devices behind a proxy and delegate

finer control (scheduling, monitoring, recovery etc) to a local community system

– Devices now form a virtual “cluster” and are presented as a single entity to the Grid.

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 11

Virtual Aggregator Services

• Similar services (that implement the same interface) are aggregated and published through a single interface at the proxy

• The aggregator service implementation orchestrates access to the services it represents (by coordinating the various components like schedulers, indexes etc.)

• A single consistent interface to all aggregated resources is presented regardless of the internal state of the “cluster”

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 12

Making an Unstable Environment Virtually Stable

• When aggregating services that provide controlled access to raw resources (like cpu cycles, memory, storage etc.), we don’t present the total aggregate but only a fraction

• This way we can mask internal failures by reallocating resources from within the cluster without notifying (and thus placing extra burden on) higher level Grid components

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 13

Dealing with Failures

• Even behind a cluster, small devices still suffer from a high number of failures and we must provide the best possible quality of service

• Solution:– A monitoring and failure recovery framework

to predict, detect, diagnose and recover from failures

– Forecasting components to support the community scheduling system providing recommendations on the resource usage plans

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 14

The Monitoring Framework

• A set of small monitoring agents installed in the devices. They collect state information and forward them to the proxy

• A set of monitoring components at the proxy that collect and analyze the information– A monitoring “gateway” makes all decisions

and orchestrates the other components– A forecasting component provides predictions

on the best possible usage plan (according to usage and failure history)

– Failure recovery components decide on the recovery plans based on predefined scenarios

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 15

• Current Grid Systems• Motivation• Project Overview• Research Issues

• Operational Overview• Conclusion

Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 16

Component Prototypes

• Currently using Globus Toolkit 3.2 middleware

• Will port to GT4 in the very near future

• Prototyping components using Java

• Prototypes are quite restrictive at this stage but they will:Help us get a better insightHelp us identify possible obstaclesProvide the skeleton for the functional final

platform

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 17

Operational Overview: Registration of Services

AGGREGATED SERVICE INTERFACES

RESOURCE META-DATA REPOSITORY

Add/Update Interface

Update Index

Discovery and Interface

IdentificationRegistration

LIVE RESOURCE INDEX

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 18

Operational Overview: Job SubmissionJob Submission

Forward

ConsultApplication Tasks

App/ Code

Job Requirements

Distribution

AGGREGATED SERVICE INTERFACES

SERVICE IMPLEMENTATION

LIVE RESOURCE INDEX

TASK EXTRACTION ENGINE

COMMUNITY SCHEDULING SYSTEM

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 19

Operational Overview: Failure Recovery

JOB META-DATA REPOSITORY

Event notification + status info

diagnosisforward

update

Event notification + status info

FORECASTING COMPONENT

LIVE RESOURCE INDEX

MONITORING GATEWAY

FAILURE DIAGNOSIS

FAILURE RECOVERY

plan

PLAN DEPLOYMENT

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 20

• Current Grid Systems• Motivation• Project Overview• Research Issues• Operational Overview

• Conclusion

Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 21

Conclusion

• The lightweight platform could provide the foundations for truly ubiquitous Grids

• Coupled with a lightweight monitoring framework, it is a big first step towards full and efficient integration of resource limited devices into the Grid

• The early prototyping phase will help us gain a better insight on the issues that might arise and will provide the skeleton for the final deliverable platform

Managed by

Questions?

…or comments, suggestions, ideas…

Contact:

www.isaiadis.com

s.isaiadis@wmin.ac.uk