Augmenting Complex Problem Solving with Hybrid Compute Units
-
Upload
hong-linh-truong -
Category
Education
-
view
303 -
download
1
description
Transcript of Augmenting Complex Problem Solving with Hybrid Compute Units
Augmenting Complex Problem Solving
with Hybrid Compute Units
Hong-Linh Truong, Hoa Khanh Dam, Aditya Ghose, SchahramDustdar
Distributed Systems Group, Vienna University of Technology
University of Wollongong
[email protected]://dsg.tuwien.ac.at/research/viecom
WESOA@ICSOC 2013, 2.12.2013 1
Outline
Motivation
Programming hybrid, collective adaptive systems in
the cloud
Hybrid compute units
Fundamental programming elements and constructs
Framework
Illustrating examples
WESOA@ICSOC 2013,
2.12.2013
2
Hybridity and
Diversity of
Computing ModelsMachine-based
Computing
Human-based
Computing
Things-based
computing
Grid
Pro
cessin
g
Unit
Pro
cessin
g
Unit
Arc
hitectu
reA
rchitectu
reC
om
m.
Com
m.
SMP
S. Dustdar, H. Truong, “Virtualizing Software and Humans
for Elastic Processes in Multiple Clouds – a Service
Management Perspective”, in International Journal of Next
Generation Computing, 2012
Ad hoc networks Web of things
WESOA@ICSOC 2013,
2.12.2013
3
Today’s complex problems need
hybridity and diversity-aware “collective
adaptive systems” (HDA-CAS)
Today’s complex problems need
hybridity and diversity-aware “collective
adaptive systems” (HDA-CAS)
Motivation -- Dynamic hybridity in a
collective Mixture of different types of resources working in
concert in the same collective
Mixture of different roles performed in the same
collective
Different functions based on different capabilities
Mixture of different quality from a single collective
Functions are performed under different quality
Resources possible offer different quality wrt their
role
Mixture of cost/benefit models
A collective might perform a function with different
costs/qualities
WESOA@ICSOC 2013,
2.12.2013
4
Motivation – HAD-CAS
Challenges/difficulties in programming HDA-
CAS in the cloud Human-in-the-loop
Elasticity of costs, quality/benefits and resources
Unified framework for humans and software
Cross-platform with complex underlying APIs
WESOA@ICSOC 2013,
2.12.2013
5
Open problem: there is a lack of suitable programming
frameworks
Open problem: there is a lack of suitable programming
frameworks
Our goals
Construct hybrid collective adaptive systems
Consist of service units abstracting software, people
and things
With different service units roles (computation, data,
management)
Be elastic in multi-cloud environments
Provide common programming features/APIs that
enable different coordination/composition ways for
CASs
WESOA@ICSOC 2013,
2.12.2013
6
Our approach: hybrid compute units for collective adaptive
systems (hCAS)
Our approach: hybrid compute units for collective adaptive
systems (hCAS)
Unified service unit model for
capturing hybridity in a collective
WESOA@ICSOC 2013,
2.12.2013
7
Modeling type of
units (e.g.,
computation, data,
monitor,) and their
dependencies
Consumption,
ownership, provisioning,
price, etc.
ElasticService
Unit
Service model
Unit Dependency
ElasticCapability
Function
The functional
capability of the unit
and interface to
access the function
Capabilities to be elastic
under different
requirements
SoftwareSoftware PeoplePeople
VolunteersProfessionals
ThingThing
Resources
Hybrid compute units for CAS
WESOA@ICSOC 2013,
2.12.2013
8
Hybrid compute unit (HCU): a set of service units
includes software-based services, human-based
services and things-based services that can be
provisioned, deployed and utilized as a collective
on-demand based on different quality, pricing and
incentive models.
Hybrid compute unit (HCU): a set of service units
includes software-based services, human-based
services and things-based services that can be
provisioned, deployed and utilized as a collective
on-demand based on different quality, pricing and
incentive models.
This paper focuses on abstracting service units and
relationships to provide fundamental programming elements
for HCU
Hybrid compute unit design –
fundamental elements
WESOA@ICSOC 2013,
2.12.2013
9
Hybrid compute unit design --
Relationships
WESOA@ICSOC 2013,
2.12.2013
10
Relationship Type HBS SBS TBS HCU
Similarity Yes Yes Yes Yes
Composition Yes Yes Yes Yes
Data Dependency Yes Yes Yes Yes
Control Dependency Yes Yes Yes Yes
Location Dependency Yes Yes Yes Yes
Forwarding Yes Yes No Yes
Delegation Yes Yes No Yes
Social Relation Yes No No Yes
Elasticity Yes Yes No Yes
Hybrid compute unit design – high-
level programming constructs
Current prototype: Fundamental elements can
be implemented as classes in high-level
programming languages
Service units and their properties
Relationship constructs
But complex runtime is still open
Relationships require complex code libraries for
different templates
Complex integration with existing clouds
WESOA@ICSOC 2013,
2.12.2013
11
Hybrid compute unit design – high-
level programming constructs
WESOA@ICSOC 2013,
2.12.2013
12
Relationships Constructs
Similarity similarity(U; V; criteria)
?similarity(U; x; criteria)
Data Dependency datadependency(U;D; [M; ]V )
?datadependency(x;D;M; V ),
?datadependency(U;D;M; x)
?datadependency(U;D; x[c]; V )
Location locationdependency(U; V; ctx; path)
Brokering delegate(U; task; V )
forward(U; task; V )
?delegate(U; task; x)
SocialRelation socialrelation(U; V; ctx; path)
?socialrelation(U; x; distance; ctx;
path).
Elasticity ?elasticity(U; elasticityReq; x)
Towards programming framework
for hCAS
WESOA@ICSOC 2013,
2.12.2013
13
HDA-CAS Coordination
and Composition Models
HDA-CAS Coordination
and Composition Models
[ICSOC12]
Illustrating Example
Evaluating quality of data (QoD) in
simulations/data analytics processes
Typical components in simulation workflows/data
analytics
Goal using hCAS to support quality of data
control of the simulations/data analytics
Reasons: reducing wasting time and computational
cost, dealing with tacit quality evaluation, etc.
WESOA@ICSOC 2013,
2.12.2013
14
pre-processing
solvingpost-
processingdata
analysis
hCAS and simulations
WESOA@ICSOC 2013,
2.12.2013
15
Only SBS is needed,
for example, in the QoD evaluation step
before pre-processing
SBS or HBS is used interchangeably or in
combination
for example, in the QoD evaluation after pre-
processing,
Only HBS is used
e.g., in QoD evaluation after solving.
hCAS and simulation
workflows/data analytics
WESOA@ICSOC 2013,
2.12.2013
16
hCAS
Excerpt -- Programming elasticity
and collectiveness in solving stepsSBS preprocessingUnit =new SBS() ; //create an instance of software unit
File data =new File(fileName);
//...
//create another unit for QoD evaluation
SBS qodEvalUnit = new SBS();
ArrayList paraType = new ArrayList();
paraType.add(File.class.getName());
//the unit must support qodEvaluate function which returns a value in [0,1]
qodEvalUnit.setFunction("qodEvaluate",paraType, Double.class.getName());
//...
//call qod evaluation unit
Double result =(Double)qodEvalUnit.execute("qodEvaluate",params);
…
//call preprocessing activity if QoD is satisfied
if (result > 0.9) {
preProcessedData=preprocessingUnit.execute("preprocessing",params);
}
WESOA@ICSOC 2013,
2.12.2013
17
Excerpt -- Programming elasticity
and collectiveness in solving stepsif (qodPreProcessedData < 0.5) {
//initiate a new unit
ICU dataScientist = new ICU();
//create a dropbox place for sharing data
DropboxAPI<WebAuthSession> scuDropbox = null;
// ....
DropboxAPI.DropboxLink link = scuDropbox.share("/hbscloud");
//ask the cloud of HBS to invoke the ICU
VieCOMHBS vieCOMHBS = new VieCOMHBSImpl();
vieCOMHBS.startHBS(dataScientist);
HBSMessage msg = new HBSMessage();
msg.setMsg("pls. use shared dropbox for communication " +link.url);
vieCOMHBS.sendMessageToHBS(dataScientist, msg);
}
WESOA@ICSOC 2013,
2.12.2013
18
Excerpt – Programming elasticity
and collectiveness in solving stepsif (qodPreProcessedData < 0.9) {
//specify some static properties of the solver
SBS solverUnit2 = new SBS("solver");
solverUnit2.function.put("DIRTY_DATA", Boolean.valueOf(true));
//specify expected cost and accuracy support
CostModel costModel = new CostModel();
costModel.price = 100; //max in EUR
costModel.usageTime = 1000 * 60 * 60; //1 hour
Quality quality = new Quality();
quality.name = Quality.ACCURACY;
quality.value = 0.95; // minimum value
ArrayList nfps = new ArrayList();
nfps.add(quality); nfps.add(costModel);
//find solvers met quality and cost needs
SBS elasticSolverUnit = (SBS)Relationship.elasticity(solverUnit2,nfps);
Object solverResult2 = elasticSolverUnit.execute("solving", params1);
}
WESOA@ICSOC 2013,
2.12.2013
19
Excerpt – Programming location
dependency
//create a new software unit for simulation solvers
SBS solverUnit = new SBS("solver");
//make sure the solver unit and the preprocessing unit are in the same
data center
Relationship.locationDependency(preprocessingUnit,solverUnit,"Amaz
onEC2:Europe");
ArrayList params1 = new ArrayList();
params1.add(preProcessedData);
//execute solver unit with input data from preprocessing unit
Object solverResult= solverUnit.execute("solving",params1);
WESOA@ICSOC 2013,
2.12.2013
20
Excerpt – Forwarding and
delegating analysis request
ICU dataScientist = new ICU();
//....
ICU fUnit = new ICU();
Relationship.socialrelation(dataScientist,
fUnit,1,"Linkedin:DataScienceGroup/TUWien");
Relationship.forward(data, fUnit);
//...
SCU studentSCU = new SCU();
//..
Relationship.delegate(data, studentSCU);
WESOA@ICSOC 2013,
2.12.2013
21
Conclusions and Future Work
Hybrid compute units for implementing hybridity
and diversity-aware collective adaptive systems
(HDA-CAS)
Design fundamental programming elements and
constructs
We are still just at the abstract programming
elements/constructs
Need to work more on mapping from language
construct abstractions to runtime APIs
Runtime system working with several underlying
frameworks for supporting different composition and
coordination techniques
WESOA@ICSOC 2013,
2.12.2013
22
Many thanks!
http://dsg.tuwien.ac.at/research/viecom
WESOA@ICSOC 2013, 2.12.2013 23