Q-ImPrESS

© ABB Group April 9, 2023 | Slide 1

Q-ImPrESSQuality Impact Prediction for Evolving Service-oriented Software

Heiko Koziolek, DECRC/I1 Industrial Software Technologies, 2010-11-26


Agenda

Context & Motivation

ABB Demonstrat

or

ApplyingQ-ImPrESS

LessonsLearned

EU Project Q-ImPrESSContext


First Discussions

01/2008

Demonstrators finished

12/201004/2009

Tools available for testing

12/2008

Requirementsfinished

12/2009

First Iteration Demonstrators

Consortium members

Q-ImPrESS tools

Ericsson demonstrator ABB demonstrator

Industrial automation: Process Control Systems


MotivationSoftware development at ABB

Continuous evolution of ABB software systems

New requirements, technologies, failure reports

Software maintenance and evolutionare a large cost factor for ABB software

development

Current practice

Design decisions for evolutions based on experience

Prototyping for new technologies, performance impacts

Limited systematic tradeoff analyses

Apply model-based prediction methodsfor systematic tradeoff analysesto save costs and achieve higher quality?


Q-ImPrESS MethodOverview

Eclipse Modeling

Framework

Q-ImPrESS Workbench Backbone

Layered Queueing Networks or Palladio/SimuCOM

Discrete Time Markov Chains

KAMP cost calculator

QVT ModelTransformations

ABB Demonstrator: Process control systemTopology


Plant / Office Network

NetworkIsolation

Device

RemoteWorkplaces

Firewall

Internet

RemoteWorkplaces

Redundant Network

Workplaces

Controllers

Servers

Fieldbus

Remote I/O andField devices

Focus on the server-side

software

Reverse Engineering in Q-ImPrESSProcess/Method


SoMoX3

Q-ImPrESSModel

C++Source

1Structural Investigation of Software Systems

3Software Model eXtractor

2Generalized Abstract Syntax Tree

SISSy1

GAST2

Reverse Engineering in Q-ImPrESSSample Result for 50KLOC subsystem


SISSy parser not robust against Microsoft C++ code Configuring of the metrics intransparent Reference decomposition essential Limited scalability, unclear if faster than manual modelling Static analysis of limited use for QoS prediction

Reverse EngineeringResults from ABB code

Lines of Code

Time for parsing, clustering

Percentage of reconstructed higher level components

Qualitative evaluation(--, -, o, +, ++)

ABB Component 1 10 0.5h / 1h 100% +

ABB Component 2 250 4h / 16h 0% --

ABB Component 3 3300 >60h / ? n/a --


Manual ModelingProcess/Method

Analyzed existing

architectural documentatio

n

Created test-bed (2 quad-

cores, emulated

controllers)

Installed and configured the system

(>100h)

Identified 4 critical usage

scenarios

Configured load drivers with typical workloads

Decided forabstr. level

(processes = components)

Recorded component transitions

Wrote script to derive transition

probabilities

Discussed validity with architects

and developers


Manual ModelingQ-ImPrESS model of the ABB process control system


Performance PredictionProcess

Applied Service

Demand Law (S = X/U)

Measured X, U for each process

(PerfMon)

Determined 20 CPU and HD resource

demands

Parametrized resource demands

(linear regr.)

Abstracted memory, network,

virtualization

Integrated demands into Q-ImPrESS

model

Performed predictions (SimuCom,

LQN)


Performance PredictionSample predictions for different allocation scenarios


Heiko Koziolek. Performance Prediction for an Industrial Control System. Submitted to ACM/SPEC International Conference on Performance Engineering (ICPE‘11)

Achieved prediction error below 30 percent Data collection consumed more time than expected Support for service-oriented systems limited Easy to analyse different evolution scenarios Many bottlenecks below the architectural level

Performance PredictionResults

Workload PerfMonMeasured

SimuComPrediction

Error (%) LQNSPrediction

Error (%)

30 17.146 12.467 27.288 12.464 27.305

60 26.681 22.366 16.174 22.343 16.260

90 31.902 32.347 1.395 32.322 1.317

120 39.016 42.432 8.754 42.329 8.490

150 51.929 51.943 0.027 51.760 0.326


Reliability PredictionData Collection

Acquired access to

bug tracking system

Selected growth model

from IEEE 1633-2008

Performed curve fitting

on bug report dates

(CASRE)

Determined failure rates

per subsystem

Validated failure rates against code

metrics

Annotated components

in Q-ImPrESS

model

Performed reliability

predictions (PRISM)


Reliability PredictionSample sensitivity analysis


Heiko Koziolek, Bastian Schlich, and Carlos Bilich. A Large-Scale Industrial Case Study on Architecture-based Software Reliability Analysis. Proc. 21st IEEE International Symposium on Software Reliability Engineering (ISSRE'10)

Reliability PredictionResults


Possible use of results for future test budget distribution

Abstraction level too high: no hardware, no concurrency

Data collection for lower abstraction levels difficult

Many statistical assumptions (e.g., Markovian control flow)

Validation inherently difficult: would take years to get data

Difficult to establish trust into the results

Data collection method not applicable for new systems

Limited step-by-step guidance and best practice known

Limited amount of case studies in literature

Tradeoff AnalysisDesign space exploration with PerOpteryx


Manually specified degrees of freedom in the model (processor speeds, component allocations, COTS)

PerOpteryx applied an evolutionary algorithm (NSGA II)to search the space spanned by the degrees of freedom

Determines optimal performance/reliability/costs tradeoffs Could be applied for future sizing of the system

Anne Martens, Heiko Koziolek, Steffen Becker, and Ralf Reussner. Automatically improve software architecture models for performance, reliability, and cost using evolutionary

algorithms. Proc. ACM/SPEC International Conference on Performance Engineering (ICPE'10)

Analytical Hierachy Process (from decision support) Requires manual pairwise comparison of candidates Applied on 4 alternatives, specified 24 weights Alternative 3 best because of the preference for reliability Method not scalable: e.g., 180 comparisons for 10 alt.

Tradeoff AnalysisWeighing Alternatives with AHP Wizard


Leo Hatvani, Anton Jansen, Cristina Seceleanu and Paul Pettersson.Integrated Tool for Trade-off Analysis of Quality-of-Service Attributes

Proc. 2nd Int. Workshop on the Quality of Service-oriented Software Systems (QUAOSS’10)

Lessons LearnedExperience with Q-ImPrESS method and tools

Integrated approach with comprehensive tool support

Limited integration with other tools and methods

Cost-effective only for large systems

Limited expressiveness of the Q-ImPrESS meta model

Missing data collection support

Complexity of evolution scenarios determines accuracy

Iterative modelling and strong goal-orientation required


Lessons LearnedCost estimations in person hours

Cost estimation (in person hours) based on our experience

Best case: input data or test-bed readily available

Worst case: unknown system, new data collection method

We actually needed more effort than the worst case!


Conclusions

Q-ImPrESS reverse engineering

Parser not robust for MS C++, clustering intransparent

Q-ImPrESS modeling tools

Well-integrated, graphical editors

Q-ImPrESS performance prediction

Accurate results possible

Q-ImPrESS reliability prediciton

Many assumptions, still primitive models

More research for efficient data collection needed

Internal ABB follow-up planned


Q-ImPrESS

Technology

Transcript of Q-ImPrESS