Agent Dynamics "Leading the way in real estate team performance" Performance Management Program
Software Engineering Research: Leading a Double-Agent Life.
-
Upload
lionel-briand -
Category
Technology
-
view
13.111 -
download
1
description
Transcript of Software Engineering Research: Leading a Double-Agent Life.
Useful Software Engineering Research: Leading a Double-Agent Life
Lionel Briand IEEE Fellow FNR PEARL Chair Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT) University of Luxembourg APSEC, Bangkok, December 3rd, 2013
SnT$Centre$
• SnT centre, Est. 2009: Interdisciplinary, ICT security, reliability, and trust (SnT)
• Luxembourg city
• 220 scientists and Ph.D. candidates, 20 industry partners
• SVV Lab: Established January 2012, www.svv.lu
• 25 scientists (Research scientists, associates, and PhD candidates)
• Industry-relevant basic and applied research on system dependability: security, safety, reliability
• Six partners: Cetrel, CTIE, Delphi, SES, IEE, Hitec, …
2
Software Everywhere
3
Software Everywhere
4
World Software Market: • $296 Billion in 2013 • $360 Billion in 2016 • Annual growth of 6%
Failures Everywhere …
5
Failures Everywhere …
6
McKinsey, U. Oxford, 2012: • 5400 large IT projects (> 15 $M) • 17% threatened the very
existence of companies • On average: 45% over budget,
56 percent less value
Software Engineering Research Funding & Relevance
• Software Engineering (SE) research should be a top priority given its importance in society.
• But that is not the case anymore (with a few exceptions). • Symptoms
– Listed priorities by research councils, relative funding – University hiring, few software engineering departments – Large centers or institutes being established or closed down
• May be partly related to (perceived) lack of relevance? – Industry participation in leading SE conferences – Application/Experience tracks not first class citizens – A very small percentage of research work is ever used or even
assessed on real industrial software – Impact project (ACM SIGSOFT)
7
Basili’s and Meyer’s Take
• Many (most?) of the advances in software engineering have come out of non-university sources
• “Academic research has had its part, honorable but limited.” (Meyer)
• Large scale labs don’t get funded, like they do in other engineering and scientific disciplines (Basili, Meyer)
• One significant difference though is that we cannot entirely recreate the phenomena we study within four walls
• Question: What is our responsibility in all this?
8
Setting the Stage
Engineering Research
• “Engineering: The application of scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines, processes, and systems.” (American Heritage Dictionary)
• Engineering research: innovative engineering solutions – Problem driven
– Real world requirements
– Scalability
– Human factors, where it matters – Economic tradeoffs and cost-benefit analysis
10
Pasteur’s Quadrant
Quest for Fundamental Understanding?
Consideration of Use?
Yes
No Yes
Pure Basic Research (Bohr)
Use-inspired Basic Research (Pasteur)
No - Pure Applied Research (Edison)
11 Donald Stokes, Pasteur’s Quadrant, 1997
Software Engineering Research
• Pure basic research in software engineering?
• As an engineering discipline, all research work should contribute to:
– Knowledge discovery but also …
– Innovative (software) engineering solutions
• Pasteur’s quadrant: Research should driven by utility and its results be put to use …
• This requires a knowledge of engineering practice
• Are the majority of projects and papers doing so?
• If not, why is that? What can we do about it? 12
Research Example
A Representative Example
• Parnin and Orso (ISSTA, 2011) looked at automated debugging techniques
• 50 years of automated debugging research • Only 5 papers have evaluated automated debugging
techniques with actual programmers • Focus since ~2001: dozens of papers ranking program
statements according to their likelihood of containing a fault
• Experiment – How do programmers use the ranking? – Do they see the bugs? – Is the ranking important?
14
Results from Parnin and Orso’s Study
• Only low performers strictly followed the ranking
• Only one out of 10 programmers who checked a buggy statement stopped the investigation
• Automated support did not speed up debugging
• Developers wanted explanations rather than recommendations
• We cannot abstract the human away in our research
• “… we must steer research towards more promising directions that take into account the way programmers actually debug in real scenarios.”
15
What Happened?
• How people debug and what information they need is poorly understood – Probably varies a great deal according to context and skills
• Researchers focused on providing a solution that was a mismatch for the actual problem
• That line of research became fashionable: a lot of (cool) ideas could be easily applied and compared, without involving human participants
• Resulted in many, many papers … • Pasteur’s quadrant: That idea was never really put to use … • Similar example: ISSTA 2012 paper on learning program
invariants (Daikon) by Matt Staats et al. • Many other examples
16
Reconciling Relevance and Science in Software Engineering
-- Implementing Pasteur’s Quadrant
Objectives
• How to implement Pasteur’s quadrant in software engineering?
• Go through recent and successful projects with industry partners – Simula Research Laboratory, Oslo, Norway
– SnT centre, U. of Luxembourg
• Summarize what happened, our experience
• Two projects, in the automotive domain, described in more detail
• Draw conclusions and lessons learned – Patterns for successful research
– Challenges and possible solutions
18
Mode of Collaboration
19
Adapted from Gorschek et al., IEEE Software 2006
Projects$Overview$(<$5$years)$
20
Company Domain Objective Notation Automation
ABB Robot controller Func. Safety Testing UML Model analysis for coverage criteria
Cisco Video conference Testing (robustness) UML profile Metaheuristic search
Kongsberg Maritime Fire and gas safety control system
Safety certification SysML + traceability Model slicing algorithm
Kongsberg Maritime Oil&gas, safety critical drivers
CPU usage analysis UML+MARTE Constraint Solver
FMC Subsea system Automated configuration
UML profile Constraint solver
WesternGeco Marine seismic acquisition
Func. Testing UML profile + MARTE Metaheuristic search
DNV Marine and Energy, certification body
Compliance with safety standards
UML profile Constraint verification
SES Satellite operator Func. Testing UML profile Metaheuristic search
Delphi Automotive systems Testing (safety+performance)
Matlab/Simulink Metaheuristic search
Lux. Tax department Legal & financial Legal Req. QA & testing
UML profile Under investigation
Project Example 1: Cisco
• Context: Video conferencing systems
• Original scientific problem: Modeling and test case generation, oracle, coverage strategy
• Practical observation: Access to test network infrastructure limited (emulate network traffic, etc.). Models get too large and complex.
• Modified research objectives: (1) How to select an optimal subset of test cases matching the time budget, (2) Modeling cross-cutting concerns
• References: Hemmati et al. (2010),
Ali et al. (2011)
21
Project Example 1: Cisco
• Context: Video conferencing systems
• Original scientific problem: Modeling and model-based test case generation, oracle, coverage strategy
• Practical observation: Access to test network infrastructure limited (emulate network traffic, etc.). Models get too large and complex.
• Modified research objectives: (1) How to select an optimal subset of test cases matching the time budget, (2) Modeling cross-cutting concerns
• References: Hemmati et al. (2010),
Ali et al. (2011)
22
Project Example 2: Siemens
• Context: 3-D image segmentation algorithms for medical applications (Siemens)
• Original scientific problem: Define specific test strategies for segmentation algorithms
• Practical observations: Algorithms are validated by using highly specialized medical experts. Expensive and slow. No obvious test oracle
• Modified research objective: Learning oracles for image segmentation algorithms in medical applications. Machine learning.
• Reference: Frouchni et al. (2011)
23
Project Example 2: Siemens
• Context: 3-D image segmentation algorithms for medical applications (Siemens)
• Original scientific problem: Define specific test strategies for (constantly evolving) segmentation algorithms
• Practical observations: Test results are validated by using highly specialized medical experts. Expensive and slow. No obvious test oracle: non-testable systems
• Modified research objective: Learning oracles for image segmentation algorithms in medical applications. Machine learning.
• Reference: Frouchni et al. (2011)
24
Project Example 3: FMC
25
• Context: Subsea integrated control systems
• Original scientific problem: integration in systems of systems
• Practical observations: Each subsea installation is unique (variant), the software configuration is extremely complex (thousands of configuration parameters), > 50% configuration faults
• Modified research objective: Real-time, automated support to the configuration process using domain specific product line modeling (focus on HW-SW dependencies) and constraint solving: : instant validation, guidance, inferred values
• Behjati et al. (2012)
Project Example 3: FMC
• Context: Subsea integrated control systems
• Original scientific problem: integration in systems of systems
• Practical observations: Each subsea installation is unique (variant), the software configuration is extremely complex (thousands of configuration parameters), > 50% configuration faults
• Modified research objective: Real-time, automated support to the configuration process using domain specific product line modeling (focus on HW-SW dependencies) and constraint solving: instant validation, guidance, inferred values
• Behjati et al. (2012) 26
Project Example 4: Kongsberg Maritime
• Context: safety-critical embedded systems in the energy and maritime sectors, e.g., fire and gas monitoring, process shutdown, dynamic positioning
• Original scientific problem: Model-driven engineering for failure-mode and effect analysis
• Practical observations: Certification meetings with third-party certifiers. Certification is lengthy, expensive, etc. Requirements-Design decisions traceability in large complex systems a priority.
• Modified research objective: Traceability between safety requirements and system design decisions. Solution based on SysML and a simple traceability language along with model slicing.
• Reference: Sabetzadeh et al. (2011)
27
Project Example 4: Kongsberg Maritime
• Context: safety-critical embedded systems in the energy and maritime sectors, e.g., fire and gas monitoring, process shutdown, dynamic positioning
• Original scientific problem: Model-driven engineering for failure-mode and effect analysis
• Practical observations: Certification meetings with third-party certifiers: Certification is lengthy, expensive, etc. Requirements-Design decisions traceability in large complex systems is an issue.
• Modified research objective: Traceability between safety requirements and system design decisions. Solution based on SysML and a simple traceability language along with model slicing.
• Reference: Sabetzadeh et al. (2011), Nejati et al. (2012)
28
Detailed Example 1
Testing Closed Loop Controllers
R. Matinnejad et al., 2013
29
Complexity$and$amount$of$so?ware$used$on$vehicles’$$Electronic$Control$Units$(ECUs)$grow$rapidly$$
More functions
Comfort and variety
Safety and reliability
Faster time-to-market
Less fuel consumption
Greenhouse gas emission laws
30
Model-based control system development has three major stages
31
Hardware-in-the-Loop Stage
Model-in-the-Loop Stage
Simulink Modeling
Generic Functional
Model
MiL Testing
Software-in-the-Loop Stage
Code Generationand Integration
Software Running on ECU
SiL Testing
SoftwareRelease
HiL Testing
Major$Challenges$in$MiLGSiLGHiL$TesIng$$
• Manual test case generation
• Complex functions at MiL, and large and integrated software/embedded systems at HiL
• Lack of precise requirements and testing Objectives
• Test selection at HiL 32
MiL$tesIng$
Requirements
The ultimate goal of MiL testing is to ensure that individual functions behave correctly and timely on any hardware configuration
Individual Functions
33
A$Taxonomy$of$AutomoIve$FuncIons$
Controlling Computation
State-Based Continuous Transforming Calculating
unit convertors calculating positions, duty cycles, etc
State machine controllers
Closed-loop controllers (PID)
Different testing strategies are required for different types of functions
34
Dynamic Continuous Controllers are present in many embedded systems including ECUs
35
Controller Plant Model and its Requirements
Plant Model
Controller(SUT)
Desired value Error
Actual value
System output+-
=<
~= 0>=
time time time
Des
ired
Valu
e &
Actu
al V
alue
Desired ValueActual Value
(a) (b) (c)Liveness Smoothness Responsiveness
x
y
z
v
w
36
37
Search$Elements$
• Search:
• Inputs: Initial and desired values, configuration parameters • (1+1) EA
• Search Objective:
• Example requirement that we want to test: liveness
! |Desired - Actual(final)|~= 0
For each set of inputs, we evaluate the objective function over the resulting simulation graphs:
• Result:
• worst case scenarios or values to the input variables that are more likely to break the requirement at MiL level
• stress test cases based on actual hardware (HiL)
37
MiL-Testing of Continuous Controllers
Exploration+Controller-plant model
Objective Functions
Overview Diagram
Test Scenarios
List of Regions Local SearchDomain
Expert
time
Desired ValueActual Value
0 1 20.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Initial Desired
Final Desired
38
• Addressed a testing problem that has been largely ignored
• We found much worse scenarios during MiL testing than our partner had found so far
• Much worse than random search
• They are running them at the HiL level, where testing is much more expensive: MiL results -> test selection for HiL
• But further research is needed:
– To deal with the many configuration parameters
– To dynamically adjust search algorithms in different subregions with different lanscapes
Conclusions$
i.e., 31s. Hence, the horizontal axis of the diagrams in Figure 8 shows the number ofiterations instead of the computation time. In addition, we start both random search and(1+1) EA from the same initial point, i.e., the worst case from the exploration step.
Overall in all the regions, (1+1) EA eventually reaches its plateau at a value higherthan the random search plateau value. Further, (1+1) EA is more deterministic than ran-dom, i.e., the distribution of (1+1) EA has a smaller variance than that of random search,especially when reaching the plateau (see Figure 8). In some regions (e.g., Figure 8(d)),however, random reaches its plateau slightly faster than (1+1) EA, while in some otherregions (e.g. Figure 8(a)), (1+1) EA is faster. We will discuss the relationship betweenthe region landscape and the performance of (1+1) EA in RQ3.RQ3. We drew the landscape for the 11 regions in our experiment. For example, Fig-ure 9 shows the landscape for two selected regions in Figures 7(a) and 7(b). Specifically,Figure 9(a) shows the landscape for the region in Figure 7(b) where (1+1) EA is fasterthan random, and Figure 9(b) shows the landscape for the region in Figure 7(a) where(1+1) EA is slower than random search.
0.30
0.31
0.32
0.33
0.34
0.35
0.36
0.37
0.38
0.39
0.40
0.70 0.71 0.72 0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.800.10
0.11
0.12
0.13
0.14
0.15
0.16
0.17
0.18
0.19
0.20
0.90 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1.00
(a) (b)
Fig. 9. Diagrams representing the landscape for two representative HeatMap regions: (a) Land-scape for the region in Figure 7(b). (b) Landscape for the region in Figure 7(a).
Our observations show that the regions surrounded mostly by dark shaded regionstypically have a clear gradient between the initial point of the search and the worst casepoint (see e.g., Figure 9(a)). However, dark regions located in a generally light shadedarea have a noisier shape with several local optimum (see e.g., Figure 9(b)). It is knownthat for regions like Figure 9(a), exploitative search works best, while for those like Fig-ure 9(b), explorative search is most suitable [10]. This is confirmed in our work wherefor Figure 9(a), our exploitative search, i.e., (1+1) EA with � = 0.01, is faster and moreeffective than random search, whereas for Figure 9(b), our search is slower than randomsearch. We applied a more explorative version of (1+1) EA where we let � = 0.03 to theregion in Figure 9(b). The result (Figure 10) shows that the more explorative (1+1) EAis now both faster and more effective than random search. We conjecture that, from theHeatMap diagrams, we can predict which search algorithm to use for the single-statesearch step. Specifically, for dark regions surrounded by dark shaded areas, we suggestan exploitative (1+1) EA (e.g., � = 0.01), while for dark regions located in light shadedareas, we recommend a more explorative (1+1) EA (e.g., � = 0.03).
6 Related WorkTesting continuous control systems presents a number of challenges, and is not yet sup-ported by existing tools and techniques [4, 1, 3]. The modeling languages that have been
13
39
Detailed Example 2
Minimizing CPU Time Shortage Risks in Integrated Embedded Software
S. Nejati et al., 2013
Today’s$cars$rely$on$integrated$systems$
• Modular and independent development
• Many opportunities for division of labor and outsourcing
• Need for reliable and effective integration processes
41
IntegraIon$process$in$the$automoIve$domain$
AUTOSAR Models sw runnables
sw runnables AUTOSAR Models
Glue
42
43
CPU$Time$Shortage$in$Integrated$Embedded$$So?ware$
• Static cyclic scheduling: predictable, analyzable • Challenge
– Many OS tasks and their many runnables run within a limited available CPU time
• The execution time of the runnables may exceed their time slot
• Our goal – Reducing the maximum CPU time used per time slot to be
able to • Minimize the hardware cost • Reduce the probability of overloading the CPU in practice • Enable addition of new functions incrementally
43
5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms
✗
5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms ✔
(a)
(b)
Fig. 4. Two possible CPU time usage simulations for an OS task with a 5mscycle: (a) Usage with bursts, and (b) Desirable usage.
its corresponding glue code starts by a set of declarationsand definitions for components, runnables, ports, etc. It thenincludes the initialization part followed by the execution part.In the execution part, there is one routine for each OS task.These routines are called by the scheduler of the underlyingOS in every cycle of their corresponding task. Inside eachOS task routine, the runnables related to that OS task arecalled based on their period. For example, in Figure 3, weassume that the cycle of the task o1 is 5ms, and the periodof the runnables r1, r2, and r3 are 10ms, 20ms and 100ms,respectively. The value of timer is the global system time. Sincethe cycle of o1 is 5, the value of timer in the Task o1() routineis always a multiple of 5. Runnables r1, r2 and r3 are thencalled whenever the value of timer is zero, or is divisible bythe period of r1, r2 and r3, respectively.
Although AUTOSAR provides a standard means for OEMsand suppliers to exchange their software, and essentiallyenables the process in Figure 1, the automotive integrationprocess still remains complex and erroneous. A major inte-gration challenge is to minimize the risk of CPU shortagewhile running the integrated system in Figure 1. Specifically,consider an OS task with a 5ms cycle. Figure 4 shows twopossible CPU time usage simulations of this task over eighttime slots between 0 to 40ms. In Figure 4(a), there are burstsof high CPU usage at two time slots at 0ms and 35ms, whilethe CPU usage simulation in Figure 4(b) is more stable anddoes not include any bursts. In both simulations, the totalCPU usage is the same, but the distribution of the CPU usageover time slots is different. The simulation in Figure 4(b) ismore desirable because: (1) It minimizes the hardware costsby lowering the maximum required CPU time. (2) It facilitatesthe assignment of new runnables to an OS task, and hence,enables the addition of new functions as it is typically done inthe incremental design of car manufacturers. (3) It reduces thepossibility of overloading CPU as the CPU time usage is lesslikely to exceed the OS task cycle (i.e., 5ms) in any time slot.Ideally, a CPU usage simulation is desirable if in each timeslot, there is a sufficiently large safety margin of unused CPUtime. Due to inaccuracies in estimating runnables’ executiontimes, it is expected that the unused margin shrinks when thesystem runs in a real car. Hence, the larger is this margin, thelower is the probability of exceeding the limit in practice.
In this paper, we study the problem of minimizing burstsof CPU time usage for a software system composed of alarge number of concurrent runnables. A known strategy toeliminate high CPU usage bursts is to shift the start time(offset) of runnables, i.e., to insert a delay prior to the start ofthe execution of runnables [5]. Offsets of the runnables mustsatisfy three constraints: C1. The offset values should not lead
to deadline misses, i.e., they should not cause the runnables torun passed their periods. C2. Since the runnables are invokedby OS tasks, the offset values of each runnable should bedivisible by the OS task cycle related to that runnable. C3. Theoffset values should not interfere with data dependency andsynchronization relations between runnables. For example,suppose runnables r1 and r2 have to execute in the same timeslot because they need to synchronize. The offset values of r1and r2 should be chosen such that they still run in the sametime slot after being shifted by their offsets.
There are four important context factors that are in line withAUTOSAR [13], and have influenced our work:
CF1. The runnables are not memory-bound, i.e., the CPUtime is not significantly affected by the low-bound memoryallocation activities such as transferring data in and out ofthe disk and garbage collection. Hence, our analysis of CPUtime usage is not affected by constraints related to memoryresources (see Section III-B).
CF2. The runnables are Offset-free [4], that is the offset ofa runnable can be freely chosen as long as it does not violatethe timing constraints C1-C3 (see Section III-B).
CF3. The runnables assigned to different OS tasks areindependent in the sense that they do not communicate withone another and do not share memory. Hence, the CPU timeused by an OS task during each cycle is not affected by otherOS tasks running concurrently. Our analysis in this paper,therefore, focuses on individual OS tasks.
CF4. The execution times of the runnables are remarkablysmaller than the runnables’ periods and the OS task cycles.Typical OS task cycles are around 1ms to 5ms. The runnables’periods are typically between 10ms to 1s, while the runnables’execution times are between 10ns = 10�5ms to 0.2ms.
Our goal is to compute offsets for runnables such that theCPU usage is minimized, and further, the timing constraints,C1-C3, discussed earlier above hold. This requires solvinga constraint-based optimization problem, and can be done inthree ways: (1) Attempting to predict optimal offsets in a de-terministic way, e.g., algorithms based on real-time schedulingtheory [6]. In general, these algorithms explore a very smallpart of the search space, i.e., worst/best case situations only(see Section V for a discussion). (2) Formulating the problemas a (symbolic) constraint model and applying a systematicconstraint solver [14], [15]. Due to assumption CF4 above,the search space in our problem is too large, resulting ina huge constraint model that does not fit in memory (seeSection V for more details). (3) Using metaheuristic search-based techniques [9]. These techniques are part of the generalclass of stochastic optimization algorithms which employsome degree of randomness to find optimal (or as optimalas possible) solutions to hard problems. These approaches areapplied to a wide range of problems, and are used in this paper.
III. SEARCH-BASED CPU USAGE MINIMIZATION
In this section, we describe our search-based technique forCPU usage minimization. We first define a notation for ourproblem in Section III-A. We formalize the timing constraints,
44
Using&runnable&offsets&(delay&3mes)&
5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms
5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms ✗
✔
Inserting runnables’ offsets
Offsets have to be chosen such that the maximum CPU usage per time slot is minimized, and further,
the runnables respect their period the runnables respect their time slot the runnables satisfy their synchronization constraints
44
45
Meta$heurisIc$search$$algorithms$
Case Study: an automotive software system with 430 runnables
Running the system without offsets
Simulation for the runnables in our case study andcorresponding to the lowest max CPU usage found by HC
5.34 ms
Optimized offset assignment
2.13 ms
- The objective function is the max CPU usage of a 2s-simulation of runnables
- The search modifies one offset at a time, and updates other offsets only if timing constraints are violated
- Single-state search algorithms for discrete spaces (HC, Tabu)
45
46
Comparing$different$search$algorithms$$
(ms)
(s)
Best CPU usage
Time to find Best CPU usage
46
47
Conclusions$
- Though schedulability analysis is a well developed field, the problem we addressed was never defined in those terms (context)
- Search algorithms to compute offset values that reduce the max CPU time needed
- Positive evaluation results: Quick and significant differences
- Huge search space with constraints - Current: Accounting for task time
coupling constraints with multi-objective search " trade-off between relaxing coupling constraints and maximum CPU time
47
What Have I Learned?
Successful Research Patterns
• Successful: Innovative and high impact
• Inductive research: Working from specific observations in real settings to broader generalizations and theories
– Software development is highly diverse
– Problems and solutions are diverse too
– Context factors matter a great deal
– Generalization: Field studies and replications, analyze commonalities
• Scalability and practicality considerations must be part of the initial research problem definition
• Researching by doing: Hands-on research. Apply what exists in well defined, realistic context, with clear objectives. The observed limitations become the research objectives. Put new technology to actual use.
• Interdisciplinary: CS, Mathematics, Engineering, or non-technical domains
49
So What?
• Making a conscious effort to understand the problem first
– A careful investigation often leads to surprises and a very different understanding of the issues at stake
– Precisely identify the requirements for an applicable solution
– More papers focused on understanding the problems
– Making experience/application tracks first class citizens in SE conferences
50
So What? - cont’d
• Better relationships between academia and industry
– Different models • Fraunhofer centres, Germany
• Research-based innovation centers in Norway
• SnT centre in Luxembourg
• Targeted grant programs
• Joint industry-academia labs (e.g., NASA SEL Lab) – Mutually beneficial setting where software development is an
object of study
– Exposing PhD students to industry practice, management and leadership: Ethical considerations (“Fix the PhD”, Nature, 2011)
– Incentives for SE academics 51
So What? - Cont’d
• Work on end-to-end solutions: Pieces of solutions are interdependent. Necessary for impact, e.g., requirements and testing.
• Beyond professors and students – Labs with interdisciplinary teams of professional scientists
and engineers within or collaborating with universities
– Used to be the case with corporate research labs: Bell Labs, Xerox PARC, HP labs, NASA SEL, etc.
– Now: Fraunhofer (Germany), Simula (Norway), Microsoft Research (US), SEI (US), SnT (Luxembourg), FBK (Italy)
– Corporate labs versus publicly supported ones?
– Key point: The level of basic funding must allow high risk and rigorous research, performed by professional scientists, focused on impact in society (Use-inspired basic research)
52
The “Classical” Model of Research and Innovation
Basic Research
Applied Research
Innovation and
Development
An Effective, Collaborative Model of Research and Innovation
Basic Research Applied
Research
Innovation & Development
• Basic and applied research take place in a rich context • Basic Research is also driven by problems raised by
applied research • Main motivation for SnT’s partnership program
B. Schneiderman, The Atlantic, Toward an Ecological Model of Research and Development, 2013
Challenges
Academic Challenges
• Our CS legacy … emancipating ourselves as an engineering discipline
– Electrical engineering: Subfield of Physics until late 19th century
– Software engineering departments?
• How cool is it? SE research is more driven by “fashion” than needs, a quest for silver bullets
– We can only blame ourselves
• Counting papers and how rankings do not help
– We are pressuring ourselves into irrelevance
• Taking academic tenure and promotion seriously
– What about rewarding impact?
• One’s research must cover a broader ground and be somewhat opportunistic – this pushes us out of our comfort zone
• Resources to support industry collaborations
– Large lab infrastructure, research engineers, time 56
Industrial Challenges
• Distinguish the manifestations of a problem from its causes is not easy in practice
• Short term versus longer term goals (next quarter’s forecast is sometimes the priority)
• Industrial research groups are often disconnected from their own business units and external researchers may be perceived as competitors
• Company’s intellectual property regulations may conflict with those of the research institution
• Complexity of industrial systems and technology
– Cannot be transplanted in artificial settings for research - Need studies in real settings
– Substantial domain knowledge is required
57
A Double-Agent Life
58
Scientist, inquisitive but discrete
Warning: No research here
A new idea (as initially perceived by our partners)
Practitioner (anonymous)
Conclusions
• Software engineering is obviously important in all aspects of society, but academic software engineering research is not always perceived the same way
• The academic community, at various levels and in particular its leadership, is partly responsible for this
• How we take up the challenge of increasing our impact will determine the future of the profession
• There is some limited progress, but far too slow
• There are solutions, but no silver bullet
• We all have a role to play in this, as deans, department chairs, professors, scientists, reviewers, conference organizers, journal editors, etc. We can all be double-agents …
59
60 96 IEEE SOFTWARE | PUBLISHED BY THE IEEE COMPUTER SOCIET Y 074 0 -74 5 9 /12 / $ 31. 0 0 © 2 012 I E E E
SOUNDING BOARD
continued on p. 93
Editor: Philippe KruchtenUniversity of British Columbia [email protected]
Embracing the Engineering Side of Software EngineeringLionel Briand
I HAVE NOW been a professional researcher in software engineering for roughly 20 years. Throughout that time, I’ve worked at univer-sities and in research institutes and collabo-rated on research projects with 30-odd pri-vate companies and public institutions. Over the years, I have increasingly questioned and re! ected on the impact and usefulness of my research work and, as a result, made it a pri-ority to combine my research with a genu-ine involvement in actual engineering prob-lems. This short piece aims to re! ect on my experiences in performing industry-relevant software engineering research across several countries and institutions.
Not So Hot AnymoreI suppose a logical start for this article is to assess, albeit concisely, the current state of software engineering research. As software engineering is widely taught in many univer-sities, due in large part to a strong demand for software engineers in industry, the num-ber of software engineering academics is sub-stantial. The Journal of Systems and Soft-ware ranks researchers every year, usually accounting for roughly 4,000 individuals ac-tively publishing in major journals.
When I started my career, software en-gineering was de" nitely a hot topic in aca-demia: funding was plentiful, and universi-ties and research institutes were hiring in record numbers. This clearly isn’t the case anymore. Public funding for software engi-neering research has at best stagnated, and in many countries, declined signi" cantly.
Hiring for research positions is limited and falls far below the number of software engi-neering graduates seeking research careers. Industry attendance at scienti" c software engineering conferences is roughly 10 per-cent, including the scientists from corporate research centers. Adding insult to injury, in many academic and industry circles, soft-ware engineering research isn’t even consid-ered to be a real scienti" c discipline. I’ll spare you the numerous unpleasant comments about the credibility and scienti" c underpin-ning of software engineering research that I’ve heard over the years.
This situation isn’t due to the subject mat-ter’s lack of relevance. Software systems are pervasive in all industry sectors and have be-come increasingly complex and critical. The software engineering profession repeatedly tops job-ranking surveys. In many cases, most of a product’s innovation lies in its software components—for an example, think of the automotive industry. In all my recent industry collaborations, I’ve observed that all the is-sues and challenges traditionally faced in soft-ware development are becoming more acute.
So how can we explain the paradox of be-ing both highly relevant and increasingly un-derfunded and discredited?
Looking for Some AnswersLike other disciplines before us, because we’re a young and still-maturing engineer-ing " eld, we lack the credibility of more
Empirical Software Engineering
• Springer, 6 issues a year
• Both research papers and industry experience reports
• High impact factor among SE research journals
• “Applied software engineering research with a significant empirical component”
61
Personal References
• Hemmati, Briand, Arcuri, Ali “An Enhanced Test Case Selection Approach for Model-Based Testing: An Industrial Case Study”, ACM FSE, 2010
• Frouchni, Briand, Labiche, Grady, and Subramanyan, “Automating Image Segmentation Verification and Validation by Learning Test Oracles”, Information and Software Technology (Elsevier), 2011.
• Sabetzadeh, Nejati, Briand, Evensen Mills “Using SysML for Modeling of Safety-Critical Software–Hardware Interfaces: Guidelines and Industry Experience”, HASE, 2011
• Sabetzadeh et al., “Combining Goal Models, Expert Elicitation, and Probabilistic Simulation for Qualification of New Technology”, HASE, 2011
• Ali, Briand, Hemmati, “Modeling Robustness Behavior Using Aspect-Oriented Modeling to Support Robustness Testing of Industrial Systems”, Journal of Software and Systems Modeling (Springer), 2011.
• Behjati, Nejati, Yue, Gotlieb, Briand, “Model-based Automated and Guided Configuration of Embedded Software Systems”, ECMFA, 2012.
• Behjati, Yue, Briand, Selic. SimPL, “A Product-Line Modeling Methodology for Families of Integrated Control Systems, Information and Software Technology”, Information and Software Technology (Elsevier), 2012.
• Nejati et al., A SysML-Based Approach to Traceability Management and Design Slicing in Support of Safety Certification: Framework, Tool Support, and Case Studies, Information and Software Technology (Elsevier), 2012
• Iqbal, Arcuri, Briand, “Empirical Investigation of Search Algorithms for Environment Model-Based Testing of Real-Time Embedded Software”, ACM ISSTA, 2012
62
Personal References II
• Nejati, Di Alesio, Sabetzadeh, Briand, “Modeling and Analysis of CPU Usage in Safety-Critical Embedded Systems to Support Stress Testing”, ACM/IEEE MODELS 2012
• Nejati, Sabetzadeh, Falessi, Briand, Coq, “A SysML-based approach to traceability management and design slicing in support of safety certification: Framework, tool support, and case studies”, Information & Software Technology, (Elsevier), 2012
• Hemmati, Arcuri, L. Briand: Achieving scalable model-based testing through test case diversity. ACM Trans. Softw. Eng. Methodology, 2013.
• Panesar-Walawege, Sabetzadeh, Briand: Supporting the verification of compliance to safety standards via model-driven engineering: Approach, tool-support and empirical validation. Information & Software Technology (Elsevier), 2013
• S. Nejati et al., “Minimizing CPU Time Shortage Risks in Integrated Embedded Software”, 28th IEEE/ACM International Conference on Automated Software Engineering, 2013
• Sabetzadeh, Falessi, Briand, Di Alesio: A goal-based approach for qualification of new technologies: Foundations, tool support, and industrial validation. Reliability Eng. & System Safety, 2013
• Briand et al., “Traceability and SysML Design Slices to Support Safety Inspections: A Controlled Experiment”, forthcoming in ACM Transactions on Software Engineering and Methodology, 2013
63
Other References
• Bertrand Meyer’s blog: http://bertrandmeyer.com/2010/04/25/the-other-impediment-to-software-engineering-research/
• Basili, “Learning Through Application: The maturing of the QIP in the SEL”, Making Software; What really works and why we believe it, Edited by Andy Oram and Greg Wilson, O’Reilly Publishers, 2011, pp.65-78.
• T. Gorschek, P. Garre, S. Larsson, C. Wohlin. A Model for Technology Transfer in Practice. IEEE Software 23(6), 2006.
• Parnin and Orso, “Are Automated Debugging Techniques Actually Helping Programmers?”, ISSTA, 2011
64