Supporting Continuous Integration by Mashing-Up Software Quality Information

software evolution & architecture lab

Department of Informatics

Martin Brandtner, Emanuel Giger, and Harald Gall University of Zurich, Switzerland http://seal.ifi.uzh.ch IEEE  CSMR-­‐WCRE,  Antwerp,  Belgium,  February  3-­‐6,  2014  

Continuous Integration (CI)

“[...] you want to ensure that everyone can easily see the state of the system and the changes that have been made to it.”

Martin Fowler

Martin Brandtner - IEEE CSMR-WCRE 2014 1

CI – Stakeholders, State & Changes

Martin Brandtner - IEEE CSMR-WCRE 2014 2

Dave  (Developer)  

Trinity  (Tester)  

So3ware  System  

State:  The  source  code  compiles  without  warnings.  

State:  Every  integra>on  test  passed  successfully.  

Changes:  Focus  on  the  source  code  of  the  system.  

Changes:  Focus  on  the  source  code  of  the  test  cases.  

CI – Toolchain

Commits  +  Issues  

Build  informa>on  

Quality  metrics  

Martin Brandtner - IEEE CSMR-WCRE 2014 3

Research Question

How do stakeholders perform in answering questions about software quality with an integrated approach, such as SQA-Mashup, compared to the use of standalone CI-tools?

Martin Brandtner - IEEE CSMR-WCRE 2014 4

SQA-Mashup – View – Developer

Martin Brandtner - IEEE CSMR-WCRE 2014 5

SQA-Mashup – View – Tester

Martin Brandtner - IEEE CSMR-WCRE 2014 6

SQA-Mashup – Timeline

Martin Brandtner - IEEE CSMR-WCRE 2014 7

SQA-Mashup – Pipe Configuration

Martin Brandtner - IEEE CSMR-WCRE 2014 8

SQA-Mashup – Pipe

Action 1 – Fetch

–  Endpoint Handler –  URL to RESTful Service

Action 2 – Format (List) –  Mapping

Martin Brandtner - IEEE CSMR-WCRE 2014 9

SQA-Mashup – User interface

Martin Brandtner - IEEE CSMR-WCRE 2014 10

Controlled user study – Setting JUnit project on GitHub

16 participants (8 control, 8 experimental) –  14 advanced graduate students,

work at least part-time in industry –  1 PhD + 1 PostDoc –  Group assignment: random

Control group: GitHub, Jenkins-CI, SonarQube Experimental group: SQA-Mashup

Martin Brandtner - IEEE CSMR-WCRE 2014 11

Controlled user study – Tasks Total: 9 tasks (3 tasks in each category)

Task categories: –  Program comprehension (e.g. software evolution) –  Testing (e.g. testing coverage) –  Cross-Domain (e.g. time-period related tasks)

Preserve in structure and semantics

1.  What have my coworkers been doing? * 2.  What have my coworkers been doing between 2013/07/16 and

2013/08/01?

Martin Brandtner - IEEE CSMR-WCRE 2014 12

*  T.  Fritz  and  G.  C.  Murphy,  “Using  informaNon  fragments  to  answer  the  quesNons  developers  ask,”  ser.  ICSE,  Cape  Town,  2010,  pp.  175–184.  

Controlled user study – Hypotheses

H10: There is no difference in the total score per subjects between the experimental and the control group. H20: There is no difference in the total time per subjects between the experimental and the control group.

Martin Brandtner - IEEE CSMR-WCRE 2014 13

Controlled user study – Results

(sec)  

Martin Brandtner - IEEE CSMR-WCRE 2014 14

Task Analysis – Score Where did the advancements come from? Total score in all other tasks: Experimental group at least as good as control group

Martin Brandtner - IEEE CSMR-WCRE 2014 15

Experimental

 group:  sig

nificantly  h

igher  

score  compared

 to  the  con

trol  group  

 

Where did the advancements come from? Total time in all other tasks: Experimental group at least as good as control group

Task Analysis – Time

Martin Brandtner - IEEE CSMR-WCRE 2014 16

Experimental

 group:  sig

nificantly  

faster  compared

 to  the  con

trol  group  

 

The results of a user study showed that participants solved tasks with 21.6% higher correctness and 57% faster with SQA-Mashup compared to standalone CI-tools.

Martin Brandtner - IEEE CSMR-WCRE 2014 17

hHp://www.ifi.uzh.ch/seal/research/tools/sqa-­‐mashup.html