ISERN Survey & Benchmark10th anniversary meta-experiment project

Session Chair, Stefan Biffl

Marcus Ciolkowski, Forrest Shull, and Dieter Rombach

1. Strategy

2. Organization

3. Survey Design

4. Benchmark Design

5. Next Steps

contact/info: ciolkows@informatik.uni-kl.de

Meta-experiment Goal

• ISERN develops empirical methodology, i.e. how to use experiments (case studies, controlled experiments).

• 1992: What is an experiment?• 2000: Many experiments conducted• 2002: How to conduct a meta-experiment; Data

Collection from the distributed individual surveys and benchmarks; Presentation of initial results, Meta-analyis with historic experiments

Research Problem: History

• ISERN has successful history on experiments with inspections, but mostly in academic environments– Variations in replications not systematically recorded:

Make best use of existing information

– Influence factors for inspection from an industry point of view should be elicited

• Methodic organization– How to acquire knowledge

– Demonstrate added value

ISERN Inspection Survey & Benchmark

• Common empirical study in ISERN– One large survey study

– Several options for inspection benchmarks

– All interested ISERN-organizations collaborate

• Run a series of small but related studies – Each will provide interesting results for the host site

– In the aggregate, a chance to compare the impact of various environmental variables on the results.

– Also include historic experiments.

Benefit

• Common ISERN experiment – Pragmatic 1st round – low risk context;

reasonable return on investment (ROI)

• Low risk/cost for software inspection due to reuse– Experimental framework (baseline document,

baseline systematic inspection technique)– Package with baseline inspection materials

• Reasonable ROI due to scalable investment– Allows organizations to commit to various levels of

effort, based on their constraints

Opportunities through ISERN Survey and Benchmark

• Work with industry as well as universities• Benefit for industry:

– Learn: under which constraints are results (positive trends) transferable?

– Study: Demonstrate benefit and gain knowledge not by chance, but based on testable hypotheses (study design).

– QIP for experiments: Maximize benefits from experiments.– Industry can introduce own questions and own factors– External experience often basis for introduction of techniques– Measurement data important for regular feedback on and

reinforcement of successful techniques in practice.

Organization: Roles in the Experiment

(Meta-)DataAnalysis

Committee

ISERN Exp.Steering

Committee

ExperimentPrepariationCommittee

Contact PersonOrg. Unit

PromoterOrganizational

Unit

Local ResearcherLibrarian

Doc/Data Server

ISERNExperimentCoordinator

QA for DataRepository

InspectorLocal ResearchSupport Staff

ISERN SupportISERN Strategy Local Research Organization

Local ScientificPromoter

Organization: People for roles

• Steering Committee: contributors to experiment• Experiment Preparation Committee: authors of

experiment documentation• Experiment Coordinator: knows experiment status• Librarian: administers experiment documentation • QA for Data Repository: checks incoming data• Meta-analysis Committee: plans data analysis

Working Group Inspection Survey

• Goal: Overview on state-of-the-practice in inspections in industry

• Characterize documents and defects in practice • Measure: How well does reading technique fit to

document/defect profile/inspector skills?

• So far: studies/questionnaires from Lund, Sweden• Working group chair: Ross Jeffery

Working Group Experiment Design

• Goal: Create experimental design and materials for distributed experiment

• Provide standard inspection techniques and baseline documents for inspection

• So far: three levels of commitment

• Working group chair: Forrest Shull, FC MD

ISERN Experiment Support

• Preparation– Baseline document and inspection material

– Training material

• Conduct– Guidance through local researcher

• Data Analysis and Presentation– Data collection (guaranty of anonymity)

– Evaluation and customized feedback to organization

Next Steps

• Negotiation and Agreement on Procedure• Commitment to conduct studies• Milestones for individual study

– Contact to organization

– Negotiate options

– Conduct study

– Data collection in ISERN repository

• Data meta-analysis (including historical data)

Questions & Open Issues

• Clear, can be discussed now:

• Unclear, should be discussed off-line:

• Working groups to resolve unclear issues into a set of clear options for participants.

Sign-up for participation

• Local researcher: ____________________• Contact: ___________________________• Max. number of partners ...:

– ... for survey:___– ... for study of standard process:___– ... for semi-controlled experiment:___

• Opportunities for collaboration (shadow experiments): _____________________