Automated Diagnosis of Feature Model Configurations

21
Automated Diagnosis of Feature Model Configurations J. White ,a , D. Benavides b , D.C. Schmidt a , P. Trinidad b , B. Dougherty a , A. Ruiz-Cortes b a Dept. of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN 37204, USA b Dept. of Computer Languages and Systems University of Seville, Avda. de la Reina Mercedes, s/n B-41012 Seville, Spain Abstract Software product-lines (SPLs) are software architectures that can be readily reconfigured for different project requirements. A key part of an SPL is a model that captures the rules for reconfiguring the software. SPLs commonly use feature models to capture SPL configuration rules. Each SPL configuration is represented as a selection of features from the feature model. Invalid SPL configurations can be created due to feature conflicts introduced via staged or parallel configuration or changes to the constraints in a feature model. When invalid configurations are created, a method is needed to automate the diagnosis of the errors and repair the feature selections. This paper provides two contributions to research on automated configuration of SPLs. First, it shows how configurations and feature models can be transformed into constraint satisfaction problems to automatically diagnose errors and repair invalid feature selections. Second, it presents empirical results from diagnosing configuration errors in feature models ranging in size from 100 to 5,000 features. The results of our experiments show that our CSP-based diagnostic technique can scale up to models with thousands of features. Key words: software product-lines, configuration, diagnosis, constraint satisfiaction, optimization 1. Introduction Background and challenges. Software product-lines (SPLs) are software architectures that can be reconfigured for different requirement sets (Clements and Northrop, 2002). SPLs are designed to facilitate the reuse of a software prod- uct across multiple projects and amortize devel- opment cost. For example, in the automotive domain, an SPL can be created that allows a car’s software to provide Anti-lock Braking Sys- tem (ABS) capabilities or simply standard brak- ing, thus allowing it to be used in cars aimed at * Corresponding author Email addresses: [email protected] (J. White), [email protected] (D. Benavides), [email protected] (D.C. Schmidt), [email protected] (P. Trinidad), [email protected] (B. Dougherty), [email protected] (A. Ruiz-Cortes) different price points. Each unique configuration of an SPL is called a variant. A core component of an SPL is a model of the points of variability within the software architec- ture. These variability points determine how the software can be reconfigured to meet new require- ment sets. Points of variability are typically not independent of one another, however, and thus an explicit set of rules must be developed to dictate legal combinations of configuration settings e.g.,a car cannot have both ABS and standard braking software controllers. Feature modeling (Kang et al., 1998) is a tech- nique for documenting the points of variability in an SPL, how the points of variability affect one another, and what constitutes a complete config- uration of the SPL. A feature model leverages fea- tures as the abstraction for documenting config- uration rules. Each feature represents an incre- Preprint submitted to Software and Systems February 10, 2009

Transcript of Automated Diagnosis of Feature Model Configurations

Page 1: Automated Diagnosis of Feature Model Configurations

Automated Diagnosis of Feature Model Configurations

J. White∗,a, D. Benavidesb, D.C. Schmidta, P. Trinidadb, B. Doughertya, A. Ruiz-Cortesb

aDept. of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN 37204, USAbDept. of Computer Languages and Systems University of Seville, Avda. de la Reina Mercedes, s/n B-41012 Seville,

Spain

Abstract

Software product-lines (SPLs) are software architectures that can be readily reconfigured for differentproject requirements. A key part of an SPL is a model that captures the rules for reconfiguringthe software. SPLs commonly use feature models to capture SPL configuration rules. Each SPLconfiguration is represented as a selection of features from the feature model. Invalid SPL configurationscan be created due to feature conflicts introduced via staged or parallel configuration or changes tothe constraints in a feature model. When invalid configurations are created, a method is needed toautomate the diagnosis of the errors and repair the feature selections.

This paper provides two contributions to research on automated configuration of SPLs. First, itshows how configurations and feature models can be transformed into constraint satisfaction problemsto automatically diagnose errors and repair invalid feature selections. Second, it presents empiricalresults from diagnosing configuration errors in feature models ranging in size from 100 to 5,000 features.The results of our experiments show that our CSP-based diagnostic technique can scale up to modelswith thousands of features.

Key words: software product-lines, configuration, diagnosis, constraint satisfiaction, optimization

1. Introduction

Background and challenges. Softwareproduct-lines (SPLs) are software architecturesthat can be reconfigured for different requirementsets (Clements and Northrop, 2002). SPLs aredesigned to facilitate the reuse of a software prod-uct across multiple projects and amortize devel-opment cost. For example, in the automotivedomain, an SPL can be created that allows acar’s software to provide Anti-lock Braking Sys-tem (ABS) capabilities or simply standard brak-ing, thus allowing it to be used in cars aimed at

∗Corresponding authorEmail addresses: [email protected] (J.

White), [email protected] (D. Benavides),[email protected] (D.C. Schmidt),[email protected] (P. Trinidad),[email protected] (B. Dougherty),[email protected] (A. Ruiz-Cortes)

different price points. Each unique configurationof an SPL is called a variant.

A core component of an SPL is a model of thepoints of variability within the software architec-ture. These variability points determine how thesoftware can be reconfigured to meet new require-ment sets. Points of variability are typically notindependent of one another, however, and thus anexplicit set of rules must be developed to dictatelegal combinations of configuration settings e.g., acar cannot have both ABS and standard brakingsoftware controllers.

Feature modeling (Kang et al., 1998) is a tech-nique for documenting the points of variability inan SPL, how the points of variability affect oneanother, and what constitutes a complete config-uration of the SPL. A feature model leverages fea-tures as the abstraction for documenting config-uration rules. Each feature represents an incre-

Preprint submitted to Software and Systems February 10, 2009

Page 2: Automated Diagnosis of Feature Model Configurations

ment in product functionality. A feature modelcan capture different types of variability, rangingfrom SPL variability (e.g., variations in customerrequirements) to software variability (e.g., varia-tions in software implementation) (Metzger et al.,2007).

Unique configurations of an SPL are capturedas selections of features from a feature model.SPL variants can be specified as a selection orconfiguration of features. Feature models of SPLsare arranged in a tree-like structure where eachsuccessively deeper level in the tree correspondsto a more fine-grained configuration option forthe product-line variant, as shown by the featuremodel in Figure 1. The parent-child and cross-tree relationships capture the constraints thatmust be adhered to when selecting a group of fea-tures for a variant.

Existing SPL research has focused on ensur-ing that features chosen from feature models arecorrect and consistent with the SPL and vari-ant requirements. For example, boolean cir-cuit satisfiability techniques (Mannion, 2002) andConstraint Satisfaction Problems (CSPs) (Be-navides et al., 2005; White et al., 2007) havebeen used to automate the derivation of a fea-ture set that meets a requirement set. Numer-ous tools have also been developed, such as BigLever Software Gears (Buhrdorf et al., 2003),Pure::variants (Beuche, 2003), FeAture ModelAnalyser (FAMA) (Benavides et al., 2007), andthe Feature Model Plug-in (Czarnecki et al.,2005a), to support the construction of featuremodels and correct selection of feature configu-rations.

Although SPL developers are equipped witha number of tools and techniques, configurationproblems may occur due to human error or otherfactors. For example, configurations of large fea-ture models may be produced using staged con-figuration (Czarnecki et al., 2004, 2005b), wherefeatures are iteratively selected by a group of par-ticipants. At each stage, one participant choosesfeatures before passing the partial configurationon to the participant in the next stage. In thesestaged configuration processes, a participant mayselect a feature that conflicts with a critical fea-

ture needed by a participant in a later stage. Dueto the complexity of feature model constraints itmay be hard to forsee these conflicts. It is alsohard to debug a configuration to figure out howto change decisions in previous stages to make thecritical feature selectable (Batory et al., 2006).

In many cases—particularly when SPLs areproduced from supply-chains—the configurationdecisions of multiple developers working in paral-lel must be integrated. Since developers may workin different departments, organizations, or geo-graphic locations configurations may be producedwith conflicting feature selections. For example,hardware developers for an automobile may de-sire a lower cost set of Electronic Control Units(ECUs) that cannot support the embedded con-troller code features needed by software develop-ers. Methods are therefore needed to (1) evaluateand debug conflicts between participants and (2)recommend modifications to the participants fea-ture selections to make them compatible.

A further challenge to maintaining error freeSPL configurations is that external influences(such as supply-chain disruption, changes in mar-ket demand, or safety regulations) may changeover time and thus invalidate previously correctconfigurations. For example, if a new governmentregulation is passed that mandates higher emis-sions standards, it may necessitate changing exist-ing software configurations. A critical question isthus often determining the lowest cost method ofupdating existing software configurations to meetthe new external regulations and requirements,which is a process termed software configurationevolution.

Although prior research has shown how to iden-tify flawed configurations (Batory, 2005; Man-nion, 2002), conventional debugging mechanismscannot pinpoint configuration errors and identi-fying corrective actions. More specifically, tech-niques needed that can take an arbitrary flawedconfiguration and produce the minimal set of fea-ture selections and deselections to bring the con-figuration to a valid state. This paper focuses onaddressing these gaps in existing research.

Solution overview and contributions. Ourapproach to debugging feature model configu-

2

Page 3: Automated Diagnosis of Feature Model Configurations

rations and automating configuration evolutiontransforms an invalid feature model configu-ration into a Constraint Satisfaction Problem(CSP) (Van Hentenryck, 1989) and then usesa constraint solver to derive the minimal setof feature selection modifications that will bringthe configuration to a valid state. We call thisconstraint-based diagnostic approach the Con-figuration Understanding and REmedy (CURE).This paper shows how CURE provides the follow-ing contributions to work on debugging errors infeature model configurations:

1. We provide a CSP-based diagnostic tech-nique, inspired by (Trinidad et al., 2007) thatcan pinpoint conflicts and constraint viola-tions in feature models

2. We provide a simplified diagnostic CSP thatis not as flexible as the full diagnostic CSPbut reduces diagnosis time

3. We show how CURE can remedy a config-uration error by automatically deriving theminimal set of features to select and deselect

4. We provide mechanisms for using CURE tomediate conflicting configuration participantfeature selection desires via cost optimizationtechniques

5. We show how CURE allows stakeholders todebug a configuration error or conflict fromdifferent viewpoints

6. We show how CURE can be used to automatesoftware evolution

7. We provide empirical results showing thatCURE’s scalability can support industrialSPL feature models containing over 5,000features.

8. We present empirical results showing the im-provement in solving time that the simplifiedCSP provides

This paper extends our previous work on auto-mated diagnosis of product-line configuration er-rors (White et al., 2008) in three ways. First,we provide a new methodology for applying ourCURE diagnosis technique to automating the evo-lution of software configurations, which allowsSPL configurations to evolved automatically over

time to meet changing external requirement sets.Second, we provide a new simplified diagnosticCSP formulation that can significantly reducesolving time for large feature models and presentnew empirical results demonstrating the improve-ment in solving time provided by the new CSP for-mulation. Third, we have significantly expandedthe discussion of CURE’s relationship with re-lated work on SPL configuration.

Paper organization. The remainder of thepaper is organized as follows: Section 2 describesthe challenges of diagnosing configuration errorsand conflicts in SPLs; Section 3 presents theCURE CSP-based technique for diagnosing con-figuration errors and conflicts; Section 4 showshow we extended CURE to support conflict me-diation, multi-viewpoint debugging, and faster di-agnosis times; Section 5 shows how CURE canbe used to automate software configuration evo-lution decisions; Section 6 presents empirical re-sults demonstrating the ability of CURE to scaleto feature models with thousands of features; Sec-tion 7 compares CURE with related research; andSection 8 presents concluding remarks.

2. Challenges of Diagnosing Configuration

Errors and Conflicts in SPLs

This section evaluates five key challenges thatarise in realistic SPL configuration scenarios. Sec-tion 3 describes how CURE addresses these chal-lenges.

2.1. Feature Model Configuration DebuggingChallenges

2.1.1. Challenge 1: Staged Configuration Errors

Staged configuration is a configuration processwhere developers iteratively select features to re-duce the variability in a feature model until a vari-ant is constructed. The need for staged configu-ration has been demonstrated (Czarnecki et al.,2004, 2005b) in the context of software supplychains for embedded automotive software. In thefirst stage, software vendors provide software com-ponents that can be provided in different config-urations to actuate brakes, control infotainment

3

Page 4: Automated Diagnosis of Feature Model Configurations

systems, etc. In the second stage, hardware ven-dors that the software runs on must provide ECUswith the correct features and configuration to sup-port the software components selected in the firststage.

The challenge with staged configuration is thatfeature selection decisions made at some point intime T have ramifications on the decisions madeat all points in time T ′ > T . For example, it ispossible for software vendors to choose a set ofsoftware component features for which there areno valid ECU configurations in the second config-uration stage. Identifying the fewest number ofconfiguration modifications to remedy the erroris hard because there can be significant distancebetween T and T ′.

This challenge also appears in larger models,such as those for software to control the au-tomation of continuous casting in steel manufac-ture (Rabiser et al., 2007). In large-scale models,configuration mimics staged configuration sincedevelopers cannot immediately understand theramifications of their current decisions. At somelater decision point, critical features that develop-ers need may no longer be selectable due to someprevious choice. Again, it is hard to identify theminimal set of configuration decisions to reversein this scenario. Section 4.2 describes how CUREaddresses this challenge by using the CSP to de-rive a configuration that includes the needed fea-ture and reverses configuration stages from previ-ous stages.

2.1.2. Challenge 2: Mediating Conflicts

In many situations the desired features andneeds of multiple stakeholders involved in config-uring an SPL variant may conflict. For example,when configuring automotive systems, softwaredevelopers may want a series of software com-ponent configurations that cannot be supportedby the ECU configurations proposed by hardwaredevelopers. Each party typically believes their in-dividual needs are critical, so it is hard to find themiddle ground to integrate the two.

Another conflict scenario arises when configu-ration decisions made for an SPL variant must bereconciled with constraints of the legacy environ-

ment in which it will run. For example, whenconfiguring automotive software for next year’scar model, a variant may initially be configuredto provide the most desired customer features,such as digital infotainment. New model cars arerarely complete redesigns, however, so developersmust determine how to run new software configu-rations on existing ECU configurations from pre-vious models. If the new software configurationis not compatible with the legacy ECU configu-ration, developers must derive the lowest cost setof modifications to either the new software or thelegacy ECU configuration. Section 4.3 describeshow CURE addresses this challenge by diagnosingthe superset of the desired conflicts and leveragingan alternate CSP optimization goal.

2.1.3. Challenge 3: Viewpoint-dependent Errors

The feature labeled as the source of an error ina feature model configuration may vary depend-ing on the viewpoint used to debug it. For exam-ple, if a configuration in the feature model shownin Figure 1 is created to include both Non-ABSController and 1 Mbit/s CAN Bus, either featurecan be viewed as the feature that is the sourceof the error. If we debug the configuration from

Figure 1: Simple Feature Model for an Automobile

the viewpoint that software features trump ECUhardware decisions, then the 1 Mbit/s CAN Busfeature is the error. If we assume that ECU deci-sions precede software features, however, then theNon-ABS Controller feature is the error.

A feature model may therefore require debug-ging from multiple viewpoints since diagnosingthe feature that causes an error in a feature modeldepends on the viewpoint used to debug it. Forsmall feature models, debugging from differentviewpoints is relatively simple. As feature mod-els grow to contain hundreds or thousands of fea-

4

Page 5: Automated Diagnosis of Feature Model Configurations

tures, the complexity of diagnosing a configura-tion from multiple viewpoints increases greatly.Section 4.2 describes how CURE addresses thischallenge by specifying feature selections thatcannot be modified by the solver during diagnosis.

2.2. Software Configuration Evolution Challenges

Software configuration evolution is a criticalpart of SPL management that involves modifyingan existing software configuration to meet newrequirements. Although building a variant for anew requirement set is a standard feature modelconfiguration problem, it can not always be per-formed without taking into consideration existingconfigurations. For example, if a key supply-chainvendor for an automobile goes out of business, au-tomotive manufacturers must account for the con-figuration of existing model year cars when deter-mining how to produce new configurations thateliminate the parts produced by the defunct ven-dor. Simply producing new configurations withno regard for the existing manufacturing, procure-ment, and other processes investments would becostly and error-prone.

The goal of configuration evolution is to trans-form a current configuration A that satisfies a re-quirement set R into a new configuration A′ thatsatisfies an evolved requirement set R′. Moreover,because there may be a substantial existing in-vestment in A, a goal of the evolution process isto find A′, such that it minimizes or maximizes afunction, ∆(A, A′). For example, developers maywish to choose evolution paths that require addingas little additional cost to A as possible and thus∆(A, A′) computes the difference in cost betweenthe legacy and evolved configurations.

2.2.1. Challenge 4: Supply-chain Disruption

The features that are present in an SPL’s fea-ture model are dependent on an organization’sability to procure specific sets of software andhardware components. For example, the produc-tion of an automobile may depend on vendorsproviding the ECU and associated software for ahigh-end infotainment system. If any vendors goout of business, decide to no longer produce oneof the requisite parts, or cannot produce sufficient

quantities of a part, SPL developers must deter-mine how to evolve the automobile’s configurationto eliminate the unavailable part.

When a part becomes unavailable, developersfirst need to determine which existing features orcombinations of features in the feature model areno longer viable selections due to the unavailableparts. In a simple scenario, a single feature mayno longer be available. For example, in the featuremodel shown in Figure 3, if the ECU requiredfor the Advanced Infotianment feature becomesunavailable, this feature will be invalid in a featureselection.

In more complicated situations, alternativeparts may be available from other vendors, butthese new parts may have different requirementsthan the old parts. For example, in the scenariodepicted in Figure 2 the ECU used for an au-tomobile’s infotainment system becomes unavail-able and an alternative ECU must be used. The

Figure 3: Supply-chain Disruption Leading to Feature Un-availability

new ECU has more substantial peak power re-quirements, however, which requires adding a newrelationship to the feature model. When supply-chain disruption occurs, developers must deter-mine cost-effective ways of evolving legacy config-urations to meet the new requirements.

A related challenge for configuration diagnosisis the ability to take a current configuration thathas been rendered obsolete via supply-chain dis-

5

Page 6: Automated Diagnosis of Feature Model Configurations

Figure 2: Supply-chain Disruption Leading to Feature Model Rule Changes

ruptions and recommend a series of modificationsto make it valid. For example, a configuration di-agnosis technique should be able to recommendhow to replace the features affected by the un-availability of a key implementation component.Section 5.1 describes how CURE addresses thischallenge by allowing modeling configuration evo-lution problems as configuration diagnosis prob-lems.

2.2.2. Challenge 5: Software Feature Expansion

Another important evolution problem thatshould have automation support is planning fea-ture expansion. Feature expansion planning is de-termining how to incorporate new features to sat-isfy new requirements, such as changing marketdemands, into existing configurations. For exam-ple, automotive designers may want to incorpo-rate automated parking assistance into the nextyear’s model of a car.

As with supply-chain disruption, developerscannot simply ignore the existing configurationof the current year’s model. Instead, developersneed to determine the minimal or lowest cost setof modifications that can be applied to the cur-rent configuration to enable the newly requiredfeatures. Moreover, developers may have certainhard requirements, such as the high cost of modi-fying a manufacturing process, that mandate cer-tain features in the current configuration not tobe modified. Section 5.2 shows how CURE canhelp analyze different feature expansion plans andgenerate cost-optimal feature expansion plans.

3. Configuration Error Diagnosis with

CURE

Configuration Understanding and REmedy(CURE) is a constraint-based diagnostic ap-proach based on creating automated SPL vari-ant diagnosis tools. Developers can use theCURE tools to identify the minimal set of featuresneeded to select or deselect to transform an invalidconfiguration into a valid configuration. More-over, depending on the input provided to CURE,a flawed configuration can be debugged from dif-ferent viewpoints or conflicts between multiplestakeholder decisions in a configuration processcan be mediated.

The key component of CURE is the applica-tion of a CSP-based error diagnostic technique.In prior work, Benavides et al. (Benavides et al.,2005) have shown how feature models can betransformed into CSPs to automate feature se-lection with a constraint solver (Jaffar and Ma-her, 1994). Trinidad et al. (Trinidad et al., 2007)subsequently described how to extend this CSPtechnique to identify full mandatory features, voidfeatures, and dead feature models using Reiter’stheory of diagnosis (Reiter, 1987). This sectionpresents an alternate diagnostic model for deriv-ing the minimum set of features that should beselected or deselected to eliminate a conflict in afeature configuration.

3.1. Background: Feature Models and Configura-tions as CSPs

A CSP is a set of variables and a set of con-straints over those variables. For example, A +

6

Page 7: Automated Diagnosis of Feature Model Configurations

B ≤ 3 is a CSP involving the integer variables A

and B. A constraint solver finds a valid labeling(set of variable values) that simultaneously satis-fies all constraints in the CSP. (A = 1, B = 2) isthus a valid labeling of the CSP A + B ≤ 3.

To build the CSP for the error diagnosis tech-nique, we construct a set of variables, F , repre-senting the features in the feature model. Eachconfiguration of the feature model is a set of val-ues for these variables, where a value of 1 indicatesthe feature is present in the configuration and avalue of 0 indicates it is not present. More for-mally, a configuration is a labeling of F , such thatfor each variable fi ⊂ F , fi = 1 indicates that theith feature in the feature model is selected in theconfiguration. Correspondingly, fi = 0 impliesthat the feature is not selected.

Given an arbitrary configuration of a featuremodel as a labeling of the F variables, developersneed the ability to ensure the correctness of theconfiguration. To achieve this constraint checkingability, each variable fi is associated with one ormore constraints corresponding to the configura-tion rules in the feature model. For example, if fj

is a required subfeature of fi, then the CSP wouldcontain the constraint: fi = 1 ⇔ fj = 1.

Configuration rules from the feature model arecaptured in the constraint set C. For any givenfeature model configuration described by a label-ing of F , the correctness of the configuration canbe determined by seeing if the labeling satisfiesall constraints in C. More detailed descriptionsof the steps for transforming a feature model to aCSP appear in (Benavides et al., 2005).

3.2. Configuration Diagnostic CSP

This section describes CURE’s diagnostic CSP.First, we introduce the diagnostic recommenda-tion variables that CURE uses to indicate whichfeatures should be selected or deselected to fix aconfiguration. Next, we describe how the diagno-sis of a flawed configuration is modeled as a CSPusing these variables. Finally, we explain how theconstraints on the diagnostic variables produce acorrect diagnosis of an invalid configuration.

3.2.1. Diagnostic Recommendation Variables

When diagnosing configuration conflicts, devel-opers need a list of features that should be se-lected or deselected to make an invalid configura-tion a valid configuration. The output of CUREis this list of features to select and deselect, asshown in Figure 4. In Step 1 of Figure 4, the

Figure 4: Diagnostic Technique Architecture for CURE

rules of the feature model and the current invalidconfiguration are transformed into a CSP. For ex-ample, o1 = 1 because the Automobile feature isselected in the current invalid configuration. InStep 2, the solver derives a labeling of the diag-nostic CSP. Step 3 takes the output of the CSPlabeling and transforms it into a series of recom-mendations of features to select or deselect to turnthe invalid configuration into a valid configura-tion. Finally, in Step 4, the recommendations areapplied to the invalid configuration to create avalid configuration where each variable fi equals1 if the corresponding feature is selected in thenew and valid configuration. For example, f7 = 1means that the 250 Kbit/s CAN Bus is selectedin the new valid configuration.

To enable the constraint solver to recommendfeatures to select and deselect, two new sets of rec-ommendation variables, S and D, are introducedto capture the features that need to be selectedand deselected, respectively, to reach a valid con-

7

Page 8: Automated Diagnosis of Feature Model Configurations

figuration. For example, a value of 1 for vari-able si ⊂ S indicates that the feature fi shouldbe added to the current configuration. Similarly,di = 1 implies that the feature fi should be re-moved from the configuration.

3.2.2. Diagnostic CSP Setup

For each feature fi ⊂ F , there are variablessi ⊂ S and di ⊂ D. After the diagnosis CSP islabeled, the values of S and D serve as the outputrecommendations to the user as to what featuresto add or remove from the current configuration,as shown in Table 1.

This table shows the complete inputs and out-puts to diagnose the invalid configuration scenarioshown in Figure 4.

The next step is to allow developers to inputtheir current configuration into the solver for di-agnosis. Rather than directly setting values forthe variables in F , developers use a special setof input variables called observations, which arecontained in the set of variables O. For each fea-ture fi present in the current flawed configuration,oi = 1; if fi is not selected in the current invalidconfiguration, oi = 0. Table 1 shows how obser-vations capture the current invalid configurationprovided as input to the solver. Observations canalso be made for a correct configuration, in whichcase CURE will state that no changes are needed.The rest of this paper assumes that the observa-tions represent an invalid configuration.

3.2.3. Diagnostic CSP Constraints

To diagnose the CSP, we want to find an alter-nate but valid configuration of the feature modeland suggest a series of changes to the current in-valid configuration to reach the valid configura-tion. A valid configuration is a labeling of thevariables in F (a configuration) such that all ofthe feature model constraints are satisfied. Foreach variable fi, the value should be 1 if the fea-ture is present in the new valid configuration thatwill be transitioned to. If a feature is not in thenew configuration, fi should equal 0.

VariablesVariable Ex-planations

fi ⊂ F : feature variablesfor the valid configuration thatwill be transitioned to; oi ⊂ O:the features selected (oi = 1)in the current invalid configu-ration; si ⊂ S: features to se-lect (si = 1) to reach the validconfiguration; di ⊂ D: fea-tures to deselect (di = 1) toreach the valid configuration

InputsCurrentConfig.

o1 = 1, o2 = 1, o3 = 0, o4 =1, o5 = 1, o6 = 1, o7 = 0

FeatureModel Rules

f1 = 1 ⇔ (f2 = 1), f1 =1 ⇔ (f5 = 1), f2 = 1 ⇒(f3 = 1) ⊕ (f4 = 1), f5 =1 ⇒ (f6 = 1) ⊕ (f7 = 1),(f6 = 1)∨(f7 = 1) ⇒ (f5 = 1),(f3 = 1)∨(f4 = 1) ⇒ (f2 = 1),f3 = 1 ⇒ (f6 = 1), f4 = 1 ⇒(f7 = 1)

DiagnosticRules

(fi ⊂ F — {(fi = 1) ⇒ (oi =1 ⊕ si = 1) ∧ (di = 0), (fi =0) ⇒ (oi = 0 ⊕ di = 1) ∧ (si =0)})

OutputsFeatures toSelect

s1 = 0, s2 = 0, s3 = 0, s4 =0, s5 = 0, s6 = 0, s7 = 1

Features toDeselect

d1 = 0, d2 = 0, d3 = 0, d4 =

0, d5 = 0, d6 = 1, d7 = 0New ValidConfig.

f1 = 1, f2 = 1, f3 = 0, f4 =1, f5 = 1, f6 = 0, f7 = 1

Table 1: Diagnostic CSP Construction

We always require f1 = 1 to ensure that theroot feature is always selected. For void featuremodels, there will be no valid solution and thesolver will respond that no solution was found.CURE could be used to detect void feature mod-els but it would be more appropriate to use a tech-nique designed for this purpose (Trinidad et al.,2007).

One key input to CURE is the CSP describingthe set of all valid feature selections from the fea-

8

Page 9: Automated Diagnosis of Feature Model Configurations

ture model (the Feature Model Rules in Table 1).Since these valid feature selections are describedas constraints over the variables in F , a valid la-

beling of F will always yield a valid feature

selection. Once a valid labeling of F is found,the goal is to determine how to modify the la-beling of O to match the valid feature selectiondenoted by the labeling of F .

First, a constraint must be introduced to modelwhen a feature in the current invalid configurationneeds to be deselected to reach the correct config-uration. If the ithfeature is included in the currentconfiguration (oi = 1), but is not in the new validconfiguration (fi = 0), we want the solver to rec-ommend that it be deselected (di = 1). For everyfeature, we introduce the following constraint todetermine if the ith feature in O needs to be des-elected1:

(fi = 0) ⇒ (oi = 0 ⊕ di = 1) ∧ (si = 0)

.If fi is not selected in the correct configura-

tion (fi = 0), then either the feature was alsonot selected in the current invalid configuration(oi = 0), or the feature needs to be deselected(di = 1). Furthermore, if a feature is not neededin the valid configuration (fi = 0) then clearly itshould not be a recommended selection (si = 0).

The solver must also recommend features to se-lect. If the ith feature is selected in the correctand valid configuration fi = 1, and not selectedin the current invalid configuration (oi = 0), thenit needs to be selected (si = 1). For each feature,we introduce the constraint

(fi = 1) ⇒ (oi = 1 ⊕ si = 1) ∧ (di = 0)

.If a feature is needed by the correct configura-

tion (fi = 1), then either the feature was presentin the invalid configuration (oi = 1) or the featurewas not present in the invalid configuration andneeds to be selected (si = 1). Clearly, a featureshould not be deselected if fi = 1 and thus di = 0.

1The symbol ”⊕” denotes exclusive or

The state of each feature, oi, in the current in-valid configuration is compared against the cor-rect state of the feature, fi, in the valid featureconfiguration. The behavior of each comparisoncan fall into four cases:

1. A feature is selected and does not need

to be deselected. If the ith feature is in thecurrent invalid configuration (oi = 1), andalso in the new valid configuration (fi = 1),no changes need be made to it (si = 0, di =0)

2. A feature is selected and needs to be

deselected. If the ith feature is in the cur-rent invalid configuration (oi = 1) but not inthe new valid configuration (fi = 0), it mustbe deselected (di = 1)

3. A feature is not selected and does not

need to be selected. If the ith feature is notin the current invalid configuration (oi = 0)and is also not needed in the new configu-ration (fi = 0) it should remain unchanged(si = 0, di = 0)

4. A feature is not selected and needs to

be selected. If the ith feature is not selectedin the current invalid configuration (oi = 0)but is present in the new correct configura-tion (fi = 1), it must be selected (si = 1)

3.3. Optimal Diagnosis Method

The next step in the CURE diagnosis processis to use the solver to label the variables and pro-duce a series of recommendations. For any givenconfiguration with a conflict, there may be mul-tiple possible ways to eliminate the problem. Forexample, in the automotive example from Sec-tion 2.1.3, the valid corrective actions were ei-ther (1) remove the 1 Mbit/s CAN Bus and selectthe 250 Kbit/s CAN Bus or (2) remove the Non-ABS Controller and select the ABS Controller.We must therefore tell the solver how to selectwhich of the (many) possible corrective solutionsto suggest to developers.

The most basic suggestion selection criteria de-velopers can use to guide the solver’s diagnosisis to tell it to minimize the number of changesto make to the current configuration, i.e., prefer

9

Page 10: Automated Diagnosis of Feature Model Configurations

suggestions that require changing as few things aspossible in the current invalid configuration. Toimplement this approach, we solve for a CSP la-beling that minimizes:

n∑

0

si +

n∑

0

di

, which is the total number of changes that the so-lution requires the developer to make. By miniz-ing this sum we therefore minimize the total num-ber of required changes.

Each labeling of the diagnostic CSP will pro-duce two sets of features corresponding to the fea-tures that should be selected (S) and deselected(D) to reach the new valid configuration. Devel-opers can ask the solver to cycle through the dif-ferent potential labelings of the diagnostic CSPto evaluate potential remedies. Moreover, eachnew labeling (new diagnosis) causes the solverto backtrack and create new values for F , whichallows developers to evaluate not only the sug-gested modifications but the configuration thatthe remedy will produce. Another way to furtherrefine the guidance for the diagnosis is to con-strain the new state captured in the labeling ofF . This technique is utilized by the extensions inSections 4.2 and 4.3.

Table 1 shows a complete set of inputs andoutput suggestions for diagnosing the automotivesoftware example from Section 2.1.3. If there aremultiple labelings of the CSP, initially only onewill be returned. After the first solution has beenfound, however, the solver can much more effi-ciently cycle through the other equally ranked setsof corrective suggestions.

3.4. Simplified CSP Formulation

Solving time can be several minutes for largefeature models. To reduce solving time, the samebasic problem can often be restated using an al-ternate CSP formulation. Below we present asimplified version of the CURE diagnostic CSP.Section 6.5 then shows that this simplified CSPformulation can provide substantial reductions indiagnosis time for larger feature models.

The CURE process uses the CSP solver to de-rive the closest valid configuration to an invalid

configuration. The closest valid configuration isdefined as a function over the variables in the setsD and S. In a standard CURE diagnosis, theclosest valid configuration is defined as the con-figuration that minimizes the sum of the variablesin the sets D and S:

G =n∑

0

si +n∑

0

di

, i.e., the diagnosis should attempt to recommendmaking as few changes as possible.

The D and S variables capture a difference ofthe invalid configuration and the closest valid con-figuration. If a feature is selected in the invalidconfiguration and not in the closest valid config-uration, the corresponding D variable is set to1. As part of the solving process, therefore, thesolver performs the comparison of the current in-valid and the closest valid configuration.

Although the difference between the closestvalid configuration and the invalid configurationmust be calculated by the CSP solver, the solverdoes not need to determine which features shouldbe selected or deselected explicitly. The processof determining whether to select or deselect a fea-ture can easily be performed in code outside of thesolver. For example, after the solver has deter-mined the closest valid configuration, Java codecan be used to compare it to the invalid config-uration and recommend which features to selector deselect via a simple difference of the oi and fi

variables.To simplify the CURE CSP, the D and S vari-

ables can be replaced with a single set of vari-ables, N , noting the feature selections that differbetween the invalid and closest valid configura-tion. The N variables do not indicate whether toselect or deselect a feature, but merely that its se-lection state differs in the two configurations. Inthe simplified CSP formulation, for each featurein the feature model, a corresponding ni variableis created. Each variable ni ∈ N can have valuezero or one. Moreover,

(ni = 1) ⇒ (oi 6= fi)

, i.e., ni will only have value 1 if the selection state

10

Page 11: Automated Diagnosis of Feature Model Configurations

of the corresponding feature differs in the invalidand closest valid configurations.

Removing the S and D variables has an impacton the expressiveness of the function that can beused to define what constitutes the closest validconfiguration. For example, if we define the opti-mization goal, G, as the function

G =

n∑

0

si +

n∑

0

2di

the solver can be instructed to preference addingfeatures to fix the configuration. Likewise, thefunction tells the solver that adding 3 featuresis more expensive than deselecting 1. Developerscan therefore exercise fine-grained control over theweighting of the selection and deselection of fea-tures when diagnosing an invalid configuration.

By removing the D and S variables, the simpli-fied CSP formulation eliminates the ability to putweights on the relative importance of feature se-lection and deselection in the diagnosis. As shownby the results in Section 6.5, this tradeoff can yielda significant improvement in diagnosis speed. Thesimplified CSP formulation is preferable, there-fore, unless fine-grained weighting of feature se-lection and deselection is needed.

4. Diagnostic Customization and Benefits

This section presents various benefits of CUREand possible ways of customizing the diagnosticprocess.

4.1. Bounding Diagnostic Method

Due to time constraints, it may not be pos-sible to find the optimal number of changes forextremely large feature models. In these cases, amore scalable approach is to attempt to find anysuggestion that requires fewer than K changes orwith a cost less than K. Rather than directly ask-ing for an optimal answer, we add the constraint

n∑

i=1

si + di ≤ K

to the CSP and ask the solver for any solution.

The sum of all variables si ⊂ S and di ⊂ D

represents the total number of feature selectionsand deselections that need to be made to reachthe new valid configuration. The sum of both ofthese sets is thus the total number of modifica-tions that must be made to the original invalidconfiguration. The new constraint, ensures thatthe solver only accepts diagnosis solutions that re-quire the developer to make K or fewer changesto the invalid solution.

The solver is asked for any answer that meetsthe new constraints. In return, the solver will pro-vide a solution that is not necessarily perfect, butwhich fits our tolerance for change. If no solutionis found, we can increment K by a factor and rein-voke the solver or reassess our requirements. Theresults in Section 6.4 show that it is significantlyfaster to search for a bounded solution rather thanan optimal solution.

If the solver cannot find a diagnosis that makesfewer than K modifications, it will state thatthere is no valid solution that fits a K changebudget.

4.2. Debugging from Different Viewpoints

As discussed in Section 2.1.3, we need the abil-ity to debug the configuration from different view-points. Each viewpoint represents a set of fea-tures that the solver should avoid suggesting toadd or remove from the current configuration.For example, using the automobile scenario fromSection 2.1.3, the solver can debug the problemfrom the point of view that hardware decisionstrump software features by telling the solver notto suggest selecting or deselecting any hardwarefeatures.

Debugging from a viewpoint works by pre-assigning values for a subset of the variables inF and O. For example, to force the feature fi

currently in the configuration to remain unalteredby the diagnosis, the values fi = 1 and oi = 1 areprovided to the solver. Since (fi = 1) ⇒ (oi =1 ⊕ si = 1) ∧ (di = 0), pre-assigning these valueswill force the solver to label si = 0 and di = 0.

To debug from a given point of view, for eachfeature fv, in that viewpoint, we first add the con-straints, fv = 1, ov = 1, sv = 0, and dv = 0,

11

Page 12: Automated Diagnosis of Feature Model Configurations

as shown in Figure 5. The solver then derives

Figure 5: Debugging from a Viewpoint

a diagnosis that recommends alterations to otherfeatures in the configuration and maintains thestate of each feature fv. The CURE diagnosticmodel can therefore be used to debug from dif-ferent viewpoints and address Challenge 3 fromSection 2.1.3.

Pre-assigning values for variables in F and O

can also be used to debug staged configuration er-rors from Challenge 1, Section 2.1.1. With stagedconfiguration errors, at some point in time T ′, de-velopers need to select a feature that is in con-flict with one or more features selected at timeT < T ′. To debug this type of conflict, developerspre-assign the desired (but currently unselectable)feature at time T ′ the value of 1 for its oi and fi

variables. Developers can also pre-assign valuesfor one or more other features decisions from pre-vious stages of the configuration that must notbe altered. The solver is then invoked to find aconfiguration that includes the desired feature atT ′ and minimizes the number of changes to fea-ture configuration decisions that were made at allpoints in time T < T ′.

4.3. Cost Optimal Conflict Resolution

As shown in Section 2.1.2, conflicts can occurwhen multiple stakeholders in a configuration pro-cess pull the solution in different directions. De-bugging tools are therefore needed to mediate the

conflict in a cost conscious manner. For exam-ple, when a car’s software configuration is incom-patible with the legacy ECU configuration, it is(probably) cheaper to change the software con-figuration than to change the ECU configurationand the assembly process of the car. The solvershould therefore try to minimize the overall costof the changes.

We can extend the CSP model to perform cost-based feature selection and deselection optimiza-tion. First, we extend the CURE model to asso-ciate a cost variable, bi ⊂ B, with each feature inthe feature model. Each cost variable representshow expensive (or conversely how beneficial) it isfor the solver to recommend the state of that fea-ture be changed. Before each invocation of the de-bugger, the stakeholders provide these cost vari-ables to guide the solver in its recommendationsof features to select or deselect.

Next, we construct the superset of the featuresthat the various stakeholders desire, as shown inFigure 6. The superset represents the ideal, al-though incorrect, configuration that the stake-holders would like to have. The goal is to finda way to reach a correct configuration from thissuperset of features that involves the lowest to-tal cost for changes. The superset is input to thesolver as values for the variables in O.

Finally, we alter our original optimization goalso that the solver will attempt to minimize (ormaximize) the cost of the features it suggests se-lecting or deselecting. We define a global costvariable G and let G capture the sum of the costsof the changes that the solver suggests, as follows:

G =

n∑

i=1

(di ∗ bi) + (si ∗ bi)

. G is thus equal to the sum of the costs of all fea-tures that the solver either recommends to selector deselect. Rather than instructing the solver tominimize the sum of S ∪D, we ask it to minimizeor maximize G.

The result of the labeling is a series of changesneeded to reach a valid configuration that opti-mally integrates the desires and decisions of thevarious stakeholders. Of course, one particular

12

Page 13: Automated Diagnosis of Feature Model Configurations

Figure 6: Constructing the Feature Selection Superset for Conflict Mediation

stakeholder may incur more cost than another inthe interest of reaching a globally better solution.Further constraints, such as limiting the maxi-mum difference between the cost incurred by anytwo stakeholders, could also be added. The medi-ation process can be tuned to provide numeroustypes of behavior by providing different optimiza-tion goals. This CSP diagnostic method enablesCURE to address Challenge 2 from Section 2.1.2.

5. Using CURE for Software Configuration

Evolution

A key difference between configuration diagno-sis and configuration evolution is that both A andA′ are valid feature model configurations. Config-uration diagnosis takes an invalid configurationO and finds a valid configuration F that is asclose as possible. Although CURE was originallydeveloped for diagnosing errors in configurations,this section shows that it can also be applied tosoftware configuration evolution problems wherethe goal is to find A′ while optimizing a function∆(A, A′). This section also presents methods forusing CURE to evolve feature model configura-tions to handle a number of SPL scenarios (suchas supply-chain disruption and feature expansion)that necessitate configuration evolution.

5.1. Supply-chain Disruption

CURE can be used to automate the process offixing configurations that have become invalid dueto supply-chain disruptions. CURE’s automateddiagnosis abilities allow developers to analyze evo-lution alternatives that optimize specific proper-ties, such as cost. The key to CURE’s abilityto automate evolution is that the legacy softwareconfiguration, A, can be modeled by the invalid

configuration O and the target evolved configu-ration, A′, can be modeled as the closest validconfiguration F .

Cure’s goal function, G, which is typically afunction of O and F , can also be used to model∆(A, A′) by defining ∆(A, A′) in terms of the O,F , S, and D variables. Thus, we model the evo-lution process as:

O = A

F = A′

G = ∆(A, A′)

R = C

R′ = C ∧ Additional CSP Constraints

For example, the Advanced Infotainment fea-ture, f4, in Figure 3 is no longer a valid featureselection because a required ECU is not available.To automate the evolution of each configurationthat has this feature selected, CURE can be usedto derive the closest valid configuration for which

R′ = C ∧ (f4 = 0)

. Moreover, designers can use the cost coefficientsintroduced in Section 4.3 to represent the rela-tive prices of different evolution options and useCURE to perform cost optimal evolution automa-tion by making G calculate the difference in costbetween the legacy and evolved configuration.

The more complex evolution scenario outlinedin Figure 2 can also be automated with CURE. Inthis case, developers need to evolve any configu-rations that include both the Advanced Infotain-ment and 50W features. Using CURE, develop-ers add the following additional constraint to thediagnostic CSP:

R′ = C ∧ ((f4 = 1) ⇒ (f5 = 1))

13

Page 14: Automated Diagnosis of Feature Model Configurations

. This additional constraint forces the CSP solverto ensure that any evolved configurations that itsuggests adhere to the new feature model rule.Again, developers can apply cost coefficients orconstraints requiring that Advanced Infotainmentbe selected in evolved configurations (f4 = 1).

5.2. Feature Expansion Planning

To automate feature expansion planning withCURE, the following new requirement set

R′ = C ∧ (∀fi ∈ New, fi = 1)

is produced that requires that each desired newfeature, fi ∈ New, is selected, i.e., the fi variablefor each newly required feature must have value1, meaning that the feature is selected. If thereare hard requirements on feature selections thatcan not be altered, these are also encoded into R′

as

R′ = C∧(∀fi ∈ New, fi = 1)∧(Hard Requirements)

For example,

R′ = C ∧ (f2 = 1) ∧ (s1 = 0 ∧ d1 = 0)

finds a configuration in Figure 3 that adds theRear Infotainment feature without modifying theForward Infotainment feature. The constraint(f2 = 1) requires that the Rear Infotainment fea-ture be selected by the CSP solver and the con-straint (s1 = 0 ∧ d1 = 0) ensures that no changesare recommended for the Forward Infotainmentfeature. As with previous examples, cost coeffi-cients and the goal function G can be defined topreference the lowest cost evolved configuration.

6. Empirical Results

Effective automated diagnostic methods shouldscale to handle feature models of production sys-tems. This section presents empirical resultsfrom experiments we performed to evaluate thescalability of CURE. We compare the scalabilityof both CURE’s optimal and bounding methodsfrom Sections 3.3 and 4.1.

6.1. Experimental Platform

To perform our experiments, we used the im-plementation of CURE provided by the ModelIntelligence libraries from the Eclipse Founda-tion’s Generic Eclipse Modeling System (GEMS)project (eclipse.org/gmt/gems). Internally,the GEMS Model Intelligence implementationof CURE uses the Java Choco ConstraintSolver (http://choco.sourceforge.net) to de-rive labelings of the diagnostic CSP. The experi-ments were performed on a computer with an In-tel Core DUO 2.4GHZ CPU, 2 gigabytes of mem-ory, Windows XP, and a version 1.6 Java VirtualMachine (JVM). The JVM was run in client modeusing a heap size of 40 megabytes (-Xms40m)and a maximum memory size of 256 megabytes(-Xmx256m).

A challenging aspect of scalability analysis isthat CSP-based techniques can vary in solvingtime based on individual problem characteristics.In theory, CSP’s have exponential worst-case timecomplexity, but are often much faster in practice.To evaluate CURE it was therefore necessary toapply it to as many models as possible. The keychallenge with this approach is that hundreds orthousands of real feature models are not readilyavailable and manually constructing them is im-practical.

To provide the large numbers of feature mod-els needed for our experiments we built a fea-ture model generator that randomly creates fea-ture models with the desired branching and con-straint characteristics. We also imbued the gen-erator with the capability to generate feature se-lections from a feature model and probabilisti-cally insert a bounded number of errors/conflictsinto the configuration. The feature model gener-ator and code for these experiments is availablein open-source form from (code.google.com/p/ascent-design-studio).

From our preliminary feasibility experiments,we observed that the branching factor of the treehad little effect on the algorithm’s solving time.We also compared diagnosis time using modelswith 0%, 10%, and 50% cross-tree constraints andobserved that each increment in the percentage ofcross-tree constraints improved performance. For

14

Page 15: Automated Diagnosis of Feature Model Configurations

example, with the optimal method and 1,000 fea-ture models, the average diagnosis time graduallydecreased from 47 seconds with 0% cross-tree con-straints to 36 seconds with 50% cross-tree con-straints. The key indicator of the solving com-plexity was the number of XOR- or cardinality-based feature groups in a model.2

Our tests limited the branching factor to atmost five subfeatures per feature. We also setthe probability of XOR- or cardinality-based fea-ture groups being generated to 1/3 at each fea-ture with children. We chose 1/3 since most fea-ture models we have encountered contain more re-quired and optional relationships than XOR- andcardinality-based feature groups. The total num-ber of cross-tree constraints was set at 10%. Wealso eliminated all diagnosis results from void fea-ture models, since void feature models producedfaster diagnostic times and would have skewed theresults towards smaller solving times.

To generate feature selections with errors, weused a probability of 1/50 that any particular fea-ture would be configured incorrectly. For eachmodel, we bounded the total errors at 5. In ourinitial experiments, the solving time was not af-fected by the number of errors in a given fea-ture model. Again, the prevalence of XOR- orcardinality-based feature groups was the key de-terminer of solving time.

6.2. Experiment 1: Bounding Method Scalability

Hypothesis. We hypothesized that CURE’sCSP-based diagnostic technique could be used todiagnose configurations with 1,000s of featureswhen the bounding method was employed.

Experiment design. To test the diagnos-tic capabilities of the bounding method, we gen-erated large numbers of feature models and aflawed configuration for each model. The CUREbounded diagnostic method was used to diagnoseeach of the flawed configurations. The speed ofthe bounding technique allowed us to test 2,000feature models at each data point (2,000 different

2XOR and cardinality-based feature groups are featuresthat require the set of their selected children to satisfy acardinality constraint (the constraint is 1..1 for XOR).

variations of each size feature model) and test thebounding method’s scalability for feature modelsup to 500 features.

With models above 500 features, we had to re-duce the number of samples at each size to 200models due to time constraints. Although thesesamples are small, they demonstrate the generalperformance of our technique. Moreover, the re-sults of our experiments with feature models upto 500 features were nearly identical with samplesizes between 100 and 2,000 models.

Analysis of results. Figure 7 shows the timerequired to diagnose feature models ranging insize from 50 to 500 features using the boundedmethod. The figure captures the worst and aver-

Figure 7: Diagnosis Time for Both Methods for Large Fea-ture Models

age solving time in the experiments. As seen fromthe results, our technique could diagnose modelswith 500 features in an average of ≈300ms.

The upper bound used for this experiment wasa maximum of 10% feature selection changes.When the feature bound was too tight for thediagnosis (i.e., more were needed to reach a cor-rect state) the solver quickly declared there wasno valid solution. We therefore discarded all in-stances where the bound was too tight to avoidskewing the results towards shorter solving times.

Figure 7 shows the results of testing the solvingtime of the bounding method on feature modelsranging in size from 500 to 5,000 features. Modelsof this size were sufficient to demonstrate scalabil-ity for common production systems. The resultsshow that for a 5,000 feature model, the averagediagnosis time was ≈ 50 seconds.

Another key variable we tested was how thetightness of the bound on the maximum num-

15

Page 16: Automated Diagnosis of Feature Model Configurations

ber of feature changes affected the solving timeof the technique. We took a set of 200 featuremodels and applied varying bounds to see howthe bound tightness affected solution time. Fig-ure 8 shows that tighter bounds produced fastersolution times. These results indicate that tighter

Figure 8: 500 Feature Diagnosis Time with BoundingMethod and Varying Bounds

bounds allow the solver to discard infeasible solu-tions quickly and thus arrive at a solution faster.

6.3. Experiment 2: Optimal Method Scalability

Hypothesis. We hypothesized that the op-timal CURE diagnostic method could also scaleto feature models with 1,000s of features. We ex-pected that the diagnostic time would be substan-tially longer than with the bounding method.

Experiment design. We tested the scalabil-ity of the optimal diagnosis method using 2,000samples below 500 features and 200 samples forall larger models. For each feature model, an in-valid configuration was generated and diagnosedwith CURE. We tracked the best, average, andworst diagnosis time for each size model.

Analysis of results. Figure 7 shows the re-sults from feature models up to 500 features. At500 features, the optimal method required an av-erage of ∼1.5 seconds to produce a diagnosis. Fig-ure 7 also shows the tests from larger models rang-ing in size up to 5,000 features. For a model with5,000 features, the solver required an average of∼3 minutes per diagnosis.

6.4. Comparative Analysis of Bounding and Op-timal Methods

Finally, we compared the scalability and qual-ity of results produced with the two methods.Figure 7 shows the bounding method performs

and scales significantly better than the optimalmethod. For feature models of up to 1,000 fea-tures, however, both techniques take less than5 seconds and the optimal method is the bet-ter choice. This result raises the question of howmuch of a tradeoff in solution quality for speed ismade when the bounding method is used over theoptimal method for larger models.

The bound that is chosen determines the qual-ity of the solution that is produced by the solver.The optimality of a diagnosis given by the bound-ing method is the number of changes suggested bythe bounding method, Bounded(S ∪ D), dividedby the optimal number of changes, Opt(S ∪ D),

which yields Bounded(S∪D)Opt(S∪D)

. Since the bounding

method uses the constraint (S ∪ D) ≤ K to en-sure that at most K changes are suggested, wecan state the worst case optimality of the boundedmethod as K

Opt(S∪D). The closer our bound, K, is

to the true optimal number of changes to make,the better the diagnosis.

Since tighter bounds produce faster solvingtimes and better results, debuggers should startwith very small bounds and iteratively increasethem upward as needed. One approach is to layeran adaptive algorithm on top of the diagnosis al-gorithm to move the bound by varying amountseach time the bound proves too tight. Anotherapproach is to employ binary search to home inon the ideal bound. We will investigate both tech-niques in our future work.

6.5. Experiment 3: Simplified CSP Formulation

Hypothesis. We hypothesized that the sim-plified CSP formulation would provide a constantreduction in solving time compared to the stan-dard complex CSP formulation.

Experiment design. We performed experi-ments to compare the diagnosis speed of the stan-dard complex CSP formulation against the simpli-fied CSP formulation, as described in Section 3.4.We used our feature model generation and con-figuration infrastructure to produce feature mod-els and flawed configurations of varying sizes andthen diagnosed them with both techniques. Theresults in this section present the differences indiagnosis times that we observed. We generated

16

Page 17: Automated Diagnosis of Feature Model Configurations

and solved feature models ranging in size from 500to 5,000 features. For each size feature model, wesolved 20 instances and tracked the average solv-ing time of the two CSP formulations.

Analysis of results. The results are shown inFigure 9. The results in this figure show the sim-

Figure 9: Comparison of Diagnosis Times for the Complexand Simplified CSP Formulations on Feature Models with500 to 5,000 Features

plified CSP technique had a much faster averagediagnosis time. For example, with 5,000 features,the simplified CSP formulation had an averagediagnosis time of less than half of the standardcomplex CSP formulation.

We took the data shown in Figure 9 and ana-lyzed it to calculate the percentage of reductionin average solving time provided by the simpli-fied CSP formulation over the complex formula-tion. Figure 10 shows the results of this analysis.For smaller feature models with 500 features, the

Figure 10: Simplified CSP Formulation Reduction in Di-agnosis Time Versus Feature Model Size

simplified technique provided a 25% reduction indiagnostic time. As the size of the feature modelsgrew, moreover, the percentage of reduction in di-agnosis time did not remain constant but grew asa function of the size of the feature models. For

example, at 2,000 features, the simplified formu-lation provided a roughly 43% decrease in averagesolving time. At 5,000 features, the simplified for-mulation provided a roughly 64% decrease in av-erage solving time.

The results indicate that for small feature mod-els, the simplified technique can provide a modestimprovement in the overall average diagnosis timeacross a number of feature models. The improve-ment comes at a reduction in the expressiveness ofthe goal functions that can be created. For largefeature models, however, the simplified CSP for-mulation provides a significant decrease in averagesolving time. Moreover, the results indicate thatthe improvement provided by the simplified CSPformulation is proportional to the size of the fea-ture model, i.e., the simplified CSP formulationsaves the most time on the most time-consumingmodels to diagnose.

6.6. Debugging ScenariosStaged configuration and viewpoint debugging

(Challenges 1 and 3 in Sections 2.1.1& 2.1.3) arespecial cases of the technique where the solver isnot allowed to modify the selection state of one ormore features (i.e., the viewpoint or the featureat time T ′). Both these special cases of debug-ging actually reduce the search space by fixingvalues for one or more of the CSP variables. Forexample, performing staged configuration debug-ging (which fixes the value for one CSP variable ona model with 1,000 features) reduced the optimalmethod’s average solving time by ≈ 2.5 secondsand the bounding method by ≈ .1 seconds.

Cost-based conflict mediation (Challenge 2 inSection 2.1.2) performs identically to the stan-dard diagnosis technique. Cost-based mediationmerely introduces a series of coefficients, bi ⊂ B

into the optimization goal. These coefficients donot increase solving time. Moreover, initiating thediagnosis method with the superset of the configu-ration participants’ desired feature selections alsodid not impact performance.

7. Related Work

This section compares our work on CURE withrelated research projects in the areas of auto-

17

Page 18: Automated Diagnosis of Feature Model Configurations

mated configuration, autonomic healing, and con-figuration quality evaluation.

7.1. Automated Configuration

In prior work, Trinidad et al. (Trinidad et al.,2007) showed how feature models can be trans-formed into diagnosis CSPs and used to identifyfull mandatory features, void features, and deadfeature models. Developers can use this diagnos-tic capability to identify feature models that donot accurately describe their products and to un-derstand why not. The technique we describedin this paper builds on this idea of using a CSPfor automated diagnosis. Whereas Trinidad fo-cuses on diagnosing feature models that do notdescribe their products, we build an alternate di-agnosis model to identify conflicts in feature con-figurations. Moreover, we provide specific recom-mendations as to the minimal set of features thatcan be selected or deselected to eliminate the er-ror.

Debugging techniques for feature models wereinvestigated by Batory et al. in Batory (2005).Their techniques focus on translating featuremodels into propositional logic and using SATsolvers to automate configuration and verify cor-rectness of configurations. In general, their worktouches on debugging feature models rather thanindividual configurations. Our approach focuseson another dimension of debugging, the abil-ity to pinpoint errors in individual configura-tions and to specify the minimal set of featureselections and deselections to remove the error.Moreover, propositional logic-based approachesdo not typically provide maximization or mini-mization as primitive functions provided by thesolver. Since, CURE uses a CSP-based approach,minimization/maximization diagnosis functional-ity is built-in.

Pure::variants (Beuche, 2003), Feature Model-ing Plugin (FMP) (Czarnecki et al., 2005a), FeA-ture Model Analyser (FAMA) (Benavides et al.,2007), and Big Lever Software Gears (Buhrdorfet al., 2003) are tools developed to help devel-opers create correct configurations of SPL fea-ture models. These tools enforce constraints onmodelers as the features are selected. None of

these tools, however, addresses cases where fea-ture models with incorrect configurations are cre-ated and require debugging. The technique de-scribed in this paper provides this missing capa-bility. These tools and our approach are comple-mentary since the tools help to ensure that cor-rect configurations are created and our techniquediagnoses incorrect configurations that are built.

Propositional logic has been used to repre-sent software product lines by Mannion Mannion(2002). These techniques are useful for represent-ing product line models, but do nothing to correctflawed feature models. Moreover, they do not takeintelligent conflict resolution into consideration.While these techniques are capable of determin-ing if a selection is valid, there is no guaranteethat the selection is also appropriate. CURE dif-fers in that it can be used to diagnose and rem-edy existing, invalid feature selections. The twoapproaches are complementary, Mannion’s tech-niques can flag flawed configurations and CUREcan be used to repair the configuration flaws.

7.2. Autonomic Healing

Dudley et al. (2004) present systems that canautonomically identify and repair system prob-lems. In the event that a system fails, theseself-healing solutions are capable of determiningthe element or elements that are responsible forthe failure. Once the nature of a failure is de-termined, a replacement element is automaticallychosen from a set of potential replacement ele-ments, thus allowing the system to recover. LikeCURE, this technique is corrective and capable oftaking an invalid system configuration, discover-ing the element or elements that make the config-uration invalid, and replacing them with elementsautonomically to create a valid system. UnlikeCURE, however, this technique does not makeuse of feature models and is subject to runtimeconstraints, making it hard to use for SPLs ordiagnosis optimization.

7.3. Configuration Quality Evaluation

Other techniques provide mechanisms for rea-soning about feature model configurations when

18

Page 19: Automated Diagnosis of Feature Model Configurations

the exact non-functional properties of each fea-ture are not known. Zhang et al. (2003) introduceBayesian Belief Networks, which predict the im-pact of feature selection on quality attributes byconsidering the effects of choosing similar variantson the quality of past projects. Jarzabek et al.(2006) present a goal-oriented analysis that placesemphasis on the satisfaction of quality attributeduring feature selection. These techniques explorea different area of configuration than CURE, theare focused on finding configurations when thereis uncertainty about the effects of configurationdecisions. In contrast, CURE examines existingconfigurations, in which the ramifications of se-lecting a particular feature are already known.

Other modeling approaches, such as COVA-MOF (Sinnema et al., 2004) and QRF (Niemelaand Immonen, 2007), examine the ramificationsof feature selection on the overall quality of thesystem. CURE also examines the effect of select-ing a feature on the overall quality of the system.Unlike CURE, however, these techniques do notexamine how to correct flawed system configura-tions. CURE is a configuration diagnosis methodand not a configuration methodology.

The use of various quality attributes to assesspotential system configurations have been investi-gated. For example, Immonen (Immonen, 2005).introduces a tool for the prediction of availabilityand reliability for a system configuration. Olu-mofin et al. (Olumofin and Misic, 2005) provides amethod that examines the availability, reliability,performance, and modifiability of potential sys-tem configurations. Like CURE, these quality at-tributes are used to determine which feature toselect at each point of variability. Unlike CURE,however, these methods are used in the designphase to construct an initial system configuration,whereas CURE is used to correct flawed featureselections.

Etxeberria and Sagardui (2008) present a man-ual technique for using domain-relevant qualityattributes to evaluate variants in product line ar-chitectures. An additional quality feature tree canbe added as a branch to extended feature models.These quality features have a predefined qualita-tive and quantitative impact on the overall sys-

tem if selected. Feature selection can thereforedetermine the overall quality of a system configu-ration and assure that the system possesses nec-essary functionality. Again, CURE is a correctivetechnique for restoring validity to flawed featureselections, whereas the method in Etxeberria andSagardui (2008) is used during the design phaseto determine valid initial configurations.

8. Concluding Remarks

Production SPLs rely on large feature modelsthat make it hard to always produce correct con-figurations. When errors due occur, the lack ofconfiguration debugging tools forces developers tomanually pinpoint errors in configurations thatcan span thousands of features. Moreover, dueto the size of many feature models and other fac-tors (such as supply-chains), SPL configurationmay be performed in stages or in parallel (Batoryet al., 2006). In these situations, the feature se-lections of individual developers can conflict andintroduce errors in feature models that are evenmore challenging to identify and correct.

Another challenge of using an SPL is main-taining configurations over time. For example,an SPL that encompasses components obtainedthrough a supply-chain may have features that be-come unavailable due to vendors exiting the mar-ket or discontinuing support. In these scenarios,developers need a way of automating the evolu-tion of existing configurations to new and correctfeature selections that take these changes into ac-count.

This paper introduced a technique called Con-figuration Understanding and REmedy (CURE)that transforms invalid configurations into CSPsand automates (1) the diagnosis of invalid configu-rations and (2) the evolution of existing configura-tions to meet changing requirement sets. CUREderives the closest valid configuration to an ex-isting configuration that meets a new set of re-quirements. Moreover, developers can customizeCURE to provide optimal cost-based error andevolution diagnostics.

CURE’s CSP-based diagnosis model is extensi-ble and can be modified to perform conflict media-

19

Page 20: Automated Diagnosis of Feature Model Configurations

tion, run faster by using a simplified CSP formu-lation, or debug from different viewpoints. Ourempirical results show CURE can scale to produc-tion feature models with 5,000 features and stillprovide a diagnosis in between 45 seconds and 4minutes. These time bounds should be sufficientfor the design-time use of this algorithm in manyproduction SPLs.

The following are lessons learned from our ef-forts with CURE thus far:

• Simplifying the diagnostic CSP can

yield substantial improvements in diag-

nosis time. A simplified version of CURE’sCSP can be used to provide significant im-provements in solving speed with a limitedreduction in diagnostic flexibility. The re-duced flexibility does not allow developers toplace limits on only feature selection or dese-lection. If developers do not need to preciselypinpoint one type of change, the simplifieddiagnostic CSP should be used.

• CURE can scale to 1,000s of fea-

tures. We learned from our experimentswith CURE that it can diagnose models withroughly 5,000 features. The results fromtechniques, such as bounding or CSP simplifi-cation, indicate that potential to scale CUREup to larger feature model sizes. The tech-niques improve CURE’s speed by reducingthe number of variables that are present inthe CSP and placing a precise bound on thesearch space.

• It is hard to predict CURE’s diagnosis

time. CURE’s diagnosis time is dependenton the structure of the constraints in the fea-ture model. For some models, CURE pro-vides very fast diagnosis. In future work, weplan to investigate heuristic diagnostic tech-niques that can speed CURE’s diagnosis timefor models it does not perform well on.

• A good K bound for CURE is hard to

ascertain. The optimality of the diagno-sis provided by the bounding method is de-termined by how close K is set to the true

minimum number of features that need to bechanged to reach a valid state. Setting anaccurate bound for K is not easy. In fu-ture work, we plan to investigate differentmethods of honing the boundary used in thebounding method.

• Configuration evolution problems can

be modeled as configuration diagnosis

problems. Although configuration evolu-tion problems do not necessarily start withan invalid feature selection, we learned thatCURE’s diagnosis techniques can still be ap-plied successfully to them. When a fea-ture model’s constraints are changed or aconfiguration’s feature selection needs to bemodified, CURE can be used to automati-cally modify legacy configurations by model-ing them as invalid configurations. Automat-ing configuration evolution can provide costoptimal modification planning, which is hardto achieve with manual evolution techniques.

The CURE diagnosis technique has beenimplemented as part of the ASCENT De-sign Studio Intelligence project and is availablein open-source format from code.google.com/

p/ascent-design-studio and www.isa.us.es/

fama.

9. Acknowledgements

We would like to thank Jose Galindo for hishard work implementing CURE in FAMA.

References

Batory, D., 2005. Feature Models, Grammars, and Prepo-sitional Formulas. Software Product Lines: 9th Interna-tional Conference, SPLC 2005, Rennes, France, Septem-ber 26-29, 2005: Proceedings.

Batory, D., Benavides, D., Ruiz-Cortes, A., 2006. Auto-mated analysis of feature models: Challenges ahead.Communications of the ACM December.

Benavides, D., Segura, S., Trinidad, P., Ruiz-Cortes,A., 2007. FAMA: Tooling a framework for the auto-mated analysis of feature models. In: Proceeding of theFirst International Workshop on Variability Modellingof Software-intensive Systems (VAMOS).

20

Page 21: Automated Diagnosis of Feature Model Configurations

Benavides, D., Trinidad, P., Ruiz-Cortes, A., 2005. Au-tomated Reasoning on Feature Models. 17th Confer-ence on Advanced Information Systems Engineering(CAiSE05, Proceedings), LNCS 3520, 491–503.

Beuche, D., 2003. Variant Management withPure:: variants. Tech. rep., Pure-Systems GmbH,http://www.pure-systems.com.

Buhrdorf, R., Churchett, D., Krueger, C., November 2003.Salion’s Experience with a Reactive Software ProductLine Approach. In: Proceedings of the 5th InternationalWorkshop on Product Family Engineering. Siena, Italy.

Clements, P., Northrop, L., 2002. Software Product Lines:Practices and Patterns. Addison-Wesley, Boston.

Czarnecki, K., Antkiewicz, M., Kim, C., Lau, S., Piet-roszek, K., October 2005a. In: FMP and FMP2RSM:Eclipse Plug-ins for Modeling Features Using ModelTemplates. ACM Press New York, NY, USA, pp. 200–201.

Czarnecki, K., Helsen, S., Eisenecker, U., 2004. StagedConfiguration Using Feature Models. Software Prod-uct Lines: Third International Conference, SPLC 2004,Boston, MA, USA, August 30-September 2, 2004: Pro-ceedings.

Czarnecki, K., Helsen, S., Eisenecker, U., 2005b. Stagedconfiguration through specialization and multi-levelconfiguration of feature models. Software Process Im-provement and Practice 10 (2), 143–169.

Dudley, G., Joshi, N., Ogle, D., Subramanian, B., Topol,B., 2004. Autonomic Self-Healing Systems in a Cross-Product IT Environment. Proceedings of the Interna-tional Conference on Autonomic Computing, 312–313.

Etxeberria, L., Sagardui, G., 2008. Variability DrivenQuality Evaluation in Software Product Lines. In: Soft-ware Product Line Conference, 2008. SPLC’08. 12th In-ternational. pp. 243–252.

Immonen, A., 2005. A method for predicting reliability andavailability at the architectural level. Research Issues inSoftware Product-Lines-Engineering and Management,T. Kakola and JC Duenas, Editors.

Jaffar, J., Maher, M., 1994. Constraint Logic Program-ming: A Survey. Journal of Logic Programming 19 (20),503–581.

Jarzabek, S., Yang, B., Yoeun, S., 2006. Addressing qual-ity attributes in domain analysis for product lines. In:Software, IEE Proceedings-. Vol. 153. pp. 61–73.

Kang, K. C., Kim, S., Lee, J., Kim, K., Shin, E., Huh,M., January 1998. FORM: A Feature-Oriented ReuseMethod with Domain-specific Reference Architectures.Annals of Software Engineering 5 (0), 143–168.

Mannion, M., 2002. Using first-order logic for product linemodel validation. Proceedings of the Second Interna-tional Conference on Software Product Lines 2379, 176–187.

Metzger, A., Pohl, K., Heymans, P., Schobbens, P.-Y.,Saval, G., 2007. Disambiguating the documentation ofvariability in software product lines: A separation of

concerns, formalization and automated analysis. In: Re-quirements Engineering Conference, 2007. RE ’07. 15thIEEE International. pp. 243–253.

Niemela, E., Immonen, A., 2007. Capturing quality re-quirements of product family architecture. Informationand Software Technology 49 (11-12), 1107–1120.

Olumofin, F., Misic, V., 2005. Extending the ATAM Ar-chitecture Evaluation to Product Line Architectures. In:IEEE/IFIP Working Conference on Software Architec-ture, WICSA.

Rabiser, R., Grunbacher, P., Dhungana, D., 2007. Sup-porting Product Derivation by Adapting and Augment-ing Variability Models. Software Product Line Confer-ence, 2007. SPLC 2007. 11th International, 141–150.

Reiter, R., 1987. A theory of diagnosis from first principles.Artificial Intelligence 32 (1), 57–95.

Sinnema, M., Deelstra, S., Nijhuis, J., Bosch, J., 2004. CO-VAMOF: A Framework for Modeling Variability in Soft-ware Product Families. LECTURE NOTES IN COM-PUTER SCIENCE, 197–213.

Trinidad, P., Benavides, D., Duran, A., Ruiz-Cortes, A.,Toro, M., 2007. Automated error analysis for the ag-ilization of feature modeling. Journal of Systems andSoftware, in press.

Van Hentenryck, P., 1989. Constraint Satisfaction in LogicProgramming. MIT Press Cambridge, MA, USA.

White, J., Czarnecki, K., Schmidt, D. C., Lenz, G., Wien-ands, C., Wuchner, E., Fiege, L., October 2007. Au-tomated Model-based Configuration of Enterprise JavaApplications. In: The Enterprise Computing Confer-ence, EDOC. Annapolis, Maryland USA.

White, J., Schmidt, D. C., Benavides, D., Trinidad, P.,Ruiz-Cortez, A., Sep. 2008. Automated Diagnosis ofProduct-line Configuration Errors in Feature Models.In: Proceedings of the Software Product Lines Confer-ence (SPLC). Limerick, Ireland.

Zhang, H., Jarzabek, S., Yang, B., 2003. Quality Pre-diction and Assessment for Product Lines. LECTURENOTES IN COMPUTER SCIENCE, 681–695.

21