HOW ROBUST IS A ROBUST POLICY COMPARING … · the case of designing a policy for stimulating the...

5th International Conference on Future-Oriented Technology Analysis (FTA) - Engage today to shape tomorrow Brussels, 27-28 November 2014

THEME 3: CUTTING EDGE FTA APPROACHES

- 1 -

HOW ROBUST IS A ROBUST POLICY? COMPARING

ALTERNATIVE ROBUSTNESS METRICS FOR ROBUST DECISION-MAKING

Jan H. Kwakkel

Erik Pruyt [email protected]

[email protected]

Delft University of Technology,

Faculty of Technology, Policy and Management,

P.O. Box 5015, 2600 GA Delft, The Netherlands

Abstract

Nowadays, decision-makers face deep uncertainties from a myriad of external factors (e.g., climate change, population growth, new technologies, economic developments) and their impacts. Traditionally, decision-makers in most policy domains, such as water management, transportation, spatial planning, and business planning, develop a policy or strategy with one or a few plausible futures in mind. If the future turns out to be different from the hypothesized futures, the policy’s performance is likely to deteriorate. The challenge then is to develop robust policies. That is, policies whose performance is insensitive to the resolution of the various uncertainties. One way of designing such robust policies is by designing the policy to be adapted over time in response to how the future actually unfolds (Erikson and Weber 2008). A key determinant for the efficacy of an adaptive policy is the specification of when and how to adapt it. A state of the art model-based decision support approach for addressing this problem is robust decision-making. The final policy design emerging from a robust decision-making analysis is depended on how robustness is being measured. To date, there is little guidance for selecting an appropriate robustness metric. In this paper we address this problem. We apply the outlined robust decision-making approach to the case of designing a policy for stimulating the transition of the European Energy system towards more sustainable functioning using three different robustness metrics. These metrics all use the mean and a measure for dispersion. We compare the policies as identified by each of the different robustness metrics and discuss the relative merits of metrics. We highlight that the different robustness metrics emphasize different aspects of what makes a policy robust. More specifically, measures that separate dispersion and the mean, effectively doubling the number of objectives, provide very valuable information on the trade-offs between the mean performance of the policy and the dispersion around this mean. We also discuss, based on our case, why analysts should use multiple robustness metrics.

Keywords: robust decision-making; deep uncertainty; adaptive policymaking, robust multi-objective optimization

Introduction

In many planning problems, planners face major challenges in coping with uncertain and changing physical conditions, and rapid unpredictable socio-economic development. How should society prepare itself for this confluence of uncertainty? Given the presence of irreducible uncertainties there is no straightforward answer to this question. Effective decisions must be made under unavoidable uncertainty (Lempert et al. 2003; Dessai et al. 2009). The acceptance

mailto:[email protected]

mailto:[email protected]



- 2 -

of uncertainty as an inevitable part of long-term decision-making has given rise to the development of new tools and approaches. These include Dynamic Adaptive Policy Pathways (Haasnoot et al. 2013), adaptive policy-making (Kwakkel et al. 2010), Real Options analysis (de Neufville and Scholtes 2011), info-gap decision theory (Ben Haim 2006), and Robust Decision Making (Lempert and Collins 2007). The shift to adaptive planning changes the way in which model-based decision support is to be structured (Weaver et al. 2013). Planning has frequently followed a predict-then-act approach (Lempert et al. 2004), where models are used to consolidate uncontested facts into a single model that is subsequently used to predict the future (Bankes 1993). In contrast, Weaver et al. (2013) argue that adaptive planning requires an exploratory modeling approach where modelers account for the various unresolvable uncertain factors by conducting series of computational experiments that systematically explore the consequences of alternative realizations of the various uncertain factors (Bankes et al. 2013).

An example of an exploratory modeling approach that can be used for designing adaptive policies is robust decision-making (Lempert and Collins 2007; Hamarat et al. 2013). This

approach naturally lends itself to participative deliberation with analysis (NRC 2009), and has been shown to be effective in complex multi-stakeholder settings (Parker et al. 2014). In this approach, a very large ensemble of plausible futures spanning the various key uncertain factors and reflecting the perspectives of the different stakeholders is created (Kwakkel and Pruyt 2013). Subsequently, this ensemble serves as a test bed for candidate policies. Through a process called scenario discovery, the key vulnerabilities and opportunities of a candidate policy are identified (Bryant and Lempert 2010). In light of this, an iterative process of (re)design of candidate policies takes place, aimed at improving the overall robustness of the policy. Typically, this redesign involves the inclusion of actions whose implementation is conditional on how the future unfolds (Hamarat et al. 2013). The challenge here is to avoid implementing these actions either too early or too late. Very recently, robust multi-objective optimization has been suggested as a technique for supporting the search for finding the right conditions (Hamarat et al. 2014).

In recent work, multi-objective optimization approaches have been used to support the design of adaptive policies. Kasprzyk et al. (2013) use it for finding feasible designs which are subsequently assessed on their robustness using robust decision making (Lempert and Collins 2007). Hamarat et al. (2014) integrate a robustness criterion directly into the optimization procedure and apply it to the development of an adaptive plan for the European energy transition. Kwakkel et al. (2014) also use a robust multi-objective optimization procedure for designing flexible climate adaptation pathways. Each of these papers uses a different way of measuring robustness. This raises the question how the choice of the robustness metric affects the final design of the adaptive policy or plan. It also raises the question whether some robustness metrics always outperform other robustness metrics. Insight into the consequences of different robustness metrics can help analysts in choosing a (set of) metric(s) that is appropriate for the case at hand, and improve awareness regarding the relative merit of alternative robustness metrics.

In this paper, we apply three different robustness metrics to the same case, allowing us to compare the results and providing insight into the relative merits of each of these three metrics. We start from the European energy transition case studied by Hamarat et al. (2014). This case focuses on finding an adaptive plan; built on the European emission-trading scheme that maximizes the potential of achieving the emission reduction targets set out by the European Commission. This is an appealing case for it focuses on a long-term planning problem under deep uncertainty. The necessary models are readily available, and preliminary analyses have already been reported on.

The paper is structured accordingly. In Section 2 we introduce the robust multi-objective optimization approach that will be used and detail the three robustness metrics. Section 3



- 3 -

presents the simulation model of the European energy system, the key uncertain factors that need to be accounted for, and the formulation of the robust multi-objective optimization problem. Section 4 contains the results for each of the three robustness metrics and their comparison. In section 5 we present the main conclusions.

Robust Multi-Objective Optimization

A general formulation of the multi-objective optimization problem is shown below, where Ω is the

total decision space, 𝑥 the vector of decision variables in the decision space, 𝐹 the multi-

objective function, fi the 𝑖𝑡ℎ objective function, cm the set of equality constraints and cn the set of

inequality constraints, 𝜀 the set of equality constraints, and 𝒳 the set of inequality constraints.

𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒𝑥∈ 𝛺 𝐹(𝑥) = [𝑓1(𝑥), 𝑓2(𝑥),… , 𝑓𝑖(𝑥)]

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑐𝑚(𝑥) = 0, ∀𝑚 ∈ ℰ 𝑐𝑛(𝑥) ≤ 0, ∀𝑛 ∈ 𝒳

There exist various approaches for solving multi-objective optimization problems. Over the last decade, substantial advances have been made through the use of genetic algorithms. Genetic algorithms use a population of solutions, which are evolved over the course of the run of the algorithm. This population can be evolved in such a way that it maintains diversity, while continually moving towards the Pareto frontier. In this way, multiple Pareto front solutions can be found in a single run of the algorithm (Deb et al. 2002). Currently, a wide variety of alternative multi-objective evolutionary algorithms are available for solving multi-objective optimization problems (Kollat and Reed 2006; Hadka and Reed 2013). One of the most popular is Non-Dominated Sorting Genetic Algorithm II (Deb et al. 2002). In this paper, we will use an extension of NSGA-II known as ε-NSGA-II (Reed and Devireddy 2004; Reed et al. 2007).

Robust optimization methods aim at finding, in the presence of uncertainty about input parameters, optimal outcomes that are not overly sensitive to any specific realization of the uncertainties (Ben-Tal and Nemirovski 1998, 2000; Bertsimas and Sim 2004; Bai et al. 1997; Kouvalis and Yu 1997). In robust optimization, the uncertainty that exists about the outcomes of interest is described through a set of scenarios (Mulvey et al. 1995). Robustness is then defined over this set of scenarios. In this way, robust optimization differs from worst-case formulations such as minimax, which can produce very costly and conservative solutions (Mulvey et al. 1995).

In robust optimization, robustness is defined over a set of scenarios. The way in which robustness over a set of scenarios is defined can affect the solutions that are being found. In this paper we consider three related definitions of robustness. The first metric is based on the intuition that a robust solution will have a good average result with very limited dispersion around it. In mathematical form

𝑓𝑖(𝑥) =

(𝜇𝑖 + 1) ∗ (𝜎𝑖 + 1), 𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑎𝑡𝑖𝑜𝑛(𝜇𝑖 + 1)

(𝜎𝑖 + 1), 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑎𝑡𝑖𝑜𝑛



- 4 -

where 𝜇𝑖 is the mean over the set of scenarios for outcome indicator 𝑖 and 𝜎𝑖 is the standard

deviation. The +1 is included to handle situations where either 𝜇𝑖 or 𝜎𝑖 is close to zero. This metric is essentially a signal to noise ratio, or a form of risk discounting. A downside of this first metric is that it does not provide insight into the trade-off between improving the mean and reducing the standard deviation in case of maximization. Even worse, functions that combine the mean and variance are not always monotonically increasing (Ray et al. 2013). An obvious solution is to include both the mean and the standard deviation as separate objectives. This is metric two:

𝑓𝑖(𝑥) = 𝜇𝑖, 𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑜𝑟 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒𝜎𝑖, 𝑎𝑙𝑤𝑎𝑦𝑠 𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒

The idea is to always reduce dispersion as measured by the standard deviation, and either maximize or minimize the mean depending on the objective. A problem with metrics one and two is that by using the standard deviation, good and bad deviations from the mean are treated equally (Takriti and Ahmed 2004). What matters for robust decision, however, is to minimize the undesirable deviations from the mean. This problem can be solved in various ways, here we consider a variant of the approach used by Takriti and Ahmed (2004), where we measure the undesirable deviations away from some target value.

𝑓𝑖(𝑥) =

𝜇𝑖, 𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑜𝑟 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒

∑(𝑥𝑘 − 𝜇𝑖)2

𝑘

𝑘=1

[𝑖𝑘 > 𝜇𝑖], 𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑎𝑡𝑖𝑜𝑛

∑(𝑥𝑘 − 𝜇𝑖)2

𝑘

𝑘=1

[𝑖𝑘 < 𝜇𝑖], 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑎𝑡𝑖𝑜𝑛

where 𝑘 is a scenario, 𝑥𝑘 is the score for the 𝑖-th outcome indicator in scenario 𝑘, and the sum is only taken over the cases that meet the specified condition.

Case and Model

The European Union (EU) has targets for the reduction in carbon emissions and the share of renewable technologies in the total energy production by 2020 (European Commission 2010). The main aim is to reach 20% reduction in carbon emission levels compared to 1990 levels and to increase the share of renewables to at least 20% by 2020. In order to meet the 2020 goals, the EU adopted the European Emissions Trading Scheme (ETS) for limiting the carbon emissions (European Commission 2010). ETS imposes a cap-and-trade principle that sets a cap on the allowed greenhouse gas emissions and an option to trade allowances for emissions. However, current emissions and shares of renewables show a fragile progress of reaching the 2020 targets. Moreover, the energy system includes various uncertainties related to e.g. technology lifetimes, economic growth, costs, learning curves, and investment preferences.



- 5 -

Whether the policy will achieve its targets is at least partly contingent on how these various uncertainties play out.

In this study, a System Dynamics (Forrester 1961; Sterman 2000; Pruyt 2013) model is used for simulating plausible futures of the EU electricity system. The model represents the power sector in the EU and includes congestion on interconnection lines by distinguishing seven different regions in the EU. These are northwest (NW), northeast (NE), middle (M), southwest (SW), and southeast (SE) Europe, United Kingdom and Ireland (UKI), and Italy (I). Nine power generation technologies are included. These are: wind, PV solar, solid biomass, coal, natural gas, nuclear energy, natural gas with Carbon Capture and Sequestration (CCS), coal gasification with CCS, and large scale hydro power. The model includes endogenous mechanisms and processes related to the competition between technology investments, market supply-demand dynamics, cost mechanisms, and interconnection capacity dynamics.

Fig. 1 shows the main sub-models that constitute this model at an aggregate level. These are, installed capacity, electricity demand, electricity price, profitability and levelised costs of electricity. At an aggregated level, there are two main factors that drive new capacity investments: electricity demand and expected profitability. An increase of the electricity demand leads to an increase in the installed capacity, which will affect the electricity price. This will cause a rising demand, in turn resulting in more installed capacity. On the other hand, decreasing electricity prices will lead to lower profitability and less installed capacity, which will result in electricity price increases. Each sub-model has more detailed interactions within itself and with the other sub-models and exogenous variables and these causal relationships drive the main dynamics of the EU electricity system. More detail on the model can be found in Loonen (2012), including a detailed descriptions of all equations and variables.

Fig. 1. The main causal loops in the EU energy model.

We are interested in exploring and analysing the influence of a set of deeply uncertain input variables on the key output variables. In order to explore this uncertainty space, not only parametric but also structural uncertainties are included. For exploring structural uncertainties,



- 6 -

several alternative model formulations have been specified and a switch mechanism is used for switching between these alternative formulations. Parametric uncertainties are explored over pre-defined ranges. Table 1 provides an overview of the uncertainties, 46 in total, that are analysed and their descriptions.

Table 1: Specification of the uncertainties to be explored

Name Description

Economic lifetime For each technology, the average lifetimes are not known precisely. Different ranges for the economic lifetimes are explored for each technology.

Learning curve It is uncertain for different technologies how much costs will decrease with increasing experience. Different progress ratios are explored for each technology.

Economic growth It is deeply uncertain how the economy will develop over time. Six possible developments of economic growth behaviors are considered.

Electrification rate The rate of electrification of the economy is explored by means of six different electrification trends.

Physical limits The effect of physical limits on the penetration rate of a technology is unknown. Two different behaviors are considered.

Preference weights Investor perspectives on technology investments are treated as being deeply uncertain. Growth potential, technological familiarity, marginal investment costs and carbon abatement are possible decision criteria.

Battery storage For wind and PV solar, the availability of (battery) storage is difficult to predict. A parametric range is explored for this uncertainty.

Time of nuclear ban A forced ban for nuclear energy in many EU countries is expected between 2013 and 2050. The time of the nuclear ban is varied between 2013 and 2050.

Price – demand elasticity A parametric range is considered for price – demand elasticity factors.

The general optimization problem we are solving is given below

𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝐹(𝐿) = [𝑓𝑐𝑜𝑠𝑡𝑠, −1 ∗ 𝑓𝑟𝑒𝑛𝑒𝑤𝑎𝑏𝑙𝑒𝑠, −1 ∗ 𝑓𝑟𝑒𝑑𝑢𝑐𝑡𝑖𝑜𝑛]

𝑤ℎ𝑒𝑟𝑒 𝐿 = [𝑙𝑑𝑓 , 𝑙𝑎𝑑 , 𝑙𝑠𝑓 , 𝑙𝑠𝑑 , 𝑙𝑝𝑟 , 𝑙𝑑𝑐𝑓 , 𝑙𝑓𝑡ℎ, 𝑙𝑡𝑟 ]

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 0.5 ≤ 𝑐𝑑𝑓 ≤ 1,

0.0 ≤ 𝑐𝑎𝑑 ≤ 0.75 0.0 ≤ 𝑐𝑠𝑓 ≤ 0.5

0.0 ≤ 𝑐𝑠𝑑 ≤ 20 .0 1.0 ≤ 𝑐𝑝𝑟 ≤ 2.0

0.0 ≤ 𝑐𝑑𝑐𝑓 ≤ 0.5

0.0 ≤ 𝑐𝑓𝑡ℎ ≤ 1.0

10 ≤ 𝑐𝑡𝑟 ≤ 40

Table 2 offers an explanation of each policy lever 𝑙 and the meaning of the subscripts. 𝑓𝑐𝑜𝑠𝑡𝑠, 𝑓𝑟𝑒𝑛𝑒𝑤𝑎𝑏𝑙𝑒𝑠, and 𝑓𝑟𝑒𝑑𝑢𝑐𝑡𝑖𝑜𝑛 are defined in line with the three alternative metrics given in the

preceding section. The constraints 𝑐𝑖 are taken from Hamarat et al. (2014) and are based on common sense and case specific considerations.



- 7 -

Table 2: List of triggers and their descriptions

Trigger Brief Description

Action 1 Desired Fraction (df) Desired fraction of renewable technologies.

Additional Decommissioning (ad) Additional fraction of non-renewable technologies to be decommissioned.

Action 2 Subsidy Factor (sf) Additional fraction of subsidy for renewables.

Subsidy Duration (sd) Duration for how long the subsidy for the renewables will be active.

Proximity (pr) Proximity of cost to the cost of the most expensive non-renewable technology.

Action 3 Decommissioning Factor (dcf) Fraction to be decommissioned for non-renewables when the gap between desired and forecasted fraction for renewables is above the Trigger.

Forecast Time Horizon (fth) Time horizon over which the forecast for the level of renewable fraction is done.

Trigger (tr) Proximity of the forecasted renewable fraction to the desired fraction.

Results

For each of the three metrics we performed an optimization. We first established the number of scenarios required for calculating the robustness. To this end, we ran the model for 10 different randomly generated solutions over 5000 scenarios generated using Monte Carlo sampling. Next, we calculated the robustness metric as a function of 1 to 5000 scenarios. This analysis suggested that around 1000 scenarios, the metric starts to stabilize. Additional scenarios add little value beyond this point. Runtime was on average about 1 week on a 6 core Xeon processor

with hyper threading. To assess whether the optimization has converged, one can look at the 𝜀-progress. This is a measure of how many new dominating solutions are being found. Early in the optimization, frequently new dominating solutions are being found, while over time this slows down. As can be seen in Fig. 2, each of three optimizations was slowing done. Note that the optimization for metric 2 was interrupted early because of runtime concerns. Based on this analysis of the convergence, we know that the solutions found by the algorithm at the end of the optimization are a close approximation of the true Pareto front. We can thus proceed with analysing the individual solutions.



- 8 -

Fig. 2. Analysis of convergence of the optimization for each of the three metrics.

Fig. 3 shows the performance of the solutions identified by each of the three metrics. For both metric 2 and 3, there is a trade-off between objectives 3 and 4, and 5 and 6. Objectives 3 and 5 measure the mean of the carbon emission reduction fraction and costs respectively, while objectives 4 and 6 are the variance metrics. Thus, a high carbon emission reduction coincides with a low score on the variance metric, while an on average cheaper solution has higher variance in the costs.



- 9 -

Fig. 3.Parallel coordinate plot for each metric. The tried solutions are in grey and the colored solutions are the solutions in the Pareto approximate set.

Fig. 3 provides insight into the overall performance of different solutions as found by the different metrics. It does not allow for easy comparison between the different metrics. To this end, we extracted the solutions as found by each metric and evaluated the performance of the combined set of solutions on 1000 scenarios. In this way, we can make a side-by-side comparison of the solutions as found by each of the three metrics Fig. 4 shows the bandwidth of outcomes across the 1000 scenarios for each of the individual solutions for the fraction renewables, the carbon emission reduction, and costs. The distribution of terminal values is shown using a Gaussian Kernel Density estimate. The figures suggest that the three metrics find complementary but not identical solutions. For example, if we look at the fraction of renewables, it appears that metric 1 finds the solutions with on average the best performance in 2050. However, it also appears that metric 3 produces solutions that have less downside dispersion than those found by metric 1.



- 10 -

Fig. 4. Side-by-side comparison of the solutions found by each of the three metrics for fraction renewables, carbon emission reduction and costs. The colors denote the metric, the individual solutions for a given metric are separated by brightness.

In order to get some more insight into the trade-offs between the three outcomes of interest as resulting from the solutions found through using the three robustness metrics, we generated the pairwise scatter plot shown in Fig. 5. We observe that for many scenarios the three metrics produce highly similar outcomes. Still, there are scenarios where the different metrics produce



- 11 -

quite different results. For example if we look at the top right scatterplot of fraction renewables against costs, metric 1 produces solutions with costs in the middle of the range, and a fraction of renewables that is also more or less in the middle. Neither metric 2 nor metric 3 produces this type of outcome. It also appears that the metric 1 has a tendency to stand out from both metrics 2 and 3. Metric 1 appears to produce solutions that show a dispersed performance.

Fig. 5. Pairwise scatterplot of solutions for the three outcomes of interest

In summary, the three metrics produce different but complementary sets of solutions. Metric 1 appears to produce solutions that are effective at reducing the emissions, but with some downside dispersion. They also appear to be more costly than many of the solutions identified



- 12 -

by metrics 2 and 3. The solutions found by metric 2 appear to be outperformed by both metric 1 and 3. This might be due to interrupting the optimization prematurely. Metric 3 produces a wide set of results, some of which are very similar to metric 1 but with less negative dispersion. Subtle differences between metric 1 and 3 can for example be found in the long negative tail on the carbon reduction fraction. Both metric 2 and 3 have the benefit of providing insight into the trade-off between the average performance and the dispersion around this average. Interestingly, in this case, the best performing solutions in terms of reducing carbon emissions are also the solutions with the least dispersion around this outcome. This might be due to the character of the adaptive policy that aggressively aims at curtailing emissions or may be due to the fact that the emission reduction fraction is calculated on absolute values rather than relative values

Based on the results of this single case, it is not possible to state that any of the three metrics is superior, or that a specific metric is highly conservative and hence costly. The main merits of the first metric are that by combining the average performance and the dispersion, it halves the search space. This is thus also the quickest in runtime. For this particular case it produces good results and solutions that are comparable to those found by metrics 2 and 3. Still, not having insight into the trade-off between the average performance and some measure of dispersion is a shortcoming of this metric. Metrics 2 and 3 both provide insight into this trade-off, which can be quite useful in decision-making. In this case study, metrics 2 and 3 are producing very similar results. There is little to separate them clearly. Partly this might be due to incomplete convergence of metric 2. Still, given their similarity in results and no real difference in the complexity of the search space, metric 3 is to be preferred on the theoretical grounds that metric 3 is a monotonically increasing objective function (Ray et al. 2013).

Discussion

The results are based on a single run of the optimization algorithm for each metric. It is good practice to assess the adequacy of solutions found through genetic algorithms by performing several replications. Genetic algorithms exploit stochasticity for effective searching, but this also introduces some randomness in the algorithm. Performing several replications with different random seeds can enhance the confidence that the identified solutions are indeed good approximations of the true Pareto front. Replications where omitted from this work because of time constraints, and will be added in future work

A second direction for improving the results reported here is by tracking the progression of the algorithm using hyper volume calculations. Hyper volume is a complement or alternative to 𝜀-progress. It measures the space spanned by the solutions in the Pareto approximate set. By tracking it over time, one gets insight into how this space is growing over time. Theoretically, hyper volume should converge to an upper limit. Using hyper volume can also be useful for reducing runtime. In the application here, the algorithm was executed for a fixed number of generations after which convergence was manually checked. It is possible to include hyper volume or 𝜀-progress as an alternative or additional stopping criterion. This might result in earlier stopping the algorithm than the current fixed 250 generations.

In this paper we used 𝜀-NSGA-II for solving the multi-objective optimization problem. This is a fairly recent algorithm with good performance characteristics. However, in recent years, progress has been made in solving multi-objective optimization problems relatively quickly using auto-adaptive genetic algorithms that adapt their search behaviour over the course of the

generations. Such auto-adaptive outperform 𝜀-NSGA-II.

Robust optimization requires evaluating the performance of a given solution over a set of scenarios. This creates substantial runtime concerns. In the case reported here, we evaluated



- 13 -

the robustness over a 1000 scenarios. Each of these scenarios requires the running of the simulation model, which takes a few seconds. Finding effective ways of reducing the size of the set of scenarios needed for calculating the robustness metric can help in substantially reducing the calculation time. If we are able to reduce the size of the set from a 1000 to 500, the runtime will also be halved.

Conclusions and Implications

There is a growing awareness of the need for flexible policies that produce satisficing results across a wide range of plausible scenarios. The efficacy of such adaptive policies hinges on the specification of when and how to adapt the policy. Robust optimization can be used as a technique for supporting this specification. As evidenced by the case study, all three alternative robustness metrics results in sets of policies that perform reasonable well across the set of scenarios.

Still, we explored the consequences of each of the three metrics. It appears that at least for this case the differences are quite small. Metric three appears to be quite good in reducing negative dispersion, while still achieving good on average performance. Metric one appears to produce on average the best overall performance, but with more downside dispersion. Metric 2 sits in between. Which metric should analysts use? Answering this question is not straightforward. However our results suggest that time permitting, analysts should at least consider using both metric 1 and metric 3. If runtime is a concern, metric 1 is the best choice. Also, if the number of objectives increases beyond three or four, using metric 3 is troublesome because of the search space becomes very large. In such cases a hybrid approach might be used where for selected indicators metric 3 is used, while for others metric 1 is used.

Regarding the case, there are also some implications. We analysed the European emission trading system and observe that in many scenarios it will be insufficient for achieving the stated goals. There is thus a need to complement the ETS policy with additional actions. It also appears that in the long run, pursuing ambitions emission reduction targets is cost effective. Our results showed that the solutions with the highest reduction of emissions where not the most expensive solutions. This is due to the fact that these solutions push for quick adoption of emission reducing technologies, kick-starting positive feedback loops related to learning effects, resulting in substantial reductions of costs over time for these technologies, and increased effectiveness over time

In this paper we looked at three related metrics. In future work, we would like to expand this by including substantially different metrics such as minimax and other regret based metrics. These might produce more diverging results. Along a different line, we would like to expand this work by looking at other cases. This will help in separating case specific effects from the effects of the metrics.

References

Bai D, Carpenter T, Mulvey J (1997) Making a case for robust optimization models. Management Science 43 (7):895-907

Bankes SC (1993) Exploratory Modeling for Policy Analysis. Operations Research 4 (3):435-449

Bankes SC, Walker WE, Kwakkel JH (2013) Exploratory Modeling and Analysis. Encyclopedia of Operations Research and Management Science, 3rd edn. Springer, Berlin, Germany

Ben Haim Y (2006) Information-Gap Decision Theory: Decision Under Severe Uncertainty. 2nd edn. Academic Press, London, UK



- 14 -

Ben-Tal A, Nemirovski A (1998) Robust convex optimization. Mathematics of Operations Research 23 (4):769-805

Ben-Tal A, Nemirovski A (2000) Robust solutions of linear programming problems contaminated with uncertain data. Mathematical Programming 88 (3):411-424

Bertsimas D, Sim M (2004) The Price of Robustness. Operations Research 52 (1):35-53. doi:10.1287/opre.1030.0065

Bryant BP, Lempert RJ (2010) Thinking Inside the Box: a participatory computer-assisted approach to scenario discovery. Technological Forecasting and Social Change 77 (1):34-49

de Neufville R, Scholtes S (2011) Flexibility in Engineering Design. The MIT Press, Cambridge, Massachusetts

Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. IEEE Transaction on Evolutionary Computation 6 (2):182-197

Dessai S, Hulme M, Lempert RJ, Pielke jr R (2009) Do We Need Better Predictions to Adapt to a Changing Climate? EOS 90 (13):111-112

Erikson EA, Weber KM (2008) Adaptive Foresight: navigating the complex landscape of policy strategies. Technological Forecasting and Social Change 75 (4):462-482

European Commission (2010) Europe 2020: A European Strategy for smart, sustainable and inclusive growth. European Commission, COM (33 2010)

Forrester JW (1961) Industrial Dynamics. MIT Press, Cambridge

Haasnoot M, Kwakkel JH, Walker WE, Ter Maat J (2013) Dynamic Adaptive Policy Pathways: A New Method for Crafting Robust Decisions for a Deeply Uncertain World. Global Environmental Change 23 (2):485-498.

doi:http://dx.doi.org/10.1016/j.gloenvcha.2012.12.006

Hadka D, Reed PM (2013) Borg: An Auto-Adaptive Many-Objective Evolutionary Computing Framework. Evolutionary Computation 21 (2):231-259

Hamarat C, Kwakkel JH, Pruyt E (2013) Adaptive Robust Design under Deep Uncertainty. Technological Forecasting and Social Change 80 (3):408-418. doi:10.1016/j.techfore.2012.10.004

Hamarat C, Kwakkel JH, Pruyt E, Loonen E (2014) An exploratory approach for adaptive policymaking by using multi-objective robust optimization. Simulation Modelling Practice and Theory. doi:10.1016/j.simpat.2014.02.008

Kasprzyk JR, Nataraj S, Reed PM, Lempert RJ (2013) Many objective robust decision making for complex environmental systems undergoing change. Environmental Modelling & Software:1-17. doi:10.1016/j.envsoft.2012.007

Kollat JB, Reed PM (2006) Comparing state-of-the-art evolutionary multi-objective algorithms for long-term groundwater monitoring design. Advances in Water Resources 29 (6):792-807.

doi:http://dx.doi.org/10.1016/j.advwatres.2005.07.010

Kouvalis P, Yu G (1997) Robust Discrete Optimization and its application. Kluwer Academic Publisher, Dordrecht, the Netherlands

Kwakkel JH, Haasnoot M, Walker WE (2014) Developing Dynamic Adaptive Policy Pathways: A computer-assisted approach for developing adaptive strategies for a deeply uncertain world. Climatic Change. doi:10.1007/s10584-014-1210-4

Kwakkel JH, Pruyt E (2013) Exploratory Modeling and Analysis, an approach for model-based foresight under deep uncertainty. Technological Forecasting and Social Change 80 (3):419-431. doi:10.1016/j.techfore.2012.10.005

http://dx.doi.org/10.1016/j.gloenvcha.2012.12.006

http://dx.doi.org/10.1016/j.advwatres.2005.07.010



- 15 -

Kwakkel JH, Walker WE, Marchau VAWJ (2010) Adaptive Airport Strategic Planning. European Journal of Transportation and Infrastructure Research 10 (3):227-250

Lempert RJ, Collins M (2007) Managing the Risk of Uncertain Threshold Response: Comparison of Robust, Optimum, and Precautionary Approaches. Risk Analysis 24 (4):1009-1026

Lempert RJ, Nakicenovic N, Sarewitz D, Schlesinger ME (2004) Chracterizing Climate-Change Uncertainties for Decision-Makers. Climatic Change 65 (1-2):1-9

Lempert RJ, Popper S, Bankes S (2003) Shaping the Next One Hundred Years: New Methods for Quantitative, Long Term Policy Analysis. RAND, Santa Monica, CA, USA

Loonen E (2012) Exploring Carbon Pathways in the EU Power Sector, Using Exploratory System Dynamics Modelling and Analysis to assess energy policy regimes under deep uncertainty. Master, Delft University of Technology, Delft

Mulvey J, Vanderbei RJ, Zenios SA (1995) Robust optimization of large-scale systems. Operations Research 43 (2):264-281

NRC (2009) Informing Decisions in a Changing Climate. National Academy Press,

Parker AM, Srinivasan SV, Lempert RJ, Berry SH (2014) Evaluating simulation-derived scenarios for effective decision support. Technological Forecasting and Social Change. doi:doi:10.1016/j.techfore.2014.01.010

Pruyt E (2013) Small System Dynamics Models for Big Issues: Triple Jump towards Real-World Complexity

Ray PA, Watkins DW, Vogel RM, Kirshen PH (2013) Performance-based evaluation of an improved robust optimization formulation. Journal of Water Resources Planning and Management 140 (6). doi:10.1061/(ASCE)WR.1943-5452.0000389.

Reed PM, Devireddy VK (2004) Groundwater monitoring design: a case study combining epsilon dominance archiving and automatic parametrization for the NSGA-II. In: Coello Coello CA, Lamont GB (eds) Applications of Multi-Objective Evolutionary Algorithms. World Scientific Publishing Co., Singapore,

Reed PM, Kollat JB, Devireddy VK (2007) Using interactive archives in evolutionary multiobjective optimization: A case study for long-term groundwater monitoring design. Environmental Modelling & Software 22:683-692

Sterman JD (2000) Business Dynamics: Systems Thinking and Modeling for a Complex World. McGraw-Hill,

Takriti S, Ahmed S (2004) On robust optimization of two-stage systems. Mathematical Programming 99 (1):109-126. doi:10.1007/s10107-003-0373-y

Weaver CP, Lempert RJ, Brown C, Hall JW, Revell D, Sarewitz D (2013) Improving the contribution of climate model information to decision making: the value and demands of robust decision making frameworks. WIREs Climate Change 4:39-60. doi:10.1002/wcc.202

HOW ROBUST IS A ROBUST POLICY COMPARING … · the case of designing a policy for stimulating the...

Documents

Transcript of HOW ROBUST IS A ROBUST POLICY COMPARING … · the case of designing a policy for stimulating the...