AI for a Responsible Power System

Mathijs de Weerdt

Associate Professor in Algorithmics group, EEMCS

Delft University of Technology

February 21, 2019

My talk in a nutshell

• Artificial Intelligence can (should) contribute towards a more responsible society.

• Algorithmic innovations can tackle concrete AI challenges in the power system.

2 / 27

Current Allocation is NotResponsible

• Part of humanity is starving

• Part of humanity consumes too much

• Running out of some of the resources

Mainly caused by optimization of profit.

We need more responsible allocations:

1 more fair across parties

2 better balance of optimal now versuslong-term effects

Does this rise new algorithmic challenges?

Doughnut Economicsby Kate Raworth [2017]

www.kateraworth.com/3 / 27

Current Allocation is NotResponsible

• Part of humanity is starving

• Part of humanity consumes too much

• Running out of some of the resources

Mainly caused by optimization of profit.

We need more responsible allocations:

1 more fair across parties

2 better balance of optimal now versuslong-term effects

Does this rise new algorithmic challenges?

Doughnut Economicsby Kate Raworth [2017]

www.kateraworth.com/3 / 27

Scientific gap

allocation decision support

with interaction

simple well understood

(behavioural) game theory

complex AI & algorithmics

algorithmic game theory

Algorithmic game theory:

• Fair allocation under strict conditions [Bergemann and Valimaki, 2010, Parkeset al., 2010]

• For relevant settings: impossibility theorems [Satterthwaite, 1975].

But these are situations we encounter —and deal with— in practice.

• Can algorithms and AI help to improve this current practice with respect toefficiency, fairness, and longer term consequences. . . ?

• Let’s look at some more concrete computational challenges in the electricity grid.

4 / 27

Scientific gap

allocation decision support with interactionsimple well understood (behavioural) game theorycomplex AI & algorithmics algorithmic game theory

Algorithmic game theory:

• Fair allocation under strict conditions [Bergemann and Valimaki, 2010, Parkeset al., 2010]

• For relevant settings: impossibility theorems [Satterthwaite, 1975].

But these are situations we encounter —and deal with— in practice.

• Can algorithms and AI help to improve this current practice with respect toefficiency, fairness, and longer term consequences. . . ?

• Let’s look at some more concrete computational challenges in the electricity grid.

4 / 27

Scientific gap

allocation decision support with interactionsimple well understood (behavioural) game theorycomplex AI & algorithmics algorithmic game theory

Algorithmic game theory:

• Fair allocation under strict conditions [Bergemann and Valimaki, 2010, Parkeset al., 2010]

• For relevant settings: impossibility theorems [Satterthwaite, 1975].

But these are situations we encounter —and deal with— in practice.

• Can algorithms and AI help to improve this current practice with respect toefficiency, fairness, and longer term consequences. . . ?

• Let’s look at some more concrete computational challenges in the electricity grid.

4 / 27

Today’s Electricity System

Today’s Electricity System

• Electricity systems support the generation, transport and use of electrical energy.

• They are large and complex and provide for everyone.

Energy generated = energy consumed at all times

How it used to be. . .• demand is predictable (at an aggregate level)

• which generators are used is decided one day in advance (unit commitment)

• minor corrections are made, based on frequency (primary control, secondarycontrol, etc.)

• a market with few actors (energy retailers)

6 / 27

The Energy Transition

The Energy Transition

Changes in the power system

• renewable energy is• intermittent• uncertain• uncontrollable• sometimes located in the distribution grid, and• has virtually no marginal costs

• new loads such as heat pumps, airconditioning,and electric vehicles are• significantly larger than other household

demand, and• more flexible (and therefore also less

predictable)

These new loads can also be part of the solution!commons.wikimedia.org/wiki/File:Electric_Car_recharging.jpg

9 / 27

Consequences for the main stakeholders

Focus on (computational) challenges regarding

1 Wholesale market operators and system operators

2 Aggregators of flexible demand

3 Distribution network operators

10 / 27

Challenges in Wholesale Market Design

Challenges for Market Operators/Regulators and ISO/TSO

1 more accurate models for bidding and market clearing• use finer granularity, power-based instead of energy-based (Philipsen et al., 2018)• deal with intertemporal dependencies caused by flexible shiftable loads• model stochastic information explicitly

but reasonable models are non-linear: interesting optimization problem

2 allow smaller, local producers and flexible loads (scalability)

11 / 27

Aggregators of flexible demand

New flexible loads can be used to match renewable generation, but

• consumers do not want to interact with the market, and

• markets do not want every consumer to interact.

Challenges for Aggregators (a new role!): demand-side management

1 design mechanism to interact with consumers with flexible demand

2 interact with both wholesale markets and distribution service/network operator

3 optimize use of (heterogeneous) flexible demand under uncertain prices anduncertain consumer behavior

12 / 27

Challenges for DSOs

Aim to avoid unnecessary network reinforcement by demand side management toresolve congestion and voltage quality issues

Challenges for Distribution network system operators

1 (Close to) real-time coordination of generation, storage and flexible loads ofself-interested agents to stay within network capacity limitations:• more agents than in traditional energy market• interaction with wholesale markets• communication may not be always reliable• more complex power flow computations (losses and limitations more relevant in

distribution)

2 Long-term decision making under uncertainty

Some of these challenges we take up in our research.

13 / 27

Challenges for DSOs

Aim to avoid unnecessary network reinforcement by demand side management toresolve congestion and voltage quality issues

Challenges for Distribution network system operators

1 (Close to) real-time coordination of generation, storage and flexible loads ofself-interested agents to stay within network capacity limitations:• more agents than in traditional energy market• interaction with wholesale markets• communication may not be always reliable• more complex power flow computations (losses and limitations more relevant in

distribution)

2 Long-term decision making under uncertainty

Some of these challenges we take up in our research.

13 / 27

Research on Responsible Multi-Party Optimization

Mission: to design (and understand fundamental properties of) planning andcoordination algorithms for responsible optimization across organizational boundaries

Scientific challenges in responsible multi-party optimization

• efficiency (optimality) and scalability,

• fairness, and

• accounting for both long- and short-term effects.

Example: Using Flexibility of Heat Pumps to Prevent Congestion (from the perspectiveof an aggregator working closely with network operator)

14 / 27

Research on Responsible Multi-Party Optimization

Mission: to design (and understand fundamental properties of) planning andcoordination algorithms for responsible optimization across organizational boundaries

Scientific challenges in responsible multi-party optimization

• efficiency (optimality) and scalability,

• fairness, and

• accounting for both long- and short-term effects.

Example: Using Flexibility of Heat Pumps to Prevent Congestion (from the perspectiveof an aggregator working closely with network operator)

14 / 27

Heat Pumps to Prevent Congestionwith Frits de Nijs, Erwin Walraven, and Matthijs Spaan [de Nijs et al., 2015, 2017, 2018a,b, 2019]

De Teuge (near Zutphen)

• pilot sustainable district in 2003

• heatpumps for heating

But: at peak (cold) times, overload of electricity infrastructure

15 / 27

Potential Solutions

1 Reinforce network to cope with peak load

2 Optimal scheduling of demand

3 Re-allocation and online coordination

4 Pre-allocation and minimizing violations

16 / 27

2. Optimal SchedulingFormulate as a mixed integer problem (MIP)

• decide when to turn on or off heat pump

• minimise discomfort (fair: squared distance to temperature set point)

• subject to physical characteristics and capacity constraint

MIP formulation

minimize[ act0 act1 ··· acth ]

h∑t=1

cost(θt ,θsett ) (discomfort)

subject to θt+1 = temperature(θt , actt , θoutt )

n∑i=1

acti,t ≤ capacityt

acti,t ∈ [off,on] ∀i , t

This scales poorly (binary decision variables: houses × time slots).But that is not the only problem. . .

17 / 27

2. Optimal SchedulingFormulate as a mixed integer problem (MIP)

• decide when to turn on or off heat pump

• minimise discomfort (fair: squared distance to temperature set point)

• subject to physical characteristics and capacity constraint

MIP formulation

minimize[ act0 act1 ··· acth ]

h∑t=1

cost(θt ,θsett ) (discomfort)

subject to θt+1 = temperature(θt , actt , θoutt )

n∑i=1

acti,t ≤ capacityt

acti,t ∈ [off,on] ∀i , t

This scales poorly (binary decision variables: houses × time slots).But that is not the only problem. . .

17 / 27

3. Re-allocation and online coordinationNot everything is known in advance (heat loss, available capacity), so adaptation maybe required

Arbitrage with Best Response (BR)

1 Plan each thermostat individually, as if unconstrained.

2 Look at the expected plan utility to determine action

3 Determine resource costs per time slot

4 Re-plan including these costs

→ Iterative process

→ Inspired by Brown’s Fictitious Play

Keep all past realizations to ensure convergence

18 / 27

3. Re-allocation and online coordinationNot everything is known in advance (heat loss, available capacity), so adaptation maybe required

Arbitrage with Best Response (BR)

1 Plan each thermostat individually, as if unconstrained.

2 Look at the expected plan utility to determine action

3 Determine resource costs per time slot

4 Re-plan including these costs

→ Iterative process

→ Inspired by Brown’s Fictitious Play

Keep all past realizations to ensure convergence

18 / 27

3. Re-allocation and online coordination

Simulation of anextreme scenario

• 182 households

• almost nocapacity forheating 18–24hours

Close to lower boundon optimal (relaxedMIP)

19

20

21

22

0 6 12 18 24 30 36 42 48

0.00

0.25

0.50

0.75

1.00

0 6 12 18 24 30 36 42 48

0

50

100

150

0 6 12 18 24 30 36 42 48

Solver

No planning

arbitrage−BR

Relaxed MIP

19

20

21

22

0 6 12 18 24 30 36 42 48

0.00

0.25

0.50

0.75

1.00

0 6 12 18 24 30 36 42 48

0

50

100

150

0 6 12 18 24 30 36 42 48

Solver

No planning

arbitrage−BR

Relaxed MIP

Dev

ices

on(#

)

Avg

.T

emp

.(◦

C)

Cu

mu

lati

vep

enal

ty

Time (h) 19 / 27

3. Re-allocation and online coordination

Simulation of anextreme scenario

• 182 households

• almost nocapacity forheating 18–24hours

Close to lower boundon optimal (relaxedMIP)

19

20

21

22

0 6 12 18 24 30 36 42 48

0.00

0.25

0.50

0.75

1.00

0 6 12 18 24 30 36 42 48

0

50

100

150

0 6 12 18 24 30 36 42 48

Solver

No planning

arbitrage−BR

Relaxed MIP

19

20

21

22

0 6 12 18 24 30 36 42 48

0.00

0.25

0.50

0.75

1.00

0 6 12 18 24 30 36 42 48

0

50

100

150

0 6 12 18 24 30 36 42 48

Solver

No planning

arbitrage−BR

Relaxed MIP

Dev

ices

on(#

)A

vg.

Tem

p.

(◦C

)C

um

ula

tive

pen

alty

Time (h) 19 / 27

3. Re-allocation and online coordination: runtime

1 2 3 4 5 6 10 15 20 25 30 35 40 45

10−2

10−1

100

101

102

Solver

arbitrage−BR

Optimal MMDP

Optimal MIP

Ru

nti

me

(s)

Agents (n) Horizon (h)

Close to optimal, good scaling, but reliable communication is essential. . .

Pre-allocate time-dependent limits per agent that limit probability of violations.

20 / 27

4. Pre-allocation and minimizing violations

Robust pre-allocation of available capacity (Wu and Dur-fee, 2010)

Frequen

cy

L

Satisfying Limits in Expectation (CMDP, Altman 1999)

Frequen

cy

L

Reduced Limits with Hoeffding’s inequality given α

Frequen

cy

L∗ L

Dynamic Relaxation of Reduced Limits by Simulation(De Nijs et al., 2017–2018)

Frequen

cy

L∗ L L

21 / 27

Simulations and Extensions

• Dynamic pre-allocation: efficient, tightlybounded violation probability, and scalable

Extensions• Re-run in case new information (and

communication) is available

• Include wholesale electricity prices to useflexibility for balancing (within constraints)

• Include electric vehicles (and other flexibleloads)

Toolbox available (soon):github.com/AlgTUDelft/ConstrainedPlanningToolbox

PhD Defense and seminar: April 4, 2019

Alg.MILPLDD+GAPS

CMDPHoeffding (CMDP), α = 0.05

Dynamic (CMDP), α = 0.005Dynamic (CG), α = 0.005

100

8

16

32

64

TCL, h = 24

Vio

latio

ns

0.005

0.050

0.5001.000

10−2

10−1

100

101

102

103

4 8 16 32 64 128 256

Num. Agents

% E

x.

Va

lue

Ru

ntim

e (

s.)

Alg.MILPLDD+GAPS

CMDPHoeffding (CMDP), α = 0.05

Dynamic (CMDP), α = 0.005Dynamic (CG), α = 0.005

100

8

16

32

64

TCL, h = 24

Vio

latio

ns

0.005

0.050

0.5001.000

10−2

10−1

100

101

102

103

4 8 16 32 64 128 256

Num. Agents

% E

x. V

alu

eR

untim

e (

s.)

22 / 27

Simulations and Extensions

• Dynamic pre-allocation: efficient, tightlybounded violation probability, and scalable

Extensions• Re-run in case new information (and

communication) is available

• Include wholesale electricity prices to useflexibility for balancing (within constraints)

• Include electric vehicles (and other flexibleloads)

Toolbox available (soon):github.com/AlgTUDelft/ConstrainedPlanningToolbox

PhD Defense and seminar: April 4, 2019

Alg.MILPLDD+GAPS

CMDPHoeffding (CMDP), α = 0.05

Dynamic (CMDP), α = 0.005Dynamic (CG), α = 0.005

100

8

16

32

64

TCL, h = 24

Vio

latio

ns

0.005

0.050

0.5001.000

10−2

10−1

100

101

102

103

4 8 16 32 64 128 256

Num. Agents

% E

x.

Va

lue

Ru

ntim

e (

s.)

Alg.MILPLDD+GAPS

CMDPHoeffding (CMDP), α = 0.05

Dynamic (CMDP), α = 0.005Dynamic (CG), α = 0.005

100

8

16

32

64

TCL, h = 24

Vio

latio

ns

0.005

0.050

0.5001.000

10−2

10−1

100

101

102

103

4 8 16 32 64 128 256

Num. Agents

% E

x. V

alu

eR

untim

e (

s.)

22 / 27

Simulations and Extensions

• Dynamic pre-allocation: efficient, tightlybounded violation probability, and scalable

Extensions• Re-run in case new information (and

communication) is available

• Include wholesale electricity prices to useflexibility for balancing (within constraints)

• Include electric vehicles (and other flexibleloads)

Toolbox available (soon):github.com/AlgTUDelft/ConstrainedPlanningToolbox

PhD Defense and seminar: April 4, 2019

Alg.MILPLDD+GAPS

CMDPHoeffding (CMDP), α = 0.05

Dynamic (CMDP), α = 0.005Dynamic (CG), α = 0.005

100

8

16

32

64

TCL, h = 24

Vio

latio

ns

0.005

0.050

0.5001.000

10−2

10−1

100

101

102

103

4 8 16 32 64 128 256

Num. Agents

% E

x.

Va

lue

Ru

ntim

e (

s.)

Alg.MILPLDD+GAPS

CMDPHoeffding (CMDP), α = 0.05

Dynamic (CMDP), α = 0.005Dynamic (CG), α = 0.005

100

8

16

32

64

TCL, h = 24

Vio

latio

ns

0.005

0.050

0.5001.000

10−2

10−1

100

101

102

103

4 8 16 32 64 128 256

Num. Agents

% E

x. V

alu

eR

untim

e (

s.)

22 / 27

Our Other Contributions to the Smart Grid

• Scheduling of charging of electric vehicles:• Computational complexity [de Weerdt et al., 2018]• Stochastic optimization [van der Linden et al., 2018]• En-route charging [de Weerdt et al., 2016]

• Market design:• Auction design for DC grids [Piao et al., 2018, 2017]• Design errors in existing markets [Philipsen et al., 2019]

• Combined: Online scheduling mechanism for flexible loads [Strohle et al., 2014]

• Long-term investments: Unbounded MDPs [Neustroev et al., 2019]

23 / 27

Conclusions

Concluding Observations

• AI and algorithms can contribute to aresponsible operation of commoninfrastructure

• More research required on fairness,incentives in market design, and effects onthe future!

Slides are available from

www.alg.ewi.tudelft.nl/weerdt/

Thanks a lot to Matthijs, Frits, Erwin, Rens, Laurens, German, Natalia, Greg, Koos,Anna, Longjian, Neil, Gleb, Jedlix, Alliander, Eneco, and all my (other) students andcollaborators!

24 / 27

References I

D. Bergemann and J. Valimaki. The dynamic pivot mechanism. Econometrica, 78(2):771–789, 2010.

F. de Nijs, M. Spaan, and M. de Weerdt. Best-Response Planning of Thermostatically ControlledLoads under Power Constraints. In Proceedings of the 29th AAAI Conference on ArtificialIntelligence, pages 615–621, 2015.

F. de Nijs, E. Walraven, M. T. Spaan, and M. M. de Weerdt. Bounding the Probability of ResourceConstraint Violations in Multi-Agent MDPs. Proc. AAAI 2017, 2017.

F. de Nijs, M. Spaan, and M. M. de Weerdt. Preallocation and Planning under Stochastic ResourceConstraints. In Proceedings of the 32th AAAI Conference on Artificial Intelligence. Association forthe Advancement of Artificial Intelligence (AAAI), Jan. 2018a.

F. de Nijs, G. Theocharous, N. Vlassis, M. de Weerdt, and M. Spaan. Capacity-aware SequentialRecommendations. In Proceedings of the 17th International Conference on Autonomous Agents andMultiagent Systems, pages 416–424. International Foundation for Autonomous Agents andMultiagent Systems (IFAAMAS), July 2018b.

F. de Nijs, M. T. J. Spaan, and M. M. de Weerdt. Multi-agent Planning Under Uncertainty forCapacity Management. In P. Palensky, M. Cvetkovic, and T. Keviczky, editors, Intelligent IntegratedEnergy Systems, pages 197–213. Springer, Cham, 2019.

25 / 27

References IIM. de Weerdt, M. Albert, V. Conitzer, and K. van der Linden. Complexity of Scheduling Charging in

the Smart Grid. In Proceedings of the Twenty-Seventh International Joint Conference on ArtificialIntelligence, pages 4736–4742, California, July 2018. International Joint Conferences on ArtificalIntelligence (IJCAI).

M. M. de Weerdt, S. Stein, E. H. Gerding, V. Robu, N. R. Jennings, and M. M. de Weerdt.Intention-aware routing of electric vehicles. IEEE Transactions on Intelligent TransportationSystems, 17(5):1472–1482, 2016.

G. Neustroev, M. M. de Weerdt, and R. A. Verzijlbergh. Discovery of Optimal Solution Horizons inNon-Stationary Markov Decision Processes with Unbounded Rewards. In Proceedings of theInternational Conference on Planning and Scheduling (ICAPS’05), 2019.

D. C. Parkes, R. Cavallo, F. Constantin, and S. Singh. Dynamic incentive mechanisms. AI Magazine,31(4):79–94, 2010.

R. Philipsen, G. Morales-Espana, M. de Weerdt, and L. de Vries. Trading power instead of energy inday-ahead electricity markets. Applied Energy, 233-234:802–815, 2019.

L. Piao, M. de Weerdt, and L. de Vries. Electricity market design requirements for DC distributionsystems. In 2017 IEEE Second International Conference on DC Microgrids (ICDCM, pages 95–101.IEEE, 2017.

26 / 27

References III

L. Piao, L. de Vries, M. de Weerdt, and N. Yorke-Smith. Electricity Market for Direct CurrentDistribution Systems: Exploring the Design Space. In Proceedings of the 15th InternationalConference on the European Energy Market, EEM’18, pages 1–5, United States, 2018. IEEE.

K. Raworth. Doughnut Economics. Seven Ways to Think Like a 21st-Century Economist. 2017.

M. A. Satterthwaite. Strategy-proofness and Arrow’s conditions: Existence and correspondencetheorems for voting procedures and social welfare functions. Journal of economic theory, 10(2):187–217, 1975.

P. Strohle, E. H. Gerding, M. M. de Weerdt, S. Stein, and V. Robu. Online mechanism design forscheduling non-preemptive jobs under uncertain supply and demand. In Proceedings of the 2014international conference on Autonomous agents and multi-agent systems, pages 437–444, 2014.

K. van der Linden, M. de Weerdt, and G. Morales-Espana. Optimal non-zero Price Bids for EVs inEnergy and Reserves Markets using Stochastic Optimization. In Proceedings of the 15thInternational Conference on the European Energy Market, EEM 2018, pages 1–5, United States,2018. IEEE.

27 / 27