Learned helplessness and generalization

Falk Lieder (lieder@biomed.ee.ethz.ch)Translational Neuromodeling Unit; University of Zurich, & ETH Zurich, Zurich, Switzerland

Noah D Goodman (ngoodman@stanford.edu)Department of Psychology; Stanford University, Stanford, CA, USA

Quentin JM Huys (qhuys@biomed.ee.ethz.ch)Translational Neuromodeling Unit; University of Zurich, & ETH Zurich and

Psychiatric University Hospital Zurich, Zurich, Switzerland

Abstract

In learned helplessness experiments, subjects first expe-rience a lack of control in one situation, and then showlearning deficits when performing or learning anothertask in another situation. Generalization, thus, is at thecore of the learned helplessness phenomenon. Substan-tial experimental and theoretical effort has been investedinto establishing that a state- and task-independent be-lief about controllability is necessary. However, to whatextent generalization is also sufficient to explain thetransfer has not been examined. Here, we show qual-itatively and quantitatively that Bayesian learning ofaction-outcome contingencies at three levels of abstrac-tion is sufficient to account for the key features of learnedhelplessness, including escape deficits and impairment ofappetitive learning after inescapable shocks.

IntroductionHelplessness is a failure to avoid punishment or obtainrewards even though they are under the agent’s con-trol. The aetiology and consequences of helplessnesshave been studied extensively in the animal learning lit-erature using the learned helplessness paradigm [1]. Inthis paradigm, helplessness is induced in healthy animalsby exposure to inescapable electric shocks. Helplessnessis then measured by the subsequent failure to escape es-capable shocks in a novel environment. The phenomenonwas first demonstrated in dogs in the context of testingthe two-factor learning theory using the shuttle-box es-cape task [2]. In the now classical version of the task,three animals are compared. A master rat experienceselectrical shocks. These come on unpredictably, but canbe terminated by some action, for instance turning awheel. A yoked rat is exposed to the exact same sequenceof shocks that are delivered to the master rat, but hasno action available to terminate the shocks. A third rat,the control, is not exposed to shocks. Compared to thecontrols, the yoked rats are impaired at acquiring newinstrumental responses, but the master rats are eitherunimpaired or may even show an improvement [1,3]. Ef-fects reminiscent of those in animal learning have beendemonstrated in humans. For instance, people who havebeen exposed to uncontrollable loud noise or insolubleproblems are more likely give up on solving anagrams ina subsequent task [4]. Hopelessness theory is a trans-lation of the helplessness concepts to the human andattributional realm [5].

Extensive animal and human experimentation andtheoretical work have clarified that the crucial compo-

nent is a perceived lack of control rather than more spe-cific explanations. First, exposure to one type of uncon-trollable reinforcer in one situation can profoundly im-pair the acquisition of many other types of behaviours(including escape, jumping, immobility, lever pressing,and complex sequential behaviours) in a wide range ofdifferent situations [1]. Second, the effect of inescapableshocks is not due to shock-induced analgesia, because italso impairs reward seeking in the absence of negativereinforcers [6]; and as uncontrollable rewards can alsoinduce helplessness [7–9]. Third, it is not reducible toan interference between learned motor inactivity as italso affects the ability to acquire behavioural suppres-sion as an escape response [10]. Furthermore, in ac-cordance with the finding that the probability of gen-eralization increases with the variability of the examples(see [11]), learned helplessness lasts longer when it is in-duced by experiencing multiple mild stressors (chronicmild stress; [12]) than when it is induced by a single se-vere stressor. In conclusion, there is strong evidence sug-gesting that helplessness is learned by generalizing fromone uncontrollable situation to believing that situationsare uncontrollable in general.

Maier and Seligman formalized controllability in termsof the conditional probability of the reinforcer RF (re-ward or punishment) given whether or not action A istaken (A or A) [1]. According to their definition, theagent has control if and only if P (RF|A) 6= P (RF|A).Huys and Dayan formalized the essence of this defini-tion, i.e. that helplessness exists when altering behaviourdoes not alter outcomes, for multiple actions, outcomes,and degrees of desirability [13]. Here we continue [13]’sargument that generalization is central to learned help-lessness by investigating two crucial interactions betweenperceived control and generalization:

1. Learning about the controllability of one situationtransfers to novel situations via generalization.

2. Abstract knowledge about control determines howstrongly the observation that a particular action hada particular effect will be generalized.

Using a hierarchical Bayesian model of action-outcome-contingencies we show that these interactions are suffi-cient to explain various deficits observed in the learnedhelplessness paradigm.

Figure 1: Hierarchical Bayesian model of state-transitions and controllability. θs,a are the transitionprobabilities for taking action a in state s (level I). Thesecond level, abstracts away from particular actions andrepresents the general outcome tendency βs of situations and its controllability αs. The third level abstractsaway from any particular state and represents how con-trollable the world is in general (c) and how much statesdiffer with respect to controllability (σ2

c ).

Methods

In a novel situation s, a rational agent may have to learnhow likely each of the available actions a1, · · · , am is tolead into each of the potential successor states s1, · · · , sn.In the absence of knowledge about the particular situa-tion s, the agent can bring experience in other situationss′ to bear on the problem, i.e. it can use its knowl-edge about one part of the transition matrix to informits belief about others. Hierarchical Bayesian formula-tions provide a normative framework for such general-izations [11,14]. In this section, we present such a modelof state-transition probabilities with three levels of hi-erarchy (see Figures 1 and 2). At the lowest level arethe probabilities that taking action a in state s will leadto state s′. At the second level, the model representsthe typical outcome probabilities of actions in any oneparticular situation s and how different the outcomes ofdifferent actions tend to be. The more actions are be-lieved to have similar outcomes, the less control thereis. At the third and most abstract level, the model rep-resents knowledge about how controllable situations arein general. In this model beliefs about the world’s con-trollability acts as an over-hypothesis that shapes howthe agent learns state-transition probabilities (cf. [14]).Concretely, the agent’s belief about the state St+1 result-ing from taking action a in state s is a multinomial distri-bution (Equation 1). The agent assumes that the transi-tion probabilities θa,s of the actions a available in states are all drawn from the same distribution: a Dirichletdistribution with the state-specific mean vector βs and

∀a, s : St+1|St = s,At = a ∼ Multinomial (θa,s) (1)

∀a, s : θa,s ∼ Dirichlet (αs · βs) (2)

∀s : − log(αs) ∼ N(c, σ2

c

)(3)

∀s : βs ∼ Dirichlet (1) (4)

c ∼ N (µ, σ2µ) (5)

σ2c ∼ InvGamma(ασ, βσ) (6)

Figure 2: The functional dependencies of the graphicalmodel in Figure 1.

a second parameter αs that determines the controllabil-ity of situation s (Equation 2). If αs goes to ∞, thenthe agent becomes sure that the transition probabilitiesθa1,s, · · · ,θaN ,s are independent of the agent’s action a.This means that the situation is uncontrollable (corre-sponding to the second notion of controllability in [13]).Values of αs close to zero corresponds to the belief thatthe transition probabilities θa1,s, · · · ,θaN ,s for differentactions are uninformative about each other and hencecan differ. To allow for the transfer of knowledge be-tween states, a further level is needed: in addition to itsbelief about the controllability αs of individual situationss, the agent also has a belief about how controllable situ-ations are in general. This belief is described by a normaldistribution on − log(αs) (Equation 3). The parameterc = E[− log(α)] expresses how controllable situations areon average, and σ2

c expresses how much controllabilityvaries from situation to situation.

The average controllability c and the variability of con-trol σ2

c are unknown properties of the world that haveto be learned from experience. We describe the agent’sprior beliefs about these quantities by a normal distri-bution on c (Equation 5) and an Inverse-Gamma distri-bution on σ2

c (Equation 6). In this model helplessnessresults from a probabilistic belief that one’s control overthe world is low on average (low c) and varies very littleacross situations (low σ2

c ).

Assuming that this hierarchical Bayesian model cap-tures the subjects’ internal representation of transitionprobabilities and control, we can examine how they inferthe controllability c of the world in general from the ob-servations o = {(s1, a1, s2) , · · · , (st−1, at−1, st)} of thestate transitions (s1, s2), · · · , (st−1, st) and their actionsa1, · · · , at. In addition, we can simulate the weakergeneralization on the situation-specific controllability αsand transition-tendency βs by computing P (αs,βs|o).Finally, we can investigate how this generalization isshaped by abstract beliefs about control (P (c, σ2

c )).

Simulations The effects of controllable and uncontrol-lable shocks were modelled by simulating the learningprocess taking place during the shocks as Bayesian in-ference on c and σ2

c . The naive subjects’ belief aboutcontrollability was modelled by a probability distribu-tion with E[α] = 10 and Var[α] = 100; the variance ofthe prior beliefs was 100 for c and 0.1 for σ2

c . To modelthe observations resulting from controllable (oc) versusuncontrollable shocks (o¬c), we assumed one observation

per second during 64 shocks lasting 60 seconds each. Forcontrollable shocks there was one action (a1) that wouldalways terminate the shock (s1 → s2) and four actionsthat did not (s1 → s1), whereas there was no such actionfor uncontrollable shocks.

Learning after exposure to shocks was modelled asBayesian inference on α,β given c = E[c|o¬c] and σ2

c =E[σ2

c |o¬c] for uncontrollable shocks versus c = E[c|oc]and σ2

c = E[σ2c |oc] for controllable shocks.

Inference was performed using Markov chainMonte-Carlo (MCMC) methods. To sample fromP (α,β, c, σ2

c |o), we used a Metropolis-Hastingsalgorithm with Gaussian random-walk proposalson c and − log(α). The proposal for βt+1 wasDirichlet(10n · βt) where n is the number of states,and the proposal for σ2

c,t+1 was an Inverse-Gamma

distribution with mean σ2c,t and variance 1. 50 Markov

chains were run for 51000 iterations with a burn-inperiod of 1000 iterations. P (α,β|c, σ2

c ,o) was com-puted in the same way. The posterior expectationof θ was computed using Monte-Carlo integration:E[θ|o, c, σ2

c ] =∫E[θ|α, β,o] · p(α,β|o, c, σ2

c )dαdβ ≈1mΣmi=1E[θ|αi,βi,o] with αi,βi ∼ P (α,β|o, c, σ2

c ).

Results

As a first step in assessing whether generalization canaccount for the differential effects of controllable ver-sus uncontrollable stress, we simulated Bayesian learningfrom these experiences according to the model shown inFigure 1. Figure 3A shows the simulated changes inperceived controllability induced by the escapable andinescapable shocks administered in the learned help-lessness paradigm [15]. After inescapable shocks, thesubjects’ perceived control c was reduced, whereas con-trollable shocks increased it. Furthermore, control-lable shocks increased the estimated variability of con-trol across situations, whereas no such change was ob-served after inescapable shocks (Figure 3B). Thus, thetwo kinds of shocks have opposite effects on the subjects’high-level beliefs about controllability.

We next asked whether the beliefs induced by uncon-trollable shocks were sufficient to impair escape learn-ing in a different task, and whether the beliefs inducedby controllable shocks have the opposite (mastery) ef-fect. We modelled these beliefs by the inferred meanand variance of c for escapable and inescapable shocksand simulated learning in the shuttle-box escape task.As a first step, we simulated learning from given obser-vations with one action that did (a1) and four actionsthat did not cancel the shock (a2, · · · , a5). Concretely,we simulated how strongly naive subjects, subjects whohad experienced inescapable shocks (yoked), and sub-jects who had experienced escapable shocks (masters)would believe that action a1 cancels the shock after hav-ing taken action a1 for 0, 1, 2, 4, 8, 16 or 32 times andeach of the four other actions 8 times. Figure 4 showsthat yoked subjects (red) acquired the escape responsemore slowly than naive subjects (blue): more evidencewas required before they believed that the action was ef-ficient in terminating the shock. Furthermore, the model

Figure 3: A: Expected controllability learned from con-trollable or uncontrollable electric shocks. The valueson the x-axis are the change ∆c relative to the con-trollability expected by naive subjects and height of thebars shows how strongly the simulated agent beliefs inthe corresponding value of c. B: Variance of controlla-bility learned from controllable and uncontrollable elec-tric shocks. The values on the x-axis are the change∆c relative to the variability expected by naive subjectsand height of the bars shows how strongly the simulatedagent beliefs in the corresponding value of σ2

c .

Figure 4: Simulated effects of controllable and uncon-trollable shocks on the speed of learning that action 1terminates the shock.

also captured mastery effects, whereby prior exposure tocontrollable shocks leads to faster learning (green; [16]).

To more quantitatively relate the learning dynamicsshown in Figure 4 to empirical data, we simulated learn-ing and decision making in the fixed-ratio operant con-ditioning task of [17]. In this task, rats have to learnto press a lever, but only every third lever press termi-nates the shock. This task was modelled as sequential-decision making. To do so, we partitioned the 60 sec-onds of each trial in [17]’s experiment into 30 bins, each2 seconds long, and simulated one decision, one observa-tion, and one belief update for every bin. The simulatedchoices were to stay still (a0), to push the lever (a1),or to perform a different action (a2). The reward forstaying still and receiving the shock was modelled as -1(r(s1, a0, s1) = −1). Moving and receiving a shock wasassumed to incur a small additional cost (r(s1, a2, s1) =−1.2). If the action stopped the shock, it was assumedto incur only the cost of movement (r(s1, a0, s2) = 0 andr(s1, ai, s2) = −0.2 for i ∈ {1, 2}). We assumed that rats

Figure 5: The left panel shows empirical data acquiredby [17] as shown in [1]. The plots on right show oursimulations of the experiment.

learn the probability P (St+1 = s2|St = s1, At = ai) thatan action ai terminates the shock (an alternative modelmight consider treating different numbers of lever pressesas separate actions). The subjects’ internal representa-tion of transition probabilities was modelled as the hi-erarchical Bayesian model shown in Figure 1. The rat’sdecision making was simulated by a sampling algorithmto produce behaviour akin to probability-matching [18].Specifically, we assumed that the rat simulates five out-comes ui,j = r(s, ai, s

′j), s

′j ∼ P (St+1|St = s,At = ai)

of each action ai and chooses the action ai for whichthe average utility 1

5

∑5j=1 ui,j was largest, and ties were

broken at random. Under these assumptions, the learn-ing dynamics shown in Figure 4 capture the qualitativeeffects of uncontrollable shocks on the probability to es-cape shock and the time required to do so [17]: yokedsubjects failed to escape more often than naive subjects(Figure 5, left panel), and when they succeeded to escapeit took them longer (Figure 5, right panel). Furthermore,our model accounts for the mastery effect that rats whohad been exposed to controllable shocks prior to the task,escaped faster than rats with no prior exposure to shock.

As outlined in the introduction, learned helplessnessimpairs not only the ability to learn from punishmentsbut also from rewards. To assess whether our modelcaptures this effect, we simulated the experiment by [6].In the experiment’s appetitive choice task, rats were re-warded with food for going into one of two chambersafter they had been trained to prefer the other cham-ber. We modelled this task as a sequence of decisions,observations, and belief updates as described above. AsFigure 6 shows our model captures that uncontrollableshocks reduced the probability that a rat would first seekout the chamber in which a reward would be delivered.Thus this apparently anhedonic behaviour can be ex-plained purely in terms of impaired associative learningdue the generalization that the world is uncontrollable.

Next, we asked whether our model can account forthe finding that the effect of learned helplessness is mostpronounced in tasks that are complex and require per-sistence. To answer this question, we simulated decision

Figure 6: Simulation of the appetitive choice distinctiontask by [6]. Our simulation captures that yoked ratsperformed worse than naive rats across all 10 blocks ofthe experiment.

Figure 7: Simulation of the experiment by [19]. Dashedlines are model predictions; diamonds are data points.The three columns correspond to the experimental con-ditions requiring 1, 2, or 3 lever presses.

making and learning in the experiment by [19]. In thisexperiment, yoked rats did learn to escape response whenone or two, but not when three lever presses were re-quired. In Figure 7, we show that the model can quanti-tatively capture the increasing penetrance of inescapableshock exposure with increasing escape response require-ments.

Discussion

Our results indicate that a normative account of gener-alization of action-outcome contingencies is sufficient toproduce a wide range of phenomena observed in learnedhelplessness experiments. The account captures (i) howhelplessness is induced by uncontrollable stressors andwhy it transfers to novel situations, (ii) why control-lable stress fails to induce helplessness, (iii) that helpless-ness results from impaired learning that different actionshave different effects, (iv) mastery effects, (v) impairedreward seeking, and (vi) the interaction between help-lessness and task requirements. This suggests that thegeneralization of experienced control may be sufficientto account for many learned helplessness effects.

Note that our model explains helplessness as the con-sequence of rational learning and generalization (cf.[11, 14]) from uncontrollable stress. Mirroring the factthat learned helplessness induces depression-like statesin healthy animals and affects healthy humans, this sug-

gests one pathway by which learned helplessness mayarise as a rational adaptation to an uncontrollable en-vironment rather than from negative biases or dysfunc-tional information processing. In the present account,the generalization that leads to helplessness is that dif-ferent actions tend to have the same effect. This gen-eralization can occur at two levels of abstraction: (i)from the outcomes of some actions in a given situationto the outcomes of all actions available in that situation,and (ii) from the controllability of one situation to thecontrollability of situations in general. Our model pre-dicts that after uncontrollable stress generalizations ofthe first type will be unusually strong. This may cap-ture overgeneralization – a frequent feature of depressivethought in humans [20] – and the model explains howthis learning style increases the risk for helplessness byfostering the belief that all actions are equal.

According to the classical notion [1], control requiresthat taking an action or not alters the probability of out-comes. Our model generalizes this notion by allowing formultiple actions, multiple outcomes, and two additionallevels of abstraction, and it expands it from a binarydistinction to a graded, quantitative measure of control-lability. As a result, our model instantiates [13]’s secondtype of controllability which captured on the achievabil-ity of different outcomes. While the notion of controlpresented here does not take into account the outcomes’desirability (type 3 in [13]), it refines [13]’s proposalof how helplessness might be learned by generalizationin two regards. First, [13] juxtaposed two extremes ofgeneralization: the controllabilities of different environ-ments are (i) independent (no generalization) or (ii) iden-tical (full generalization). Arguably, both extremes cor-respond to pathologically inaccurate models of the world.By contrast, the hierarchical generative model proposedhere formalizes the more realistic intermediate assump-tion that although some situations are more controllablethan others, knowing about the controllability of one sit-uation is informative about the controllability of othersituations. Second, inference in this model captures animportant aspect of attribution: Was the outcome dueto the action I took or due to the situation I was in?In the model, a perceived lack of control induces mis-attributions that impair learning: the over-hypothesisthat the world is uncontrollable renders implausible anyinterpretation according to which different actions havedifferent effects. Therefore the perceived lack of controlbiases people to attribute the outcome of taking actiona in state s to the state s rather than to the actionthat they have taken. For extreme helplessness the situ-ation’s action-independent outcome tendency βs will beupdated just as much as the outcome probabilities θs,a oftaking action a in this situation. Conversely, the action-specific outcome probabilities θs,a will be updated nomore than βs, and the outcomes of actions b, c, d, · · ·will influence the belief about θs,a almost as much as theoutcomes of action a itself. This increases the amount ofevidence required to discover that there is an action thatachieves the goal while most other actions do not, andthis is how the perceived lack of control impairs learn-

ing. Thus, according to our model, helpless behaviourin simple tasks results from slowed learning of transitionprobabilities. This complements a recent model of howthe perceived lack of control impairs planning in com-plex, sequential decision problems [21].

Despite the encouraging results reported here, moreresearch is needed to establish the validity of this mod-elling approach. Our simulation of the experiment re-ported in [19] did not fully capture the rats’ learningdynamics and overestimated their performance in thesimplest condition. These discrepancies could be dueto the simplistic assumption that rats do not associateshock termination with the number of lever presses butonly with their most recent action. As a result, the fitachieved by the reported simulation is not substantiallybetter than the fit obtained by reducing the subjectiveintensity of the shock (data not shown). Therefore theresults of our simulations (Figure 7) could be mimickedby analgesia [19, 22]. Furthermore, our model failed tocapture the precise pattern of the data in [6]. However,the two hypotheses might be discernible using data thatreveal the dynamics of learning, or by directly probingsubjects’ beliefs about outcome probabilities in the ab-sence of rewards.

There are important aspects of helplessness that ourmodel does not capture yet. It needs to assume sur-prisingly weak priors to explain why helplessness can belearned so rapidly—an assumption that is difficult to ex-tend to mature animals with a lifetime of experience.Conversely, it would be challenging to explain with ourmodel why even severe shock-induced helplessness tendsto fade within 48 hours. To capture these two sugges-tions of temporal (in)stability of helplessness, our modelcould be extended by replacing the constant c by a Gaus-sian random walk C(t). This would take into accountthat how much control a person has, changes through-out his or her life. It could be used to explain why avery brief period of uncontrollable stress can induce help-lessness despite a lifetime of experience in a controllableenvironment. This extension may also explain why thespeed of learning about controllability increases with theenvironment’s perceived volatility (cf. [23, 24]). Beyondthese cognitive signatures of helplessness, our model hasyet to engage with the affective and subjective aspectsof helplessness.

Our model suggests a number of avenues for futurework. As normative inference, exposure to sufficient con-trollability will lead to the correct inference, suggestingthat experience of control should heal helplessness, andthus that the escape deficit should be temporary. Whyand how the experience of shocks that an animal did notattempt to escape may nurture helpless beliefs may ne-cessitate adaptations in the inference. Nevertheless, themodel does capture that controllable stress can immu-nize subjects against the effects of uncontrollable stress,and suggests two computational mechanisms: first ahigher expected controllability, and second a higher esti-mate of the variability of control across situations. If theinitial expected controllability is higher, then a given re-duction in perceived control may no longer be sufficient

to induce learned helplessness. Alternatively, a higherestimate of the variability of control across situationswould reduce the strength of the generalization from un-controllable stress in one situation to the controllabil-ity of the world in general. This theoretical work does,however, also echo the likely limited benefits of exposurebecause a subjects’ helpless choices in some situations,paired with strong tendencies to generalise, can producesufficient evidence of lack of control to drown any islandsof controllability.

Our model may be able to illuminate why and how theeffects of chronic-mild stress [12] differ from the effectsof severe-acute stress in their scope, severity, and du-ration. The results suggests that the underlying mecha-nisms can be understood in terms of well-studied generalgeneralization phenomena [11]. For instance, since thestrength of a generalization increases with the variabil-ity of the examples, the experience of multiple stressorsshould render the effects of chronic mild stress more gen-eral than the equivalent amount of stress experienced ina single situation.

Learned helplessness is a behavioural paradigm withparallels in humans and animal models, and with es-tablished validity in research on depression [25]. Usingmethods from reinforcement and machine learning, thiswork has shown that abstract learning and generalizationabout controllability explain many of the key features oflearned helplessness in animals.

References[1] S. F. Maier and M. E. Seligman, “Learned helpless-

ness: Theory and evidence.,” J EXP PSYCHOL GEN,vol. 105, pp. 3–46, 1976.

[2] B. Overmier and M. Seligman, “Effects of inescapableshock upon subsequent escape and avoidance respond-ing.,” J COMP PHYSIOL PSYCH, vol. 63, pp. 28–33,1967.

[3] S. F. Maier and L. R. Watkins, “Stressor controllabilityand learned helplessness: the roles of the dorsal raphenucleus, serotonin, and corticotropin-releasing factor.,”NEUROSCI BIOBEHAV R, vol. 29, pp. 829–841, 2005.

[4] C. Peterson, S. F. Maier, and M. E. P. Seligman, LearnedHelplessness: A theory for the age of personal control.OUP, 1993.

[5] L. Y. Abramson, G. I. Metalsky, and L. B. Alloy, “Hope-lessness depression: A theory-based subtype of depres-sion,” Psychol. Rev., vol. 96, pp. 358–372, 1989.

[6] R. A. Rosellini, J. P. DeCola, and N. R. Shapiro, “Cross-motivational effects of inescapable shock are associativein nature.,” J EXP PSYCHOL ANIM B, vol. 8, pp. 376–388, 1982.

[7] F. Goodkin, “Rats learn the relationship between re-sponding and environmental events: An expansion ofthe learned helplessness hypothesis,” LEARN MOTIV,vol. 7, 1976.

[8] R. L. Welker, “Acquisition of a free-operant-appetitiveresponse in pigeons as a function of prior experience withresponse-independent food,” LEARN MOTIV, vol. 7,pp. 394–405, 1976.

[9] Q. J. M. Huys, J. Vogelstein, and P. Dayan, “Psychiatry:Insights into depression through normative decision-making models,” in Adv. Neural Inf. Process. Syst. 21(D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou,eds.), pp. 729–736, MIT Press, 2009.

[10] R. L. Jackson, S. F. Maier, and P. M. Rapaport, “Ex-posure to inescapable shock produces both activity andassociative deficits in the rat,” Learn. Motiv., vol. 9,pp. 69–98, 1978.

[11] J. B. Tenenbaum and T. L. Griffiths, “Generalization,similarity, and bayesian inference.,” Behav Brain Sci,vol. 24, 2001.

[12] P. Willner, “Validity, reliability and utility of the chronicmild stress model of depression: a 10-year review andevaluation,” Psychopharm, vol. 134, pp. 319–29, 1997.

[13] Q. J. M. Huys and P. Dayan, “A bayesian formulationof behavioral control.,” Cognition, vol. 113, pp. 314–328,2009.

[14] C. Kemp, A. Perfors, and J. B. Tenenbaum, “Learningoverhypotheses with hierarchical bayesian models,” Dev.Sci., vol. 10, pp. 307–321, 2007.

[15] M. Seligman and S. Maier, “Failure to escape traumaticshock.,” J EXP PSYCHOL, vol. 74, pp. 1–9, 1967.

[16] J. R. Volpicelli, R. R. Ulm, A. Altenor, and M. E. P.Seligman, “Learned mastery in the rat,” LEARN MO-TIV, vol. 14, pp. 204–222, 1983.

[17] R. D. Hannum, R. A. Rosellini, and M. E. Seligman,“Learned helplessness in the rat: Retention and immu-nization.,” DEV PSYCHOL, vol. 12, pp. 449–454, 1976.

[18] R. J. Herrnstein and D. H. Loveland, “Maximizing andmatching on concurrent ratio schedules.,” J Exp AnalBehav, vol. 24, pp. 107–116, 1975.

[19] M. Seligman and G. Beagley, “Learned helplessness inthe rat.,” J COMP PHYSIOL PSYCH, vol. 88, pp. 534–541, 1975.

[20] A. T. Beck, A. J. Rush, B. F. Shaw, and G. Emery,Cognitive therapy of depression. Guilford Press, 1979.

[21] F. Lieder, N. D. Goodman, and Q. J. M. Huys, “Con-trollability and resource-rational planning.,” Cosyne Ab-stracts 2013, 2013.

[22] R. L. Jackson, S. F. Maier, and D. J. Coon, “Long-termanalgesic effects of inescapable shock and learned help-lessness,” Science, vol. 206, no. 4414, pp. 91–93, 1979.

[23] T. E. J. Behrens, M. W. Woolrich, M. E. Walton, andM. F. S. Rushworth, “Learning the value of informationin an uncertain world.,” Nat Neurosci, vol. 10, pp. 1214–1221, 2007.

[24] C. Mathys, J. Daunizeau, K. J. Friston, and K. E.Stephan, “A bayesian foundation for individual learningunder uncertainty.,” Front Hum Neurosci, vol. 5, 2011.

[25] L. Y. Abramson, L. B. Alloy, M. E. Hogan, W. G.Whitehouse, M. Cornette, S. Akhavan, and A. Chiara,“Suicidality and cognitive vulnerability to depressionamong college students: a prospective study.,” J Ado-lesc, vol. 21, pp. 473–487, 1998.