Heather M. Wied (Previous Name: Heather M. Lueck)

E-mail: wied.heather@gmail.com

Education

M.D./Ph.D. Program, August 2008 – present

Ph.D., August 2014

M.D., expected May 2016

University of Maryland School of Medicine Medical Scientist Training Program

Post-Baccalaureate Premedical Program, July 2005 – June 2006

The Johns Hopkins University

B.A., June 2003

University of California, Los Angeles

Major in Psychology with a Minor in Neuroscience (summa cum laude)

Honors & Awards

Distinctions

F30 NIH NRSA Ruth L. Kirschstein Predoctoral MD/PhD Fellowship, March 2012 -

present

T32 NIH Predoctoral Training Grant, Program in Neuroscience, September 2010 -

August 2011

The Candidates’ Speaker (student chosen to address graduates at commencement),

UCLA College of Letters and Science Commencement, June 2003

Charles E. and Sue K. Young Undergraduate Student Award for 2002-2003, UCLA

Distinguished Bruin Award, UCLA Alumni Association, 2003

Departmental Honors, UCLA Psychology Department, 2002

Associations

American Association for the Advancement of Science (AAAS), April 2013 - present

Society for Neuroscience, October 2008 – present

Combined Accelerated Program in Psychiatry, University of Maryland School of

Medicine, October 2008 – June 2010

Alumni Scholar, UCLA, November 2002 – present

Phi Beta Kappa, June 2003 – present

Neuroscience Undergraduate Society, UCLA, September 2002 – June 2003

Golden Key International Honours Society, May 2002 – present

Psi Chi, National Honors Society in Psychology, September 2001 – present

Research, Employment and Teaching Experience

Predoctoral Research, University of Maryland School of Medicine, Program in

Neuroscience, 2010-2014. Supervised by Geoffrey Schoenbaum, M.D., Ph.D.

Faculty/Student Research, University of Maryland School of Medicine, Program in

Neuroscience, Summer 2009. Supervised by Elizabeth M. Powell, Ph.D.

Clinical and Computational Auditory Neuroscience Internship, Johns Hopkins School of

Medicine, Department of Neurology, Summer 2009. Supervised by Dana Boatman-

Reich, Ph.D., CCC-A. Funded by NIDCD.

Senior Research Assistant, Johns Hopkins School of Medicine, Department of

Neurology, 2006-2008. Supervised by Dana Boatman-Reich, Ph.D., CCC-A

Clinical Research Tutorial, Johns Hopkins School of Medicine, Department of

Neurology, 2006. Supervised by Dana Boatman-Reich, Ph.D., CCC-A

Teaching Assistant, Fieldwork in Behavioral Modification, UCLA Department of

Psychology, Fall 2004. Supervised by O. Ivar Lovaas, Ph.D.

Faculty/Student Research, UCLA, Department of Psychology, 2001-2004. Supervised by

O. Ivar Lovaas, Ph.D.

Faculty/Student Research, UCLA, Department of Psychology, 2002-2003. Supervised by

Michael S. Fanselow, Ph.D.

Teaching Assistant, Behavioral Modification, UCLA Department of Psychology, Spring

2002. Supervised by O. Ivar Lovaas, Ph.D.

Senior Discrete Trial Instructor, Lovaas Institute for Early Intervention, 2004.

One-to-One Behavioral Instructor, Lovaas Institute for Early Intervention, 2002-2004.

Research Publications

Published Manuscripts

Wied HM*, Sadacca BF*, Johnson W, Saini S, Jones J, Berg BA, Schoenbaum G.

Orbitofrontal cortex signals inferred value. (in preparation). (* shared first

authorship)

Cooch NK, Stalnaker TA, Wied H, Chaudhary S, McDannald MA, Liu TL, Schoenbaum

G. Orbitofrontal lesions eliminate value-selective activity in cue-responsive ventral

striatal neurons. (Submitted).

McDannald MA, Wegener M, Wied HM, Liu TL, Stalnaker TA, Jones JL, Esber GR,

Trageser J, Schoenbaum G. Orbitofrontal neurons acquire responses to ‘valueless’

Pavlovian cues during unblocking. (Submitted).

Stalnaker TA, Cooch NK, McDannald MA, Liu TL, Wied H, Schoenbaum G. (2014)

Orbitofrontal neurons infer the value and identity of predicted outcomes. Nature

Communications, 5:3926 doi: 10.1038/ncomms4926.

Wied HM, Jones JL, Cooch NK, Berg BA, Schoenbaum G. (2013). Disruption of model-

based behavior and learning by cocaine self-administration in rats.

Psychopharmacology, 229 (3): 493-501.

Quinn JJ, Wied HM, Liu D, Fanselow MS. (2009). Post-training excitotoxic lesions of

the dorsal hippocampus attenuate generalization in auditory delay fear conditioning.

European Journal of Neuroscience, 29(8): 1692–1700.

Sinai A, Crone NE, Wied HM, Franaszczuk PJ, Miglioretti D, Boatman-Reich D. (2009).

Intracranial mapping of auditory perception: event-related recordings and

electrocortical stimulation. Clinical Neurophysiology, 120(1): 140-149.

Quinn JJ, Wied HM, Ma QD, Tinsley MR, Fanselow MS. (2008). Dorsal hippocampus

involvement in delay fear conditioning depends upon the strength of the tone-

footshock association. Hippocampus, 18(7): 640-654.

Boatman DF, Trescher WH, Smith C, Ewen J, Los J, Wied HM, Gordon B, Kossoff EH,

Gao Q, Vining EP. (2008). Cortical auditory dysfunction in benign rolandic epilepsy.

Epilepsia, 49(6): 1018-1026.

Wied HM, Morrison PF, Gordon B, Zimmerman AW, Vining EP, Boatman DF. (2007).

Cortical auditory dysfunction in childhood epilepsy: electrophysiologic evidence.

Current Pediatric Reviews, 3(4): 317-327.

Published Abstracts and Presentations

(Please note - previous name Heather M. Lueck)

Wied HM, Cooch N, Jones JL, Berg BA, Schoenbaum G. (2013). Cocaine use disrupts

behavior and learning that depends on inferred value. The Catecholamines Gordon

Research Conference. West Dover, VT.

Wied HM, Cooch N, Jones JL, Berg BA, Schoenbaum G. (2013). Cocaine use disrupts

behavior and learning that depends on inferred value. National MD/PhD Student

Conference. Keystone, CO.

Wied HM, Cooch N, Jones JL, Esber G, Lucantonio F, Johnson W, Schoenbaum G.

(2013). Assessing the effects of cocaine use on different forms of value based

behavior and learning. Winter Conference on Brain Research. Breckenridge, CO.

McDannald MA, Wegener M, Wied HM, Liu TL, Stalnaker TA, Jones JL, Esber GR,

Trageser J, Schoenbaum G. (2012). Signaling reward prediction for value and identity

in rodent orbitofrontal cortex during Pavlovian unblocking. 2012 Society for

Neuroscience Annual Meeting. New Orleans, LA.

Wied HM, Ribeiro A, Leach JB, Powell EM. (2009). Using biomaterials to direct neural

cell fate response to ligands. World Stem Cell Summit.

Wied HM, Hinchey T, Sinai A, Crone N, Los J, Franaszczuk P, Boatman-Reich D.

(2009). Intracranial mapping of human auditory association cortex using simple and

complex sounds. The Association for Research in Otolaryngology.

Wied HM, Sinai A, Crone N, Franaszczuk P, Miglioretti D, Boatman-Reich D. (2008).

Intracranial mapping of auditory perception: event-related recordings and

electrocortical stimulation. University of Maryland School of Medicine Medical

Student Research Day.

Fanselow MS, Quinn JJ, Tinsley MR, Lueck HM, Liu D, Martikyan A. (2003). The

hippocampus, episodic retrieval and associative fear conditioning. Society for

Neuroscience.

Tinsley MR, Quinn JJ, Lueck H, Fanselow MS. (2002). Anisomycin attenuates

consolidation of predator fear. Society for Neuroscience.

Research Support

Ongoing Research Support

F30 DA033100, 03/08/2012 - present

The Role of Dopaminergic Error Signaling in Outcome-Specific Learning

Role: PI

Completed Research Support

T32 NS063391, 09/01/2010-08/31/2011

University of Maryland School of Medicine, Training Program in Neuroscience

PI: Margaret M. McCarthy, Ph.D.

Role: Trainee

Abstract

Title of Dissertation: The Orbitofrontal Cortex and Inferred Value: Neural Correlates and

the Effects of Cocaine

Heather M. Wied, Doctor of Philosophy, 2014

Dissertation Directed by:

Geoffrey Schoenbaum, M.D., Ph.D.

National Institute on Drug Abuse

Branch Chief and Senior Investigator, Tenured

Cellular Neurobiology Research Branch

Behavioral Neurophysiology Research Section

The orbitofrontal cortex (OFC) is important for guiding behavior and making

decisions based on predicted outcomes. Recently, the Schoenbaum lab, using a sensory

preconditioning task, demonstrated that the OFC is critical for the ability to make

decisions in novel situations based on predicted outcomes, and that it is not essential for

decisions based off of previous experiences. This study also demonstrated that the OFC

was important for learning when expectations about predicted outcomes were not met.

This study laid the foundation for this investigation which seeks to understand the

neurophysiological mechanism of how the OFC might signal this inferred value, and if

cocaine exposure disrupts OFC's ability to use inferred value to adequately guide

behavior and new learning.

To test the hypothesis that the OFC is signaling inferred value in the sensory

preconditioning task, electrodes were implanted into the OFC in rats. During the

preconditioning phase the mean firing rate of neurons to the stimulus-stimulus cues were

strongly correlated. This correlation was still significant at the time of testing, suggesting

the conditioned cues firing rate predicted the preconditioned cues firing rate.

Cocaine addiction is characterized by impaired decision-making. Evidence

suggests that cocaine-induced changes in the OFC may lead to difficulty with flexible

and adaptive model-based behavior. The second part of this study tests the hypothesis

that exposure to cocaine through self-administration would disrupt the ability of animals

to use inferred value to adequately guide behavior and new learning. Rats self-

administered cocaine for 14 days, and after a four-week withdrawal period, were tested

using sensory preconditioning and blocking. The previous exposure to cocaine disrupted

both the expression of behavior and learning that is contingent upon inferred values, but

it did not affect behavior or learning when normal behavior could be supported by cached

values. These results are similar to the effects observed when the OFC is inactivated

during the critical sensory preconditioning probe test and during blocking. Moreover,

they are likewise consistent with the idea that dysfunction in decision making that arises

in drug addiction is potentially mediated through neural deficits within the OFC.

The Orbitofrontal Cortex and Inferred Value:

Neural Correlates and the Effects of Cocaine

by

Heather M. Wied

Dissertation submitted to the Faculty of the Graduate School of the

University of Maryland, Baltimore in partial fulfillment

of the requirements for the degree of

Doctor of Philosophy

2014

©Copyright 2014 by Heather M. Wied

All rights reserved

iii

Acknowledgements

I offer my sincerest gratitude to Dr. Geoffrey Schoenbaum for being my advisor

and offering his support throughout my graduate school training. He has been an

outstanding mentor, teacher and advocate. He routinely offered insights for professional

growth and achievement, including constructive feedback on the presentation and

communication of scientific concepts. Dr. Schoenbaum has been incredibly supportive of

my goals in becoming a clinician-scientist, understanding the difficult timelines of the

MD/PhD program and helping to ensure I return to clinic rotations on time. I am also

grateful for the members of my thesis committee: Drs. Matthew Roesch, Gregory Elmer,

Asaf Keller and Patricio O’Donnell.

I want to thank the entire Schoenbaum lab for the many discussions about science

and the orbitofrontal cortex. Their dedication was inspiring and made it a joy to be in the

lab. In particular, I cannot thank Dr. Brian Sadacca enough for his patience and guidance

as I learned about electrophysiology and how to analyze the data.

I also want to acknowledge the University of Maryland School of Medicine and

the MD/PhD Program and Program in Neuroscience for providing me with this

outstanding opportunity. Additionally, I want to thank my fellow MD/PhD students for

their support through these years.

Lastly, I simply could not have persisted without the constant love and support of

my family and friends, especially my husband, Aaron, my parents John and Pamela

Lueck, and Douglas and Christine Wied. I am most indebted to Aaron and his unfailing

love and support of my dreams of becoming a clinician-scientist. And, of course, to

iv

Christopher: Thank you for understanding no matter what time of day or night, that

mommy was going to the “lab” and that I always missed you and your daddy!

v

Table of Contents

I. Introduction ................................................................................................................. 1

1. Orbitofrontal Cortex: Anatomy................................................................................ 2

2. Orbitofrontal Cortex: Behavior ................................................................................ 7

A. Reversal Learning ................................................................................................ 8

B. Outcome Devaluation ......................................................................................... 11

C. Unblocking ......................................................................................................... 14

D. Pavlovian-to-Instrumental Transfer ................................................................... 16

E. Pavlovian Overexpectation ................................................................................ 17

F. Sensory Preconditioning .................................................................................... 20

3. Orbitofrontal Cortex: Neurophysiology ................................................................. 26

4. Cocaine: Effects on the Orbitofrontal Cortex ........................................................ 29

A. Cocaine: Effects on Orbitofrontal Cortex Dependent Behaviors ....................... 32

B. Cocaine: Effects on Orbitofrontal Cortex Neurophysiology.............................. 37

II. Investigating the role of the orbitofrontal cortex in inferred value ........................ 42

1. Introduction ............................................................................................................ 42

2. Materials and Methods ........................................................................................... 43

A. Subjects .............................................................................................................. 43

B. Apparatus ........................................................................................................... 44

C. Surgical Procedures ............................................................................................ 45

D. Histology ............................................................................................................ 45

E. Behavioral Task.................................................................................................. 45

F. Single-Unit Recording........................................................................................ 48

3. Results .................................................................................................................... 49

A. Sensory Preconditioning Behavioral Results: .................................................... 49

B. Sensory Preconditioning in Probe Tests that demonstrated A>C Behavior....... 51

C. Neural Probe Test Results: ................................................................................. 55

4. Discussion .............................................................................................................. 60

III. Disruption of model-based behavior and learning by cocaine self-administration in

rats… ............................................................................................................................... 63

1. Abstract .................................................................................................................. 63

2. Introduction ............................................................................................................ 64

3. Materials and Methods ........................................................................................... 67

A. Subjects: ............................................................................................................. 67

vi

B. Apparatus: .......................................................................................................... 67

C. Surgical procedures: ........................................................................................... 68

D. Self-Administration: ........................................................................................... 69

E. Sensory Preconditioning: ................................................................................... 69

F. Inferred Value Blocking: .................................................................................... 71

G. Response measures: ........................................................................................... 72

H. Statistical Analyses: ........................................................................................... 73

4. Results .................................................................................................................... 73

A. Self-Administration: ........................................................................................... 73

B. Sensory Preconditioning: ................................................................................... 74

C. Inferred Value Blocking: .................................................................................... 78

5. Discussion .............................................................................................................. 80

IV. Conclusions ............................................................................................................ 85

References ........................................................................................................................ ix

vii

List of Figures

Figure I.1 Comparative anatomy of the human, monkey and rat frontal cortex. ............... 4

Figure I.2 Anatomical relationships of the OFC in rat and monkey. .................................. 6

Figure I.3 Effect of OFC lesions on reversal learning. ..................................................... 10

Figure I.4 Effect of OFC lesions on outcome devaluation. ............................................. 13

Figure I.5 Effect of OFC lesions on blocking. ................................................................. 15

Figure I.6 Effect of OFC lesions on Pavlovian-to-instrumental transfer. ........................ 17

Figure I.7 Effects of OFC inactivation on changes in behavior after overexpectation.... 19

Figure I.8 Effect of OFC inactivation on sensory preconditioning. ................................ 24

Figure I.9 Effect of OFC inactivation on learning based on inferred value. .................... 25

Figure I.10 Conditioned responding and cue-evoked activity summates at the start of

compound training. ............................................................................................... 28

Figure I.11 Decreased gray matter concentration in the OFC. ......................................... 30

Figure I.12 Decreased glucose metabolism in the OFC. .................................................. 31

Figure I.13 Changes in spine density. ............................................................................... 32

Figure I.14 Effect of cocaine on reversal learning. ........................................................... 34

Figure I.15 Effect of cocaine on outcome devaluation. .................................................... 35

Figure I.16 Effect of cocaine on pavlovian overexpectation. .......................................... 37

Figure I.17 The effect of cocaine on OFC cue-selectivity neurons in reversal learning. . 39

Figure I.18 The effect of cocaine on OFC cue-selectivity neurons in Pavlovian

overexpectation. .................................................................................................... 41

Figure II.1 Sensory preconditioning. ................................................................................ 46

viii

Figure II.2 Sensory preconditioning: Number of nosepokes during preconditioning and

probe test. .............................................................................................................. 51

Figure II.3 Distribution of sessions that demonstrated greater responding to cue A than to

cue C. .................................................................................................................... 52

Figure II.4 Sensory preconditioning behavior results from the sessions that demonstrated

A greater than C responding. ................................................................................ 53

Figure II.5 Sensory preconditioning: Number of nosepokes during preconditioning and

probe test from sessions that demonstrated A greater than C responding. ........... 55

Figure II.6 Proportion of neurons with significant cue-selective activity during the

preconditioning. .................................................................................................... 56

Figure II.7 Proportion of neurons with significant cue-selective activity during the 10

second auditory cues during probe test. ................................................................ 58

Figure II.8 Cue-evoked activity during probe test. ........................................................... 59

Figure III.1 Self-administration. ....................................................................................... 74

Figure III.2 Cocaine self-administration disrupts the expression of behavior that depends

upon inferred values. ............................................................................................. 77

Figure III.3 Cocaine self-administration disrupts learning that depends upon inferred

values. ................................................................................................................... 79

1

I. Introduction

The interest and modern understanding of the frontal lobe was sparked by a young

railroad worker in Cavendish, Vermont named Phineas Gage. He survived a remarkable

explosion on September 13, 1848 that propelled an iron tamping bar into his head and

through his frontal lobes, including the orbitofrontal cortex. After his unlikely recovery

from the physical wounds, the attending physician, Harlow (1868), described a change in

Gage's behavior as follows:

"He is fitful, irreverent, indulging at times in the grossest

profanity (which was not previously his custom),

manifesting but little deference for his fellows, impatient of

restraint or advice when it conflicts with his desires, at times

pertinaciously obstinate, yet capricious and vacillating,

devising many plans of future operations, which are no

sooner arranged than they are abandoned in turn for others

appearing more feasible. A child in his intellectual capacity

and manifestations, he has the animal passions of a strong

man. Previous to his injury, although untrained in the

schools, he possessed a well-balanced mind, and was looked

upon by those who knew him as a shrewd, smart

businessman, very energetic and persistent in executing all

his plans of operation. In this regard his mind was radically

changed, so decidedly that his friends and acquaintances

said he was 'no longer Gage.'"

Similar to the reported changes in Gage's behavior, Damasio (1994) described patients

with orbitofrontal damage as being impulsive, disinhibited and perseverative. Oftentimes,

patients with prefrontal lesions have difficulty with insight, inference and decision-

making. Orbitofrontal cortex damage appears to leave patients inflexible and their

2

behavior is more stimulus-driven. Many times patients are observed as acting for instant

gratification instead of thinking about long term consequences. These behavior changes

are often very evident to family and friends; but too often patients with frontal lobe

lesions are not recognized as readily in the medical field, as they pass most neurological

exams.

The orbitofrontal cortex continues to be of great interest and many recent

behavioral and neurophysiological studies have begun to better elucidate its role. The

anatomy, behavior and neurophysiology of such experiments will be reviewed. In

addition, recent evidence suggests that the psychostimulant cocaine affects the

orbitofrontal cortex as well, leading to behaviors oftentimes seen in patients with

orbitofrontal cortex damage. This appears to include impulsive decisions and stimulus-

based responding. The evidence of how cocaine may affect the orbitofrontal cortex is also

reviewed.

1. Orbitofrontal Cortex: Anatomy

The orbitofrontal cortex (OFC) is located in the ventral part of the frontal lobe.

The OFC has remarkably similar connections across species, offering important clues as

to the critical function of this brain region, despite anatomical differences between

humans, primates and rodents. Although the OFC is loosely defined in terms of borders

throughout the different species, and the cytoarchitecture is different, especially in

comparison of humans and primates to rodents (Figure I.1), the connections are very

similar. Rose and Woolsey (1948) defined the prefrontal cortex in terms of the

projections of the mediodorsal nucleus of the thalamus (MD). Though the regions in

3

rodents lack the dense granular layer characteristic of much of the primate prefrontal

cortex, defining it according to the MD projections allows homologies to be drawn

between prefrontal regions in non-primate and primate species. This includes the rodent

orbitofrontal cortex. Specifically, the projections of the medial and central mediodorsal

thalamus to orbital and agranular ventral and dorsal insular areas of the rat prefrontal

cortex is homologous to the primate orbitofrontal region (Leonard, 1969, Krettek and

Price, 1977, Groenewegen, 1988, Ray and Price, 1992, Schoenbaum and Setlow, 2001)

as seen in Figure I.2.

The non-primate mediodorsal nucleus of the thalamus receives afferents that are

similar to the primate MD afferents including from the amygdala, medial temporal lobe,

and ventral pallidum/ventral tegmental area, and olfactory input from the piriform cortex

(Krettek and Price, 1977, Groenewegen, 1988, Ray and Price, 1992). In rodents, the OFC

that is similar to the primate orbitofrontal region includes the dorsal and ventral agranular

cortex as well as lateral and ventrolateral orbital regions; however, the defined area of the

orbitofrontal cortex that is similar across species does not include the structures that

surround the medial wall of the hemispheres, including the medial or ventromedial wall

in non-primates (Roesch and Schoenbaum, 2006).

4

Figure I.1 Comparative anatomy of the human, monkey and rat frontal cortex.

(a,b) Architectonic maps of the medial (top) and orbital (bottom) surfaces of the

frontal lobe in humans (Ongur et al., 2003) (a) and monkeys (Carmichael and Price,

1994) (b). (c) Medial (top) and lateral (bottom) frontal cortex in rats (Palamero-

Gallagher and Zilles, 2004). Agranular cortex lacks layer IV. Dysgranular cortex

contains a rudimentary layer IV. Granular cortex has a well-developed layer IV. Layer

IV neurons are described as granular because their cell bodies are small and round,

and changes in this layer are clearly visible as one transitions from agranular to

granular cortex.

Abbreviations: AC, anterior cingulate area; AON, anterior olfactory nucleus; c,

caudal; cc, corpus callosum; Fr2, second frontal area; I, insula; i, inferior; Ia, agranular

infralimbic cortex; IL, infralimbic cortex; l, lateral; LO, lateral orbital area; m, medial;

M1, primary motor area; MO, medial orbital area; o, orbital; p, posterior; Par, parietal

cortex; Pir, Piriform cortex; PL, prelimbic cortex; r, rostral; s, sulcal; v, ventral; VO,

ventral orbital area. Numbers indicate cortical fields, except that after certain areas,

such as Fr2 and AC1, they indicate subdivisions of cortical fields. *Source of image (Wallis, 2012)

5

Afferent input to OFC is also similar across species. In both primates and rodents,

the OFC receives information from all sensory modalities (Carmichael and Price, 1995).

Additionally, the connections between the OFC and limbic system, including the

amygdala, are extensive (Carmichael and Price, 1995). These reciprocal connections of

the basolateral amygdala to the agranular insular cortex are thought to be involved in

affective and motivational aspects of learning (Brown and Schafer, 1888, Kluver and

Bucy, 1939, Weiskrantz, 1956, Everitt and Robbins, 1992, LeDoux, 1996, Holland and

Gallagher, 1999, Davis, 2000, Gallagher, 2000, Baxter and Murray, 2002, Roesch and

Schoenbaum, 2006). The OFC also has strong efferent projections to the ventral striatum

overlapping with innervations from limbic structures such as the amygdala and

hippocampus (Groenewegen et al., 1987, McDonald, 1991, Haber et al., 1995, Haber and

Knutson, 2010). As seen in Figure I.2, the similarities across species in the connections of

the OFC with limbic structures and the ventral striatum are compelling. The similarities

in these connections suggest similarities in functional interactions with the major

components of the forebrain across humans, primates and non-primates species including

rodents. Anatomically, the OFC is an ideal candidate for the integration of sensory and

associative information. This region is ideally suited to evaluating outcomes, as it

receives input from all sensory modalities.

6

Figure I.2 Anatomical relationships of the OFC in rat and monkey.

Based on the connections with mediodorsal thalamus, amygdala and striatum, the orbital and

agranular insular areas in rat prefrontal cortex are homologous to primate orbitofrontal cortex.

In both species, orbitofrontal cortex receives conspicuous input from sensory cortices,

associative information from amygdala and outputs to the motor system through striatum.

Abbreviations: AId, dorsal agranular insula; AIv, ventral agranular insula; c, central; m,

medial; ABL, basolateral amygdala; rABL, rostral basolateral amygdala; CD, caudate; NAc,

nucleus accumbens core; VP, ventral pallidum; LO, lateral orbital; VO, ventral orbital,

including ventrolateral and ventromedial orbital regions.

* Source of image (Roesch and Schoenbaum, 2006)

OFC

OFC

Rat Monkey

MDMD

Amygdala Striatum Amygdala

AId 1211/13

14

CD

NAc

VP

ABL

mcmcAIv LO VO

rABL

NAc

VP

Striatum

Sensory Information

Motor Output

7

2. Orbitofrontal Cortex: Behavior

The OFC has been thought to be important for guiding behavior and making

decisions based on predicted outcomes. Corticolimbic circuits in the brain have been

implicated in signaling information about expected outcomes and their general value.

Encoding the general or cached value of items facilitates rapid and transitive decision

making — comparing, say, apples and oranges — and discrepancies in this predicted

value also drive learning. Recent work suggests that brain circuits also maintain

information specific to particular outcomes and that decisions - and learning - can be

driven by changes in outcome information even when value remains unchanged.

When our actions are made simply on the basis of previously experienced

rewards, we use these cached or so called “model-free” value representations. In these

situations, the value of an action or predictive cue is represented in a common currency

without any knowledge about the specific form and features of the reward it predicts. In

fact, the sequential structure of the cue-reward association is made without thinking about

the specifics of the reward. Although model-free representations allow for fast actions

without thinking, there is often much more involved in some decisions that we make. For

these more intricate and computationally demanding decisions that require flexibility,

model-based representations are essential; decisions and actions are made based on

expected future outcomes according to a more complex model of the environment. This

allows decisions to be made even if a person has never experienced the direct reward

from taking this action since it can be predicted in the moment from the previously

acquired model of the task or environment.

8

The OFC is thought to be important for associative learning. Over the years, it has

become clear that it has a specific and critical role in the ability to integrate and signal

information about specific outcomes (Burke et al., 2008, McDannald et al., 2011,

McDannald et al., 2005, Gallagher et al., 1999, Izquierdo et al., 2004, Ostlund and

Balleine, 2007). In this role, it appears to be most important in novel, unfamiliar or

unprecedented situations, integrating associative information from other areas about past

experiences and providing an in-the-moment prediction about what may happen based off

of the history of previous experiences. Although initially the OFC was shown to be

critical for reversal learning and thus important for flexible and outcome-guided

behavior, its role as a point of integration in this network to in-the-moment guided

behaviors has become increasingly more clear from a series of elegant behavioral

experiments. This includes outcome devaluation, blocking, Pavlovian-to-instrumental

transfer, Pavlovian overexpectation and, most recently, sensory preconditioning. These

experiments help to more clearly define the role of the orbitofrontal cortex in model-

based behaviors.

A. Reversal Learning

In reversal learning, animals are taught that one stimulus leads to a reward, while

another stimulus leads to no reward or to a punishment. Once these cue-outcome

associations are well established, they are then reversed. The ability to understand that

the cue-outcome associations have switched requires the animal to be flexible and to

adapt to its environment. An animal must be able to change its behavior in the event of an

9

unexpected outcome. It has been found that the OFC is required for this flexibility, as

animals with OFC lesions across species are unable to adapt easily to these changes in the

cue-outcome associations. Importantly, animals are able to learn the initial discrimination

as fast as control animals (Schoenbaum et al., 2003). While the animals with OFC lesions

can eventually acquire the new cue-outcome associations after a much longer period of

time and many more trials, they have a markedly difficult time reversing their behaviors

when these cue-outcome associations are then reversed back to the original associations.

This difficulty in reversal learning tasks has been repeatedly shown throughout many

species and experiments, as seen in Figure I.3 (Jones and Mishkin, 1972, Rolls et al.,

1994, Bechara et al., 1997, Meunier et al., 1997, Chudasama and Robbins, 2003, Fellows

and Farah, 2003, McAlonan and Brown, 2003, Hornak et al., 2004, Izquierdo et al., 2004,

Pais-Vieira et al., 2007, Bissonette et al., 2008, Reekie et al., 2008, Teitelbaum, 1964,

Butter, 1969, Schoenbaum et al., 2003). However, reversal learning is not specific, and it

has also been suggested that the OFC is important for response inhibition to a previously

rewarded cue (Jones and Mishkin, 1972, Eagle et al., 2008, Ferrier, 1876, Izquierdo and

Jentsch, 2012, Jentsch and Taylor, 1999, Man et al., 2009), or that it is important for

updating specific outcome expectancies to guide decisions (Schoenbaum et al., 2009,

Holland and Gallagher, 2004, Wallis, 2007).

10

Figure I.3 Effect of OFC lesions on reversal learning.

(a) Rats were trained to sample odors at a central port and then respond at a nearby

fluid well. In each odor problem, one odor predicted sucrose and a second quinine.

Rats had to learn to respond for sucrose and to inhibit responding to avoid quinine,

and then the odor-outcome associations were reversed. Shown is the number of trials

required by controls and OFC-lesioned rats to meet a 90% performance criterion. OFC

lesions have intact retention of a previously learned discrimination, but impaired

reversal learning. Adapted from (Schoenbaum et al., 2003) (b) Group mean errors to

criterion for initial learning and nine serial reversals in object reversal learning in

monkeys with bilateral OFC lesions. OFC-lesioned monkeys are specifically impaired

in reversal learning. Adapted from (Izquierdo et al., 2004) (c) Initial stimulus-

reinforcer association learning and reversal learning performance in subjects with

damage to the ventromedial prefrontal region and in controls. Initial learning and

reversal performance is expressed as trials before the learning criterion is met.

Subjects with lesions make more than twice as many errors as healthy controls in the

reversal phase of the task. *P < 0.05; **P < 0.01. Adapted from (Fellows and Farah,

2003)

*Source of image (Lucantonio et al., 2012)

11

B. Outcome Devaluation

Outcome devaluation experiments in rodents and monkeys have led to a much

better understanding of the function of the OFC (Izquierdo and Murray, 2010, Holland

and Rescorla, 1975, Holland and Straub, 1979, Gallagher et al., 1999, Pickens et al.,

2005, Pickens et al., 2003, Izquierdo and Murray, 2004, Izquierdo et al., 2004, Machado

and Bachevalier, 2007, West et al., 2011, Murray et al., 2007). From these experiments, it

is apparent that the OFC is critical for integrating information about reward predictive

cues and the new value of those rewards. In outcome devaluation experiments, a neutral

cue is paired with the delivery of a reward such as an appetitive food outcome. Once this

association is well established, the outcome is devalued by pairing the reward with illness

or satiation, thereby decreasing the conditioned response to the reward-predictive cue.

This decrease in responding is examined by looking at the behavior when the conditioned

cue is presented by itself in extinction. This decrease in responding to the cue occurs

immediately, even though the cue was never explicitly paired with the devalued reward.

The decline in conditioned responding after outcome devaluation is not new learning

since the cue was never paired directly with the devalued reward. Instead, the animal

must presumably use the previously acquired association between the cue and the non-

devalued food, combined with an updated representation of the outcome and its new

value, to guide or modify responding. Notably, animals with OFC lesions do not show

this ability (Figure I.4b and c) (Izquierdo and Murray, 2010, Gallagher et al., 1999,

Pickens et al., 2005, Pickens et al., 2003, Izquierdo and Murray, 2004, Izquierdo et al.,

2004, Machado and Bachevalier, 2007, West et al., 2011, Murray et al., 2007). The

12

animals with OFC lesions do not show an impairment in their ability to learn the initial

cue-outcome association or in their ability to devalue the food, only in their ability to

apply this knowledge to guide their behavior in the final test. The OFC is thought to be

important for helping to imagine the outcome that is coming when the cue is presented,

such that, although the animals have never directly experienced the associated cue

leading to the devalued reward, they are able to imagine that the cue would lead to the

devalued reward decreasing their response to the cue. Accordingly, human fMRI studies

have also demonstrated that there are blood-oxygen-level dependent (BOLD) signal

changes in the OFC when the value of expected outcomes is changed, providing

additional support to the idea that the OFC signals changes in value of outcomes

(O'Doherty et al., 2000a, Gottfried et al., 2003) (Figure I.4a).

13

Figure I.4 Effect of OFC lesions on outcome devaluation.

(a) Changes in BOLD signal in human orbitofrontal cortex after reinforcer

devaluation. Subjects were scanned during presentation of odors of different foods.

Subsequently, one food was devalued by overfeeding and then subjects were

rescanned. Subjective appetitive ratings of the odor (top) and BOLD response to the

odor-predicting cue in orbitofrontal cortex (bottom) decline for satiated but not

nonsatiated foods. (b) Changes in Pavlovian conditioned responding in sham and

OFC-lesioned rats after reinforcer devaluation. Rats were trained to associate a light

cue with food. Subsequently, the food was devalued by pairing it with LiCl-induced

illness and response to the cue was assessed in a final probe session. Rats with OFC

lesions fail to show any effect of devaluation on conditioned responding (percentage

of time in food cup), despite normal conditioning and devaluation of the food reward.

(c) Changes in discriminative responding in sham and orbitofrontal-lesioned monkeys

after reinforcer devaluation. Monkeys were trained to associate different objects with

different food rewards. Subsequently, one food was devalued by overfeeding and then

discrimination performance was assessed in a probe test. The figure illustrates a

difference score comparing post- and pre-satiation bias; OFC-lesioned monkeys fail to

bias their choices away from objects associated with the satiated food. *P < 0.05; **P

< 0.01. Adapted from (Gottfried et al., 2003, Gallagher et al., 1999, Izquierdo et al.,

2004).

*Source of image (Lucantonio et al., 2012)

14

C. Unblocking

McDannald et al. (2011) provided additional evidence of the OFC’s role in

outcome-related learning. In order to determine the differences between outcome-specific

and value-based learning in the OFC, they used a Pavlovian unblocking procedure

(Kamin, 1969). In this procedure, a rat is trained that a cue is associated with a particular

food outcome. After this association is learned, a second cue is presented in compound

with the first cue, but leading to the same food outcome. In this situation, learning to the

second cue is blocked because the reward is completely predicted by the first cue as the

food outcome is exactly the same. In unblocking, the second cue is presented in

compound with the first cue, but is followed by a different food outcome, either in

quantity of food (value) (Holland, 1984) or in type of food (identity) (Rescorla, 1999).

This new outcome goes against the animals’ original expectation. Therefore, in

unblocking, learning occurs to the second cue. In McDannald et al.’s (2011) study, they

were able to determine the role of OFC and support the specific-outcome theory by

demonstrating that OFC lesions in rats made prior to training procedures leave

unblocking based on value fully intact but impair unblocking based on specific reward

features such as identity (Figure I.5). These rats had no problems in conditioning and

could fully differentiate between two different reward flavors. However, this study

specifically demonstrated that the OFC is critical for learning about changes in reward

identity.

15

Figure I.5 Effect of OFC lesions on blocking.

Contributions of the orbitofrontal cortex to blocking. The effects of pre-training OFC

lesions on unblocking driven by changes in reward flavor or increases in reward value

are summarized (McDannald et al., 2011). (A) Rats were trained to associate three

visual cues with different flavors and quantities of reward. In an unblocking phase

these visual cues were compounded with novel auditory and aspects of the reward

selectively changed. A probe test was administered in which the novel auditory cues

were presented in isolation and extinction. (B) Representative control (left) and

neurotoxic lesion (right) are shown. (C) Food cup responding to cues X and Y are

shown for control (left) and OFC-lesioned rats (right). Only control rats showed

evidence of learning to cue Y, the cue signaling a change in reward flavor. (D) Food

cup responding to cues X and Z are shown for control (left) and OFC-lesioned rats

(right). Both control and OFC-lesioned rats showed evidence of learning to cue Z, the

cue signaling an increase in reward number.

*Source of image (McDannald et al. 2014)

16

D. Pavlovian-to-Instrumental Transfer

Preceding McDannald et al.'s (2011) results, Balleine's lab had demonstrated

results that support the idea the OFC lesions do not impair instrumental learning or

Pavlovian conditioning, but impair specific transfer of information. In Pavlovian-to-

instrumental transfer, rats are taught that one response leads to one preferred outcome,

while another response leads to another equally preferred outcome. During this training

period, the animals are also taught that one context leads to the same outcome as the first

response, while a second context leads to the second equally preferred outcome. Once

these associations are well established, a transfer test is given in which the context is

presented in combination with the instrumental responding while no rewards are given.

When rats are presented with the context cues, the animal then performs the action that

leads to the same reward more than the action that leads to the other reward. This

phenomenon is called specific Pavlovian-to-instrumental transfer and depends upon the

cues and actions evoking a representation of the same specific rewards. When the OFC is

lesioned, rats fail to change their behaviors to match the context cues thus failing to

demonstrate this specific form of Pavlovian-to-instrumental transfer (Ostlund and

Balleine, 2007, Scarlet et al., 2012) (Figure I.6).

17

E. Pavlovian Overexpectation

As previously mentioned, the specific role of the OFC in reversal learning is hard

to identify: is it due to an inability to understand that changes have occurred to the

outcome, is it due to a deficit in response inhibition, or is it due to changes in some other

function such as impaired learning mechanisms? Support for the latter hypothesis comes

from work using Pavlovian overexpectation (Takahashi et al., 2009). In Pavlovian

overexpectation, animals are taught that one cue predicts a certain reward and that a

second cue also predicts the same reward. When these cues are then presented in

Figure I.6 Effect of OFC lesions on Pavlovian-to-instrumental transfer.

Effect of OFC lesions on Pavlovian-to-instrumental transfer. Mean lever presses per

minute (+ SEM) during the pre-CS period (Pre-CS), the CS that predicted the same

outcome as the response (Same), and the CS that predicted a different outcome (Diff).

*Source of image (Ostlund and Balleine, 2007)

18

compound together, there is an increase in responding in normal animals. This is

interpreted as the animals having integrated the predictions of the two cues, in effect

predicting or imagining that they will be getting double the reward. In fact, the animals

receive the same amount of reward during the compound cue as they were previously

given with the individual cues. In a probe test, in which the cues are presented alone

again without reward, the response to the cues is decreased. This decrease in responding

is thought to be due to the fact that the summed predictions for the cues were not met.

Thus, this discrepancy between the predicted and received outcomes drives inhibitory

learning which leads to decreased responding when the cues are presented alone again.

Takahashi et al. (2009) demonstrated that inactivation of the OFC with baclofen and

muscimol in rats on compound days eliminates this increase in responding to the

compound cues seen in controls (Figure I.7). This suggests that the inactivation of the

OFC did not allow the rats to summate their expectations to the compound of the cues;

rather, they were unable to integrate the two associations together and, thus, did not

expect to receive double the reward. Furthermore, when the OFC-inactivated animals

were then presented the cues alone, without inactivation, they did not decrease their

responding, thereby demonstrating they did not undergo the inhibitory learning that the

control animals showed. Takahashi et al. (2009) provides support to the idea that the OFC

may be important for inhibiting or reversing previously acquired responses due to its role

in signaling expected outcomes.

19

Figure I.7 Effects of OFC inactivation on changes in behavior after

overexpectation.

Top and bottom rows of plots indicate control and OFC-inactivated groups,

respectively. (A) Percentage of responding to food cup during cue presentation across

10 days of conditioning. (B) On left, percentage of responding to food cup during cue

presentation across 4 days of compound training. On right, red and blue bars indicate

average normalized percentage responding to A1/V1 and A2, respectively. (C)

Percentage of responding to food cup during cue presentation in the probe test. Line

graph shows responding across the eight trials and the bar graph shows average

responding in these eight trials. Red, blue, and white colors indicate responding to A1

or A1/V1, A2, and A3 cues, respectively (*p < 0.05; **p < 0.01). Error bars = SEM.

Adapted from (Takahashi et al., 2009). *Source of image (Takahashi et al., 2009)

20

F. Sensory Preconditioning

Sensory preconditioning offers the unique ability to directly assess if behavior is

driven by cached value or inferred value (Jones et al., 2012). As in model-free

reinforcement learning, cached value representations are values that are based off of prior

reinforcement history. Conversely, inferred value, specifically a prediction of the

outcome, must be calculated in-the-moment based upon the relationship between the cues

and the environment. This requires a knowledge of the causal structure of the

environment, or a model-based representation.

Most importantly, sensory preconditioning was able to help sort out the two

dominant theories of the function of the OFC: the neuroeconomic view and the

associative view. Neuroeconomics is the study of the neural control of value-based or

economic decision making. The associative view believes the OFC anticipates outcomes

and engages in model-based reinforcement learning in which there is a map of the

environment that helps to make predictions about outcomes and rewards. Therefore, the

associative view does not require that the OFC be used only in value-guided decisions,

but that a new value would be estimated in a new experience or situation based off prior

experiences.

While the economic value is typically measured through preferences, it appears to

include both cached and inferred value, with the cached value based upon previous

experiences. This is a simple model that is fast and efficient, as long as experiences never

change. But because not all situations are the same or similar, the fact that this cached

value is inflexible makes this model problematic in explaining complex decision making.

21

The cached value is not associated with the outcome itself. This makes it difficult to

update cached values, as they do not take into account the changes in the value of the

expected reward.

The model-based system is a flexible system and allows for new experiences to

have predicted outcomes. It also allows for learning to occur when these predictions do

not match the expected outcome. In this flexible system, there is an associative model of

the world that helps us make decisions with the most up-to-date information. This

associative view of the world helps us predict outcomes across many different scenarios

and different environments and also allows us to constantly learn as we update our

predictions through each and every experience, whether it agrees with what we predicted

or not. The associative view predicts the OFC is providing the flexibility to adapt with

each new experience.

The alternative hypothesis is that the OFC performs the same function in all

settings and that it specifically contributes to value-guided behavior and learning when

value must be inferred or derived from model-based representations. Under this

interpretation, this would hold true even for economic decision making. Similar ideas

were tested and supported in an experiment using sensory preconditioning (Jones et al.,

2012).

In a sensory preconditioning experiment run in the Schoenbaum lab (Jones et al.,

2012), rats were exposed to a pairing of two auditory cues (A→B and C→D) (Figure I.8).

After the rats were exposed to these pairings, one of these cues (B) was paired with a

reward, while the other cue (D) predicted no reward. Once B and D were successfully

conditioned, and thus predictive of reward (B) or no reward (D), the rat was presented

22

with the preconditioned cues (A and C) that were originally paired with the cue that

predicts the reward (B) or no reward (D). The rats demonstrated a strong response to the

preconditioned cue (A) that was paired with the cue that predicted reward (B). But they

did not respond to the cue (C) that had been paired with the cue that did not predict

reward (D). The role of the OFC became evident when the OFC was inactivated by

baclofen and muscimol on the critical probe test with the presentation of A and C alone.

Rats still responded to B but not A, C or D. Because rats still responded to B, it is clear

that the OFC is not critical for cached value. This is contrary to what would be predicted

if neuroeconomics and value-based theories of the OFC include cached value. Instead,

OFC inactivation affected the ability of the rats to infer value to A, which is the cue they

had never directly experienced before with reward. In control animals, the OFC was

available to help rats predict that if B predicts reward, then, as in a classic syllogism, A

may also predict a reward. But because the OFC was offline at the time of the probe test,

the rats did not have access to this cognitive map of the environment, or model-based

representation, that would have helped them predict that A leads to B that leads to

reward. This demonstrates that control animals had access to these predictions about

outcomes and inferred value to the preconditioned cue (A) due to the ability to access this

associative map of their environment. It appears then in sensory preconditioning

differential responding to A versus C (but not B and D) reflects the ability of the animals

to access a model-based representation. In this experiment, animals were able to integrate

the preconditioned stimulus-stimulus associations that did not have any value with the

subsequent associations between the stimulus and reward / non-reward.

23

Jones’ et al. (2012) experimental findings clearly support the outcome-specific

theories of the OFC, as OFC inactivation had no effect on B and D responding (the

cached value cues) but significantly impaired the differential responding to A and C (the

inferred value cues) (Figure I.8). Neuroeconomics and value theories would have

predicted that OFC inactivation would have led to impaired responding to both cached

value cues (B and D) and inferred value cues (A and C), but this was not the case.

Therefore, consistent with the specific-outcome theory, the OFC is necessary for

responding to inferred value or model-based representations of the outcome, but not

when responding can be based on a cached value.

24

Figure I.8 Effect of OFC inactivation on sensory preconditioning.

The OFC is necessary when behavior is based on inferred value. Figures show the

percentage of time spent in the food cup during presentation of the cues during each of

the three phases of training: preconditioning (A and B), conditioning (C and D), and

the probe test (E and F). OFC was inactivated only during the probe test. Cannulae

positions are shown below; vehicle (black circles), OFCi (gray circles). *P < 0.05,

**P < 0.01 or better. Error bars show SEM.

*Source of image (Jones et al., 2012)

25

Jones et al. (2012) went on further to demonstrate that not only was the OFC

required for inferred value, but that it was also critical for learning in a blocking task

based on inferred value. Control animals were impressively able to use the inferred value

cue to block learning, while rats whose OFC was inactivated during training were unable

to use inferred value to block learning (Figure I.9).

Figure I.9 Effect of OFC inactivation on learning based on inferred value.

The OFC is necessary when learning is based on inferred value. Figures show the

percentage of time spent in the food cup during presentation of the cues during

blocking (A and B) and the subsequent probe test (C and D). OFC was inactivated

during blocking. *P < 0.05, **P < 0.01 or better. Error bars show SEM.

*Source of image (Jones et al., 2012)

26

This sensory preconditioning experiment confirmed much of what we know about

the OFC, but provided a novel setting that was able to combine findings into one setting:

its role in predicting specific outcomes and its role in learning. Sensory preconditioning

demonstrated the role of the OFC in using model-based representations for behavior and

learning independent of reversal or outcome devaluation. This study led me question if

the OFC is signaling this inferred value in sensory preconditioning.

3. Orbitofrontal Cortex: Neurophysiology

Numerous studies have demonstrated that neural activity in the orbitofrontal

cortex (OFC) anticipates expected outcomes (Schoenbaum et al., 1998, Levy and

Glimcher, 2011, Padoa-Schioppa and Assad, 2006, O'Doherty et al., 2002, Gottfried et

al., 2003, Thorpe et al., 1983, Tremblay and Schultz, 1999, Kennerley et al., 2011, Sul et

al., 2010), and signal features of these expected outcomes (Schoenbaum et al., 1998,

Delamater, 2007, Ostlund and Balleine, 2007, Steiner and Redish, 2012, Luk and Wallis,

2013). However, it has also been proposed that the OFC signals a value that is

independent of those features (Padoa-Schioppa, 2011, Levy and Glimcher, 2012). The

literature that supports the role of the OFC as a value encoder is supported by single unit

activity and blood-oxygen-level dependent (BOLD) responses in the OFC that track

value (Padoa-Schioppa and Assad, 2006, Plassmann et al., 2007, Levy and Glimcher,

2011) (O'Doherty et al., 2000b, Gottfried et al., 2003). However, it cannot be ruled out

that the value in these studies is a feature of the outcome. Specific features which may

include a value feature may still be the underlying basis of the neural signal.

27

Recently, the Schoenbaum lab demonstrated OFC's neural involvement in a

Pavlovian overexpectation task (Takahashi et al., 2013) demonstrating that the OFC

signals novel estimates regarding the expected outcomes in Pavlovian overexpectation, as

described above. The OFC was demonstrated to be necessary for learning in this

Pavlovian overexpectation task (Takahashi et al., 2009). This study showed that OFC

neural activity demonstrated in-the-moment integration of cue-evoked expectations for

the expected outcome. Takahashi et al. (2013) demonstrated that OFC neural activity

increased to reward-predictive cues across training. Subsequently, these OFC neurons

increased their firing immediately when two cues were presented in compound; when the

animals did not get what they expected, this was clearly reflected in the firing rates of the

neurons as their activity decreased across compound training sessions (Figure I.10).

Additionally, when these cues were then presented again by themselves after compound

training, OFC neural activity was also decreased. These data show that the OFC reflects

the prediction of the expected outcome, and that this signaling becomes important in

updating associative representations in other areas of the brain as expectations are not

met.

28

The results from the Pavlovian overexpectation experiment strongly support the

role of the OFC in estimating new outcomes due to spontaneous integration of outcome

expectations. Specifically, these results do not support proposals that OFC represents a

value in common neural currency as neuroeconomics may suggest (Levy and Glimcher,

2011, Montague and Berns, 2002, Padoa-Schioppa and Assad, 2006, Padoa-Schioppa,

2011, Padoa-Schioppa and Assad, 2008, Plassmann et al., 2007). This is because the

neural summation was greater than the sum of the parts. Indeed, this summation

represents a novel expectation of something the rats have never experienced.

Figure I.10 Conditioned responding and cue-evoked activity summates at the

start of compound training.

Line plot indicates the ratio between normalized firing to A1/V and A2 during each

compound training session (CP1–CP4). N indicates number of cue-responsive neurons

in each session. A1/A2 ratio increased significantly in the compound phase of the

probe, and then gradually decreased (ANOVA, ∗∗p < 0.01, ∗p < 0.05). The line plot in

inset indicates normalized firing to A1/V and A2 across six trials in the second half of

the compound probe session, with red diamonds for A1 and blue squares for A2. Error

bars = SEM. *Source of image (Takahashi et al., 2013)

29

As demonstrated from both behavioral and neurophysiological experiments cited

above, there is increased evidence that OFC is signaling information about specific-

outcomes, even in new situations where the animal may imagine the value of the

upcoming outcome.

As described, the OFC is important for guiding behavior and making decisions

based off predicted outcomes and the specific-features of those outcomes. In particular,

the OFC is important to drive learning when expected outcomes or predictions are not

met.

4. Cocaine: Effects on the Orbitofrontal Cortex

Recent evidence strongly suggests that cocaine can cause structural, molecular

and functional changes in the orbitofrontal cortex (OFC) (Lucantonio et al., 2012).

Although human studies are able to demonstrate there are differences in the OFC of

people who abuse cocaine and those who do not, these studies are generally unable to

determine if the cocaine caused the differences or if the differences were there prior to

the cocaine use. In contrast, animal studies have helped to determine that cocaine has

specific effects on the OFC.

OFC structural abnormalities have been reported, including decreased gray

matter concentration as seen in Figure I.11 and decreases in frontal white matter integrity

in psychostimulant users (Franklin et al., 2002, O'Neill et al., 2001b, Ersche et al., 2011)

(Volkow et al., 1988, O'Neill et al., 2001a, Lyoo et al., 2004). These anatomical

differences have also been observed in functional neuroimaging, where a decrease in

30

glucose metabolism has been seen throughout the brain, and are particularly decreased

when actively using cocaine (Figure I.12) (London et al., 1990, Volkow et al., 1990).

When cocaine users are actively craving cocaine during periods of abstinence, the OFC is

hypermetabolic (Volkow et al., 1991). However, with prolonged withdrawal, the OFC in

cocaine users appears to be hypoactive (Volkow et al., 1993, Adinoff et al., 2001,

Volkow et al., 2001).

Figure I.11 Decreased gray matter concentration in the OFC.

On the left, whole brain renderings showing regions of decreased gray matter density (p < .01)

in the cocaine-dependent group relative to the control group. From left to right: left and right

sagittal, posterior, and anterior, right and left side, and ventral and dorsal views of regions

showing decreased gray matter density. On the right, sagital slices through the midline show

continuous deficits in gray matter extending from the ventromedial orbitofrontal (VMOC)

cortex through the anterior cingulate (AC) region.

*Source of image (Franklin et al. 2002)

31

In animal studies, rodents that self-administered amphetamines produced an

altered dendritic morphology in different areas of the brain after prolonged withdrawal

(Figure I.13). Interestingly, the medial prefrontal cortex pyramidal neurons and nucleus

accumbens medium spiny neurons demonstrated an increase in dendritic length and spine

density, while the OFC demonstrated a decrease in spine density (Kolb et al., 2004,

Crombag et al., 2005).

Figure I.12 Decreased glucose metabolism in the OFC.

Images of the brain in a healthy control and in an individual addicted to cocaine. The

images were obtained with positron emission tomography and [18F]fluoro-2-

deoxyglucose to measure glucose metabolism, which is a sensitive indicator of

damage to the tissue in the brain. Note the decreased glucose metabolism in the OFC.

*Source of image (Volkow and Li, 2004)

32

A. Cocaine: Effects on Orbitofrontal Cortex Dependent Behaviors

The above data shows that there are changes structurally and molecularly in the

OFC that occur due to cocaine and psychostimulant abuse. However, the evidence that

cocaine affects the OFC functionally comes from experiments that have demonstrated

Figure I.13 Changes in spine density.

On the right, mean (± SEM) spine density (the number of spines/10 μm) on apical and

basilar dendrites of pyramidal neurons in the medial and orbitofrontal cortices in the

amphetamine self-administration (red bars), sucrose-reward training (orange bars) or

in untreated control groups (white bars). (* or Ɨ

, significant difference at p<0.05). On

the left, brain regions (shaded areas) and neuronal types analyzed for alterations in

spine density as a function of past experience with amphetamine or sucrose taking.

Abbreviations: CA1, CA1 field of the hippocampus; DG, dentate gyrus; Nacc, nucleus

accumbens (shell); MPC, medial prefrontal cortex (Cg3); OFC, orbital frontal cortex

(AID); a, apical dendrites; b, basilar dendrites. Adapted from Paxinos and Watson

(1997).

*Source of image (Crombag et al., 2005)

33

deficits in OFC-dependent behaviors, including reversal learning, outcome devaluation,

and Pavlovian overexpectation.

Jentsch et al. (2002) demonstrated that cocaine resulted in reversal learning

deficits. In this study, monkeys were given intraperitoneal injections of cocaine (2 mg/kg

or 4 mg/kg) for 14 days. After this cocaine exposure, the monkeys were tested after 9 or

30 days of withdrawal from cocaine. As would be expected in an OFC-dependent

behavior, the monkeys who were exposed to cocaine were still able to perform the

original discrimination, but had difficulty and made many more errors when these cue-

outcome associations were reversed (Figure I.14). Since these findings, these same

deficits in reversal learning have been demonstrated across species, including in humans

(Fillmore and Rush, 2006, Ersche et al., 2008) and rats (Schoenbaum et al., 2004, Calu et

al., 2007) (Figure I.14).

34

As previously noted, there are limitations to the specificity of reversal learning

and to the explanation of what the deficits actually mean. It is important, therefore, to

consider what other OFC-dependent behaviors cocaine affects. These long-lasting effects

were demonstrated in an outcome devaluation experiment where the cocaine-exposed rats

performed similarly to OFC-lesioned rats (Figure I.15) (Schoenbaum and Setlow, 2005).

Rats were exposed to cocaine injections (30 mg/kg) for 14 days and underwent

withdrawal for 3 weeks. As seen in the OFC-lesioned experiment, rats demonstrated

Figure I.14 Effect of cocaine on reversal learning.

(a) Incorrect responses in monkeys exposed to non-contingent cocaine (4 mg/kg, once

daily for 14 d) or saline in a reversal task. Compared with controls, cocaine-treated

monkeys showed similar acquisition but impaired discrimination-reversal learning. (b)

Rats were trained to self-administer cocaine (0.75 mg/kg per infusion, 4 hours per day

for 14 days) and then tested on the an odor discrimination reversal task, after

approximately 3 months of withdrawal from the drug. Cocaine self-administration has

no effect on retention but impairs reversal learning. (c) Consecutive incorrect

responses immediately after the change in reward contingencies in chronic cocaine

users. Cocaine users show response perseveration to the previously rewarded stimulus.

*P < 0.05; **P < 0.01. (Jentsch et al., 2002, Ersche et al., 2008, Schoenbaum and

Shaham, 2008).

*Source of image (Lucantonio et al., 2012)

35

normal conditioning and normal extinction of conditioned responding, but importantly,

they were unable to change their conditioned responding in response to devaluation of the

food. They continued to respond to the cue that predicted the devalued cue. The animals

that were exposed to cocaine, it was observed, were not able to update their expectations

about the outcome to the conditioned cue. The cocaine-exposed rats were unable to use

the representation of the outcome value to guide behavior, as seen in animals with OFC

lesions.

Most recently, the effect of cocaine exposure was demonstrated to affect

Pavlovian overexpectation. The results were once again comparable to the deficits seen

Figure I.15 Effect of cocaine on outcome devaluation.

Effect of cocaine on changes in conditioned responding as a result of reinforcer

devaluation in rats. Shown is the mean percentage of time spent in the food cup during

presentation of the food-predicting cue after devaluation in the extinction probe test.

Red bars, devalued groups; blue bars, non-devalued groups. During the devaluation

test, cocaine-exposed rats fail to change conditioned responding as a result of previous

devaluation of the food reward. *p < 0.05. Adapted from Schoenbaum and Setlow,

2005.

*Source of image (Lucantonio et al., 2012)

36

with OFC inactivation during the compound cues (Lucantonio et al., 2014). In this

experiment, rats self-administered cocaine (0.75 mg/kg) for 14 days and then went

through withdrawal for three weeks. After withdrawal, rats failed to show summation

during compound training in addition to not showing spontaneous reduction in

responding to the compounded cue in the later probe test (Figure I.16).

As evidenced by previous OFC lesion and inactivation studies, cocaine use

appears to have very similar behavioral results as OFC lesions and inactivation,

demonstrating that cocaine has the potential for functionally disrupting the OFC.

Specifically, cocaine use does not appear to impair basic learning, but causes a specific

deficit in the ability to adjust behaviors in response to unexpected outcomes. These

impairments do not appear to be short term. In animal models, these deficits appear to

last for weeks to months after the last drug exposure. Thus, it seems very likely that the

OFC functional deficits that cocaine causes may contribute to the long term difficulties

seen with drug relapse and addiction.

37

B. Cocaine: Effects on Orbitofrontal Cortex Neurophysiology

The evidence from behavioral experiments described above suggests that cocaine-

exposed animals that undergo withdrawal have deficits in OFC-dependent behaviors. The

Figure I.16 Effect of cocaine on Pavlovian overexpectation.

Top and bottom rows of plots indicate sucrose and cocaine groups, respectively. (B)

Percentage of responding to food cup during cue presentation across 4 days of

compound training. Red and blue bars on the right indicate average normalized

percentage responding to A1+V1 and A2, respectively. (C) Percentage of responding

to food cup during cue presentation in the probe test. Red, blue, and white colors

indicate responding to A1 or A1+V1, A2, and A3 cues, respectively (*p < 0.05; **p <

0.01). Error bars = SEM.

*Source of image (Lucantonio et al., 2014, in review)

38

OFC acts within a circuit, and there are other areas of the brain that are also affected by

cocaine. However, further implications of systemic cocaine on the OFC are demonstrated

in neurophysiology experiments.

Similar to behavioral experiments, OFC neurons seem to be directly affected by

previous cocaine exposure even after months of withdrawal. The Schoenbaum lab has

demonstrated that cocaine-exposed rats do not develop neural correlates as controls do in

reversal learning and overexpectation (Lucantonio et al., 2014, Stalnaker et al., 2006).

Stalnaker et al. (2006) demonstrated that cocaine-exposed rats did not differ in OFC

neuronal baseline firing rates, or percentage of cue-selective or outcome-selective

neurons during initial learning. However, cocaine-exposed animals did not demonstrate

the same signaling of predicted outcomes as control animals, as their outcome-selective

neurons did not become activated during cue-sampling. In fact, the cue-selective activity

in cocaine-exposed animals was just as likely to signal an unpredicted outcome as a

predicted outcome (Figure I.17). Therefore, cocaine exposure resulted in OFC neurons

failing to signal the expected outcome. Additionally, unlike controls, rats that were

exposed to cocaine and performed poorly in the reversal learning task did not show

evidence of reversal encoding in the OFC. These results suggest that the rats who were

exposed to cocaine do not have the same ability to learn from unexpected outcomes.

39

Furthermore, in a recent study by the Schoenbaum lab (Lucantonio et al., 2014)

similar results were seen in Pavlovian overexpectation. Rats exposed to cocaine failed to

learn in response to changes in expected outcomes. However, the neural signature also

demonstrated that the cocaine-exposed rats’ OFC neurons did increase firing to reward-

predictive cues in training. But unlike the control animals, these neurons did not increase

Figure I.17 The effect of cocaine on OFC cue-selectivity neurons in reversal

learning.

Cue-selectivity indices for neurons that developed outcome-expectant firing during

the pre-criterion block, firing differentially after the rat's response, in anticipation of

either sucrose or quinine delivery. On the top row are shown the populations that

developed quinine-expectant firing, and on the bottom row are shown the populations

that developed sucrose-expectant firing. Red or blue bars represent neurons that were

significantly selective for one or the other of the two odors. In both quinine-expectant

and sucrose-expectant populations, neurons in control rats were more likely to develop

cue-selectivity to the cue that predicted their preferred outcome. Thus the distribution

for quinine-expectant neurons is skewed to the left, and that in sucrose-expectant

neurons in skewed to the right. In contrast, in both populations in cocaine-treated rats,

neurons were equally likely to develop cue-selectivity to either cue. Thus, the

distributions are symmetrically distributed around zero. *Source of image (Stalnaker et al., 2009)

40

in firing when the cues were played in compound and did not show spontaneous decline

in the activity of OFC neurons during extinction (Figure I.18).

The neural data offer further support and understanding of the behavioral results

seen in animals exposed to cocaine. The failure of OFC to signal expected outcomes

leads to a lack of feedback and therefore they are unable to learn when they experience

unexpected outcomes. Cocaine-exposed rats’ OFC neurons do not seem to signal

integration of the cue-evoked expected outcomes. This could possibly explain the delays

seen in reversal learning and the lack of overexpectation behavior. This could also

explain why human cocaine abusers have such difficulty in changing their drug-seeking

behaviors, as they are not able to update their expectations as easily when they

experience the negative consequences of drug use.

41

Figure I.18 The effect of cocaine on OFC cue-selectivity neurons in Pavlovian

overexpectation.

Conditioned responding and cue-evoked activity did not summate at the start of

compound training in the cocaine group. A. Distribution of summation of index scores

for firing to A1 and A2 in the compound probe for control sucrose animals on the top

row and cocaine-exposed animals on the bottom row. Black bars represent neurons in

which the difference in firing was statistically significant (t-test, p < 0.05). B. Line

plot indicates the ratio between normalized firing to A1+V and A2 during each

compound training session (CP-CP4). N's indicate the number of cue-responsive

neurons in each session. A1/A2 ratio increased significantly in the compound phase of

the probe for controls and then gradually decreased; while the A1/A2 ratio did not

increase significantly in the compound phase of the probe, or in subsequent sessions,

for cocaine-exposed animals.

*Source of image (Lucantonio et al., 2014, in review)

42

II. Investigating the role of the orbitofrontal cortex in inferred

value

1. Introduction

The orbitofrontal cortex (OFC) is thought to be an important part of a network for

signaling information about expected outcomes. One critical role the OFC plays within

this network appears to be that of the integrator, incorporating information about events

and predicting outcomes in novel situations or environments not specifically experienced

in the past.

Neural activity of the OFC has been shown to anticipate expected outcomes

(Schoenbaum et al., 1998, Levy and Glimcher, 2011, Padoa-Schioppa and Assad, 2006,

O'Doherty et al., 2002, Gottfried et al., 2003, Thorpe et al., 1983, Tremblay and Schultz,

1999, Kennerley et al., 2011, Sul et al., 2010, Takahashi et al., 2013). As described

previously, the Schoenbaum lab recently demonstrated that the OFC signals the novel

estimates regarding the expected outcomes in a Pavlovian overexpectation task

(Takahashi et al., 2013). In this study, it was demonstrated that neural activity in the OFC

at the time of summation increases suddenly on the very first trial of the compound cue,

then declines as the expectations of the compound cue are not met. These data clearly

demonstrate that OFC reflects the ability to predict expected outcomes in the moment.

Additional evidence supporting these conclusions is seen in that rats exhibited impaired

learning as their OFC neural signaling was absent (Takahashi et al., 2013).

43

As discussed previously, Jones et el. (2012) demonstrated that the OFC was

necessary for responding based on inferred but not cached value in a sensory

preconditioning task, supporting the view of the OFC as critical to integration.

Inactivation of the OFC during the probe test with baclofen and muscimol revealed that

rats were unable to infer value to the preconditioned cue that had previously been paired

with the reinforced cue, while the rats still responded to the cue that had been paired

directly with the reward. This experiment confirmed much of what we know about the

OFC, although sensory preconditioning was a novel setting that was able to combine

findings into one setting: its role in predicting specific outcomes and its role in learning.

Sensory preconditioning allows us to demonstrate the role of the OFC in using model-

based representations for behavior and learning independent of reversal or outcome

devaluation. The Jones et al. (2012) sensory preconditioning study demonstrated that the

OFC is involved in this ability to infer value, but in order to understand what the OFC is

actually doing neurophysiologically, we can look more specifically at what the OFC is

signaling during performance in the task. Based on these results, and the previous

findings that OFC neural activity correlated with predicting outcomes, it is hypothesized

that the OFC is signaling the inferred value in sensory preconditioning, even though the

preconditioned cue has never been paired directly with the reward.

2. Materials and Methods

A. Subjects

Twenty-nine male Long-Evans rats were obtained at 250-300g from Charles

River Labs, Wilmington, MA. Upon arrival, they were housed individually and were

44

given ad libitum access to food and water, except during behavioral training and testing.

During sensory preconditioning, rats were given 10 minutes of water each day after

behavior sessions. Rats were maintained on a 12 hour light/dark cycle and were trained

and tested during the light cycle. All behavior and recording were conducted at National

Institute on Drug Abuse, Intramural Research Program, in accordance with University of

Maryland School of Medicine and National Institutes of Health (NIH) guidelines.

B. Apparatus

Behavioral training and testing were conducted in four aluminum boxes

measuring 18" x 18" with sloping walls on the front and back of the box that narrowed to

12" x 12" at the bottom. The left and right walls were straight. In the center of either the

left or right wall a recessed dipper was placed approximately 2 cm above the floor grid.

The sucrose dipper delivered 0.04 mL of a 10% sucrose solution. Auditory cues were

used during the behavioral training and testing. The tone speaker was located on the wall

above the dipper and calibrated to ~70 dB. During Run 1, the stimuli included a tone,

siren, clicker and white noise. The rats were exposed to the tone cues produced by an

audio generator that played a pure tone frequency and a siren tone that alternated between

two frequencies every 100 ms. The clicker (2 Hz) was mounted on the wall above the

dipper, and a white noise speaker calibrated to ~65 dB was placed external to the

recording box. During Run 2, four distinct customized auditory cues were played for the

additional round of sensory preconditioning using an Arduino Uno USB Board

(Smartprojects, Italy).

45

C. Surgical Procedures

Using aseptic techniques, drivable bundles of sixteen 25-µm diameter Formvar-

insulated nichrome wires (80% nickel/20% chromium) (A-M Systems, Sequim, WA),

were manufactured and implanted as in prior recording experiments (Roesch et al.,

2007a). Electrodes were implanted in the left (n=16), right (n=8) or bilateral (n=5) OFC:

3.0 mm anterior to bregma, 3.2 mm laterally, and 4.0 mm ventral to the surface of the

brain in each rat.

D. Histology

At the end of the study, the rats were euthanized with an overdose of isoflurane.

The final electrode position was marked by the passage of current through each

microwire. The brains were then perfused with 0.9% saline and removed from the skulls,

fixed with 10% formalin and transferred to a 30% sucrose 0.1PBS at 4ºC. Brains were

then cut 50 µm, counterstained and electrode placements confirmed.

E. Behavioral Task

i) Sensory Preconditioning:

The sensory preconditioning procedure consisted of three phases (Figure II.1) , using

procedures similar to those in our prior studies (Jones et al., 2012):

46

(a) Preconditioning:

Rats were shaped to retrieve 10% liquid sucrose from the dipper (n = 28 total

subjects). After shaping, all rats received two days of sensory preconditioning involving

auditory cues. Each day, rats were exposed to two blocks of six trials. A block of trials

consisted of presentations of paired auditory cues (A→B and C→D). Specifically, a 10 s

presentation of cue A would be followed immediately by a 10 s presentation of cue B, in

a six-trial block. The second block would consist of six 10 s presentations of cue C, each

followed immediately by a 10 s presentation of cue D. The inter-trial intervals varied

from 3-6 minutes. In Run 1, auditory cues A and C were either white noise or clicker

Figure II.1 Sensory preconditioning.

Figures show the percentage of time spent in the food during presentation of the cues

during each of the three phases: preconditioning (A), conditioning (B), and the probe

test (C). The experimental timeline is schematized at the bottom of the figure.

*p<0.05. Error bars = SEM.

47

counterbalanced, and auditory cues B and D were either siren or tone counterbalanced. In

Run 2, all four customary auditory cues were counterbalanced for cues A, B, C and D.

The order of cue blocks was counterbalanced across each of the two sensory

preconditioning days.

(b) Conditioning:

After two days of preconditioning, rats underwent Pavlovian conditioning for six

consecutive days. Each day rats received a single training session in which a total of 12

trials were presented: six trials of cue B paired with a 10% liquid sucrose reward and six

trials of cue D paired with no sucrose reward. Auditory cue B was paired with three rises

of the sucrose dipper during the 10 s presentation (at 2.9, 6.4 and 9.4 s with each dipper

completing its lift in 0.6 s). Alternatively, auditory cue D was presented alone for 10 s

with no sucrose reward delivered. Trials were counterbalanced including three trials of

alternating blocks for a total of six reinforced auditory cue B presentations and six

unreinforced auditory cue D presentations. The inter-trial intervals varied between 3-6

minutes.

(c) Probe Test:

After conditioning, a single probe test day included reminder training of three

trials of B paired with sucrose and three trials of D unpaired. After the presentations of B

and D, rats were exposed to counterbalanced auditory cues A and C presented in

extinction (six presentations each). The inter-trial intervals varied between 3-6 minutes.

48

(d) Response measures:

During sensory preconditioning, the percentage of time the rats spent with their

head in the sucrose well during cue presentation was measured by an infrared photo beam

positioned at the front of the dipper. The analysis for auditory cues presented was for the

10 s of cue presentation. Baseline response measures in the corresponding pre-cue

periods were subtracted from the cue-evoked response to account for individual variation

in baseline activity. For probe tests, only the first two trials of the preconditioned cues

were analyzed for behavior.

(e) Statistical Analyses:

Data were acquired using Plexon Multichannel Acquisition Processor systems

(Dallas, TX). Raw data were processed in Matlab (Natick, MA) to extract number of

nosepokes and percentage time spent in the sucrose well for sensory preconditioning. For

all statistical tests, an alpha level of 0.05 was used. All behavioral data were analyzed in

Statistica, using repeated measures analysis of variance with Bonferroni post-hoc testing

when appropriate (p < 0.05).

F. Single-Unit Recording

Neural activity was recorded using four identical Plexon Multichannel

Acquisition Processor Systems (Dallas, TX) attached to the behavior chambers described

above. Signals from the electrode wires were amplified and filtered as described in

Roesch et al. (2007a). Waveforms with greater than a 2.5:1 signal to noise ratio were

extracted from active channels with event timestamps and were not inverted before data

analysis.

49

Wires were screened for activity daily; if no activity was detected, the rat was run

in the behavioral task and after the session the electrode assembly was advanced 40 to 80

µm.

i) Statistical Analyses

Units were sorted using Offline Sorter software from Plexon Inc. (Dallas, TX).

Sorted files were then processed and analyzed in Matlab to extract unit timestamps and

relevant event markers.

3. Results

A. Sensory Preconditioning Behavioral Results:

We recorded a total of 43 rounds of training and probe testing in the rats. Twenty-

eight rounds of the training and probe tests were from Run 1 with the auditory stimuli

that was used in previous experiments (Wied et al., 2013, Jones et al., 2012).

Additionally, fifteen rats that still had good neural data were trained and tested again with

a new set of auditory cues in Run 2.

i) Preconditioning:

In preconditioning, all rats were taught to associate two pairs of neutral auditory

cues (A→B and C→D). Food cup responding and number of nosepokes during the

presentation of each cue was taken as an index of conditioning. Rats responded at or near

baseline levels to all cues (Figure II.1A and Figure II.2A). A one-way ANOVA

comparing the percent of time the rats spent in the food cup to each cue demonstrated no

significant effects (F(3,126) = 1.02; p = 0.39). Similarly, a one-way ANOVA comparing the

50

number of nosepokes into the food cup demonstrated to each cue revealed no significant

effects (F(3,126) = 1.38; p = 0.25).

ii) Conditioning:

Over the six days of conditioning, all rats learned to discriminate between the

rewarded auditory cue B and the non-rewarded auditory cue D (Figure II.1B). Rats

increased responding to B, but not to D, during these six conditioning sessions. A 2-

factor ANOVA (cue x session) revealed significant main effects of cue (F(1,280) = 192.12,

p < 0.0001) and session (F(5,280) = 15.54, p = 0.0001), and a significant interaction

between cue and session (F(5,280) = 17.13, p < 0.0001). As seen in the control animals

previously (Wied et al., 2013) the rats acquired a discrimination between B and D

responding in session 1. Bonferroni post-hoc testing demonstrated that rats responded

significantly more to B than D in all sessions (ps < 0.0001) and responded significantly

more to B in sessions 3-6 than in session 1 (ps < 0.05). This increase in responding and

discrimination appears to be due to the prior experience with reward retrieval, as the

recording animals had a longer history of shaping than in Jones et al. (2012).

iii) Probe Test:

In the probe test, all rats responded more to B, the cue paired with sucrose, than to

any other auditory cue including the non-reinforced cue (D) and the preconditioned cues

(A, C). (Figure II.1C).

A two-factor ANOVA (reward predictive x primary versus preconditioned cues)

analyzing responding indicated a significant main effect of reward predictive cues (F(1,84)

= 190.65, p < 0.0001), a significant main effect of primary versus preconditioned cues

51

(F(1,84) = 215.19, p < 0.0001), and a significant interaction between reward predictive cues

and primary versus preconditioned cues (F(1,84) = 185.67, p < 0.0001). Bonferroni post-

hoc testing showed that rats responded significantly more to B than A, C, and D (p<

0.0001). Additionally, there was not a significant difference in the number of nosepokes

to A and C; t(42) = 0.86, p = 0.40 (Figure II.2B).

B. Sensory Preconditioning in Probe Tests that demonstrated A>C Behavior

As clearly shown in Figure II.1 and Figure II.2, there was no A greater than C

behavioral effect overall in the probe test as seen in previous studies. As stated from the

hypothesis, the central question is what neurons signal when sensory preconditioning is

successful. In order to represent an analysis of successful data, we examined only the

animals and sessions that demonstrated greater responding to A than to C in the probe

test (Figure II.3). Of the 43 rounds of training and probe testing we only saw 26 probe

tests that demonstrated greater responding to cue A than to cue C which was significantly

Figure II.2 Sensory preconditioning: Number of nosepokes during

preconditioning and probe test.

Figures show the number of nosepokes during presentation of the cues during

preconditioning (A) and the probe test (B). Error bars = SEM.

52

less than expected (χ2

1,N=43 = 4.85, p = 0.03) (Figure II.3). These sessions did not differ in

their behavior on a two-factor ANOVA (cue x group) in preconditioning in percent

responding (F1,201 = 0.15, p = 0.70) or nosepokes (F1,201 = 0.15, p = 0.70). These

sessions also did not differ in a three-factor ANOVA (cue x group x session) for

conditioning days (F1,430 = 0.02, p = 0.88 ) when compared to all 43 probe tests. The only

difference was their behavior on probe test (Figure II.4). The following data were from

22 rats, 15 sessions from Run 1 and 11 sessions from Run 2.

i) Preconditioning:

In preconditioning, rats responded at or near baseline levels to all cues (Figure

II.4A and Figure II.5B). A one-way ANOVA demonstrated no significant effects when

Figure II.3 Distribution of sessions that demonstrated greater responding to cue

A than to cue C.

Twenty-six out of 43 sessions demonstrated rats nosepokes to cue A, the cue paired

with the reinforced cue (B), more than to cue C, the cue paired with the non-reinforced

cue (D), on the first two trials of the probe test.

53

comparing the percent of time the rats spent in the dipper (F(3,75) = 1.83; p = 0.15) or

when comparing the number of nosepokes into the dipper (F(3,75) = 1.27; p = 0.29) to each

cue.

ii) Conditioning:

Over the six days of conditioning, rats learned to discriminate between the

rewarded auditory cue B and the non-rewarded auditory cue D (Figure II.1). Rats

increased responding to B, but not to D during these six conditioning sessions. A 2-factor

ANOVA (cue x session) revealed a significant main effect of cue (F(1,150) = 187.16, p <

Figure II.4 Sensory preconditioning behavior results from the sessions that

demonstrated A greater than C responding.

Figures show the percentage of time spent in the food cup during presentation of the

cues during each of the three phases from the 26 sessions that demonstrated A greater

than C behavior during probe test: preconditioning (A), conditioning (B), and probe

test (C). The experimental timeline is schematized at the bottom of the figure.

*p<0.05. Error bars = SEM.

54

0.0001) and a significant interaction between cue and session (F(5,150) = 2.60, p = 0.028).

Bonferroni post-hoc testing demonstrated that rats responded significantly more to B than

D in all sessions (ps < 0.0001), and responded significantly more to B in session 6 than in

session 1 (p < 0.05).

iii) Probe Test:

In the probe test, rats responded more to B, the cue paired with sucrose, than to D,

the non-reinforced cue (Figure II.4C and Figure II.5B). Rats exhibited significantly

greater responding within the average of the first two trials to the preconditioned cue (A),

that had been previously paired with the rewarded cue (B), than to the preconditioned

control cue (C), that had been previously paired with the non-reinforced cue (D).

A two-factor ANOVA (reward predictive x primary versus preconditioned cues)

analyzing responding indicated a significant main effect of reward predictive cues (F(1,50)

= 190.92, p < 0.0001), a significant main effect of primary versus preconditioned cues

(F(1,50) = 64.82, p < 0.0001), and a significant interaction between reward predictive cues

and primary versus preconditioned cues (F(1,50) = 94.17, p < 0.0001). Bonferroni post-hoc

testing showed that rats responded significantly more to B than D (p< 0.0001) and

significantly more to A than to either C or D (p’s < 0.05). Similarly, there was a

significant difference in the number of nosepokes to A and C; t(25) = 4.27, p = 0.0002.

These results demonstrate greater response to A, the cue paired previously with the

reinforced cue (B), than to C, the cue previously paired with the non-reinforced cue (D).

55

C. Neural Probe Test Results:

The results to be presented below came from the 26 sessions in which we

observed evidence of rats responding to A more than C during probe test. The neural data

were analyzed comparing the 10 s auditory cues.

i) Preconditioning:

On the first day of preconditioning, we recorded 193 neurons. Figure II.7A and

Figure II.7C show that 46 of these neurons exhibited either a significant excitatory phasic

response (n = 32) and/or inhibitory phasic response (n = 15) over baseline to at least one

of the auditory cues. Additionally, there was a strong correlation demonstrating that A-C

firing significantly correlated to B-D firing (r = 0.88, p < 0.0001). Similar firing patterns

are seen in paired auditory cues A and B as well as to C and D (Figure II.6D).

Figure II.5 Sensory preconditioning: Number of nosepokes during

preconditioning and probe test from sessions that demonstrated A greater than C

responding.

Figures show the number of nosepokes during presentation of the cues during

preconditioning (A) and the probe test (B) from the 26 sessions that demonstrated A

greater than C behavior during probe test. Error bars = SEM.

56

ii) Probe Test:

We recorded 225 neurons in the probe test in animals that demonstrated

numerically greater responding to the inferred value cue A than to the control cue C.

Figure II.7A shows that 64 of these neurons exhibited either a significant excitatory

Figure II.6 Proportion of neurons with significant cue-selective activity during

the preconditioning.

A: Proportion of units that were significantly responsive to one of the four cues in

preconditioning by either increasing or decreasing firing rate compared to baseline. B:

Index summary of all excitatory (●) and inhibitory (◌ ) units recorded during

preconditioning. A significant correlation demonstrated A-C firing predicted B-D

firing ( r = 0.88, p < 0.0001). C: Proportion of cue-responsive units that increased

firing rate compared to baseline during preconditioning to auditory cues A,B,C and/or

D. D. Population responses of all 32 cue-responsive neurons to each set of 10 s cues:

A→B cues (in black, with blue and black bars indicating the period of each cue

presentation), and C→D (in gray, with red and gray bars indicating the period of each

cue presentation). Gray shading represents SEM.

57

phasic response (n = 44) and/or inhibitory phasic response (n = 22) over baseline to at

least one of the cues. As seen in Figure II.7C, the majority of the cue-evoked neurons

fired to B paired with reward (n = 27), while cue D that had been paired with no reward

had the least number of cue-responsive neurons (n = 6). Preconditioned cues A and C had

equal numbers of neurons represented (n's = 8). As seen in Figure II.7B, a significant

correlation demonstrated that B-D firing predicted A-C firing ( r = 0.25, p = 0.044).

Although the proportion of cue-responsive units that fired to the various auditory cues

changed over conditioning, we see the same correlation as we did on the first day of

preconditioning. As B became predictive of reward and D of non-reward, the relationship

between cues A and B, as well as C and D did not fade from the original significant

correlation seen in Figure II.6B.

58

The distributions of index scores for mean firing to A-C and B-D in the probe test

are seen in Figure II.8A and Figure II.8C. The neurons that fired significantly different

from baseline, shown in black, demonstrated a significant shift from zero for B-D (p <

0.05), but not for cues A-C. These differences were further demonstrated when

population responses to all 44 cue-responsive neurons were plotted (Figure II.8B and

Figure II.8D), and a paired t-test comparing cues B and D demonstrated a significant

Figure II.7 Proportion of neurons with significant cue-selective activity during

the 10 second auditory cues during probe test.

A: Proportion of units that were significantly responsive to one of the four cues by

either increasing or decreasing firing rate compared to baseline during probe test

during the 10 s auditory cues. B: Index summary of all excitatory (●) and inhibitory

(◌ ) units recorded during probe test. A significant correlation demonstrated B-D

firing predicted A-C firing ( r = 0.25, p < 0.05). C: Proportion of cue-responsive units

that increased firing rate compared to baseline to auditory cues A,B,C and/or D during

probe test.

59

difference in excitatory mean firing rates (t31 = 4.02, p < 0.001), but no significant

difference between A and C (t13 = 0.95, p = 0.36).

Figure II.8 Cue-evoked activity during probe test.

A,C: The distribution of index scores for mean firing to A-C and B-D during the

probe test. The black bars represent excitatory neurons in which the difference in

firing to cues was significantly different from baseline. The distribution of index

scores did not shift from zero for A-C (p> 0.05) (fig. A), but did significantly shift

above zero for B-D (p < 0.05) (fig. C) . B,D: Population responses of all 44 cue-

responsive neurons to A and C (fig. B), and B and D (fig. D). There were no

significant differences in mean firing to A versus C, but there were significant

differences in mean firing rates to B versus D (t31 = 4.02, p < 0.001). Light shadowing

in blue, red, gray and black represent SEM.

60

4. Discussion

In this study, we were able to analyze sessions that responded more to the inferred

value cue A that had been preconditioned with the rewarded cue B than to cue C that had

been preconditioned with the non-rewarded cue D. Throughout training and testing there

was an increase in the number and fraction of neurons responding to B from the first day

of preconditioning to the probe test. On probe test, there were significantly more neurons

with greater mean firing rates to the reinforced cue B than to the non-reinforced cue D.

Additionally, we saw a significant correlation that demonstrated B firing correlated with

A firing, and D firing correlated with C firing on probe test. This was the same

correlation that was observed on the first day of preconditioning. Although it is clear

from these scatter plots and correlations that A-B and C-D firing rates separate with

learning, it appears these relationships do not go away completely. In fact, the significant

correlation on probe test suggests there is still a strong enough relationship between these

cues to make an inference.

However, it cannot be ruled out that due to the lack of overall A greater than C

behavior, and the lack of a significant difference seen between A and C mean firing, that

the OFC does not signal this inferred value. From previous experiments, we know that

OFC is somehow involved in inferred value and, therefore, instead of signaling this

inferred value, perhaps the OFC is critical because it signals the meaning or value of B.

We also cannot rule out that the apparent sessions that we analyzed showing A greater

than C behavior were essentially random, as this lack of responding was statistically

different from what was expected from previous experiments. The behavioral results in

this study are different from our previous studies in which we saw very few failed

61

sessions. In the few sessions that did fail in previous studies, typically the animals’

behavior was at baseline, and not the C greater than A behavior seen in many of our

failed sessions in this experiment. Therefore, even in the sessions we analyzed, we are

not able to conclusively state that we are analyzing actual conditioned responding versus

just random behavior. If we are observing just random behavior, then failure to see

encoding in OFC is exactly what we would have predicted.

There are a number of differences in this recording experiment in comparison to

our previous experiments in which the majority of rats demonstrated the ability to infer

value. This was the first time this behavior was run in recording rigs rather than behavior

rigs. The recording rigs are a different size and require the animals to behave with a cable

attached to their heads for all conditions of the training and testing. Additionally, due to

the cable attached to their head, typically the rats needed more shaping in order to

retrieve the reward. Moreover, the reinforcer was a liquid sucrose reward delivered by a

dipper. This differed from previous successful sensory preconditioning studies that used a

food reward (Wied et al., 2013, Jones et al., 2012). The change was made for recording in

order to eliminate the artifact caused by the chewing of the sucrose pellets. Furthermore,

this was the first time that we ran a second round of behavior with new auditory cues in

order to record more neurons.

Interestingly, there is promising data that demonstrate that B-D firing correlates

with A-C firing even though we do not see a significant difference in the mean firing

rates between the inferred value cue and the cue that predicted the non-reinforced cue.

This may be due to the limited number of neurons that responded to these cues. To more

clearly determine if the OFC is signaling these inferred values, or if the OFC is signaling

62

the meaning or value of B, it will be necessary to run more rats in a similar task as above,

perhaps using sucrose reward pellets instead of a liquid sucrose reward to eliminate one

variable that was changed from the original successful behavioral studies. Although

chewing artifact may be seen, the most important aspect is that the rats must demonstrate

the behavior robustly in order to make any definitive conclusions about what the OFC is

signaling. These experiments are currently being conducted in the Schoenbaum lab and

we hope to clarify the role of the OFC neurophysiologically in its ability to infer value.

63

III. Disruption of model-based behavior and learning by cocaine

self-administration in rats1

1. Abstract

Rationale: Addiction is characterized by maladaptive decision-making, in which

individuals seem unable to use adverse outcomes to modify their behavior. Adverse

outcomes are often infrequent, delayed, and even rare events, especially when compared

to the reliable rewarding drug-associated outcomes. As a result, recognizing and using

information about their occurrence puts a premium on the operation of so-called model

based systems of behavioral control, which allow one to mentally simulate outcomes of

different courses of action based on a knowledge of the underlying associative structure

of the environment. This suggests that addiction may reflect, in part, drug-induced

dysfunction in these systems. Here we tested this hypothesis.

Objectives: To test whether cocaine causes deficits in model-based behavior and

learning independent of requirements for response inhibition or perception of costs or

punishment.

Methods: We trained rats to self-administer sucrose or cocaine for 2 weeks. Four

weeks later, the rats began training on a sensory preconditioning and inferred value

blocking task. Like devaluation, normal performance on this task requires representations

of the underlying task structure; however, unlike devaluation, it does not require either

response inhibition or adapting behavior to reflect aversive outcomes.

Results: Rats trained to self-administer cocaine failed to show conditioned

responding or blocking to the preconditioned cue. These deficits were not observed in

1 Heather M. Wied, Joshua L. Jones, Nisha K. Cooch, Benjamin A. Berg, and Geoffrey Schoenbaum.

As published in Psychopharmacology, August 16, 2013.

64

sucrose-trained rats nor did they reflect any changes in responding to cues paired directly

with reward.

Conclusions: These results show that cocaine disrupts the operation of neural

circuits that mediate model-based behavioral control.

Keywords: addiction, cocaine, orbitofrontal, sensory preconditioning, blocking

2. Introduction

Addiction is characterized by a pattern of maladaptive decision-making in which

subjects continue to seek drugs despite serious adverse consequences (Gawin, 1991,

Mendelson and Mello, 1996). This pattern of behavior is captured in the diagnostic

criteria for substance dependence in the DSM-IV-TR (APA, 2000), which require drug-

seeking to be persistent in the face of attempts to stop or cut back in order to avoid

various negative outcomes or costs. While there are a number of ways to describe such

apparent deficits – inability to inhibit established behaviors (Jentsch and Taylor, 1999,

Winstanley et al., 2010), increased desire or wanting of the drug (Robinson and Berridge,

2008, Robinson and Berridge, 2000), loss of sensitivity to punishment or costs

(Vanderschuren and Everitt, 2004, Deroche-Gamonet et al., 2004) or disruption of the

normal homeostatic set point of reward and anti-reward systems (Koob and Le Moal,

2008) – none of these sufficiently captures the complex array of the behavioral deficits.

Addicted individuals clearly can act affirmatively to obtain desired outcomes and can

even learn relatively complex strategies or policies in order to obtain drugs. They can

also redirect behavior when confronted with immediate consequences or costs. Yet,

65

despite these apparently intact abilities, they are unable to modify drug seeking in

response to real world events that most people would respond to appropriately.

One distinguishing feature of these real world events – one which is typically not

captured by experimental manipulations – is that the certain consequences of drug use

tend to be highly variable and unpredictable, delayed, and even rare events. Often, these

consequences are adverse, such as financial and family problems, physical injury, or

incarceration. All of these are all highly likely over the long-term; however, they tend to

be unlikely in the very short term, at least as compared to the immediate goal and certain

experience associated with the drug-associated cues and drug high, or alleviation of

withdrawal symptoms. Further, these long-term events are unlikely to happen over and

over again, so it is difficult to learn about them in the same way that one can learn about

the repetitive, immediate rewarding aspects of drug use. As a result, the successful

application of information about these consequences likely requires so-called model-

based systems of behavioral control, which allow one to utilize information about the

causal structure of the environment, which may be only anecdotally or incidentally

related to the behavior at hand, in order to mentally simulate the possible outcomes of a

behavior (Daw et al., 2005, Niv et al., 2006). Failure of these systems would result in

behavior – and learning - that is heavily dependent on cached or pre-computed policies

developed through prior direct experience. To that end, learning and behavior after

cocaine exposure appear normal or even enhanced under conditions in which model-free

systems are sufficient (Wyvell and Berridge, 2001, Taylor and Horger, 1999, Taylor and

Jentsch, 2001, Roesch et al., 2007b, Simon et al., 2007, Mendez et al., 2010).

66

Thus, the core behavioral features of addiction may reflect, in part, either a pre-

existing or drug-induced dysfunction in the model-based control of behavior (Lucantonio

et al., 2012). Consistent with this idea, studies using reinforcer devaluation have shown

that exposure to psychostimulants, either passively or by active self-administration,

causes long-lasting deficits in the ability of rats to adjust learned behaviors after changes

in the value of the associated reward (Nelson and Killcross, 2006, Schoenbaum and

Setlow, 2005). These deficits are observed spontaneously after devaluation (i.e. without

new learning) and appear in the face of apparently normal acquisition of both the

behavior (CS→US or R→US) and of the new reward value (US→illness).

While these studies are consistent with the idea that some drugs affect model-

based processing, they are similar and thus provide only a single data point. Further, they

require the animal to appreciate an adverse outcome in order to redirect or withhold

responding. This leaves open, at least somewhat, questions over whether the deficits

reflect problems with specific functions, such as sensitivity to punishment or the ability to

inhibit responding, rather than a more general impairment in the integrity of model-based

processing, which would be orthogonal to these idiosyncratic task requirements. The case

that drugs can disrupt model-based processing would be much strengthened by the

provision of convergent evidence from a task that shares the requirement for inferential

reasoning that defines reinforcer devaluation, but does not depend on either of these more

specific capabilities. Here we provide such evidence, using a sensory preconditioning and

inferred value blocking task in which we have previously demonstrated the role of the

orbitofrontal cortex in model based behavior and learning (Jones et al., 2012).

67

3. Materials and Methods

A. Subjects:

Forty-five adult male Long-Evans rats (Charles River Laboratories, Wilmington,

MA) (n = 30 controls including 13 naïve, non-surgical controls and 17 sucrose-trained

surgical controls; n = 15 cocaine-trained) weighing 275-325 g on arrival were

individually housed and given ad libitum access to food and water, except during

behavioral training and testing. During these procedures, rats were food deprived to 85-

90% of their baseline body weight. Rats were maintained on a 12-hour light/dark cycle

and trained and tested during the light cycle. Experiments were performed at the

University of Maryland School of Medicine and National Institute on Drug Abuse,

Intramural Research Program, in accordance with University of Maryland and NIH

guidelines.

B. Apparatus:

Self-administration and behavioral training and testing were conducted in

standard behavior boxes measuring 30.48 cm x 25.40 cm x 30.48 cm (Coulbourn

Instruments) enclosed within sound-attenuating boxes. During self-administration, each

chamber was equipped with two 4-cm wide levers on opposite walls located 8 cm above

the grid floor. The lever located on the right side of the chamber was active and

retractable and led to activation of the infusion pump, while the lever on the left side of

the box was inactive and stationary. For cocaine self-administration, each rat had silastic

tubing that was protected by a metal spring that extended from the intravenous catheter to

a liquid swivel (Instech Laboratories, Plymouth Meeting, PA) mounted to an arm fixed

68

outside the behavior chamber. Tygon tubing extended from the liquid swivel to an

infusion pump located outside the chamber. For sucrose self-administration, 0.04 mL of a

10% sucrose solution was placed in a recessed dipper in the center of the right wall

approximately 2 cm above the floor grid.

Sensory preconditioning and blocking were conducted in a different room, but

using the same type of standard apparatus in sound attenuating boxes. A recessed food

cup was placed in the center of the right wall approximately 2 cm above the floor. The

feeder was mounted outside the behavior chamber and delivered 45 mg sucrose pellets

into the food cup. Auditory cues were used during the behavioral training and testing.

The tone speaker was located on the wall above the food cup and calibrated to ~70 db.

The tone cues were produced by an audio generator that produced three tones, one as a

pure tone frequency, and a siren tone that alternated between two frequencies every 100

ms. In addition, a clicker (2 Hz) was mounted on the wall above the food cup, and a

white noise speaker calibrated to ~65 db was placed external to the behavior box behind

the right wall. Visual cues that were used during inferred value blocking included a house

light and cue light that were placed on the wall to the left or the right of the food cup

approximately 10 cm above the floor.

C. Surgical procedures:

Thirty-two rats underwent surgery for implantation of chronic intravenous jugular

catheters. Rats were anaesthetized with ketamine (100 mg/ kg, i.p., Sigma) and xylazine

(10 mg/kg, i.p., Sigma). A silastic catheter was inserted into the right jugular vein and

passed subcutaneously to either the top of the skull where it was attached to a modified

22-gauge cannula (Plastics One, Roanoke, VA) and mounted to the rat’s skull with dental

69

cement and skull screws or fixed to a cannula on nylon mesh which was sutured

subcutaneously between the scapulae. Carprofen (5 mg/kg, s.c., Pfizer) was given during

surgery and for two days post-surgery to relieve post-operative pain. After surgery, and

throughout self-administration training, catheters were flushed daily with

gentamicin/saline solution (0.16% gentamicin, BioSource International) to prevent

infection.

D. Self-Administration:

Rats recovered for 14 days following surgery before starting self-administration

training. These rats were trained to lever press for cocaine-HCl (0.75 mg/kg/infusion; n =

15) or sucrose (10% w/v; n = 17) for 14 days (Figure III.1). Rats were trained to press the

active lever to self-administer cocaine under a fixed ratio (FR1) schedule of

reinforcement with each press delivering a 4 s infusion of cocaine. Each infusion was

followed by a 40 s timeout period. All self-administration sessions lasted 3 h, with 15-

min timeout periods after each hour. To prevent overdose, the number of cocaine

infusions was limited to 20 per hour such that when the rat had received 20 cocaine

infusions the active lever was retracted for the remainder of the hour. The same

procedures were followed for animals that self-administered sucrose, except the rats were

shaped to press the active lever to receive 0.04 mL of a 10% sucrose solution delivered in

a recessed dipper.

E. Sensory Preconditioning:

Catheterized rats (n=32) began sensory preconditioning four weeks after the end

of self-administration. At the same time, we also trained naïve rats (n=13) who had

received neither surgery nor any prior training. The sensory preconditioning procedure

70

consisted of three phases (Figure III.2), using procedures identical to those in our prior

study (Jones et al., 2012):

i) Preconditioning:

Rats were shaped to retrieve sucrose pellets from the food cup in a single session

that included 10 deliveries of two 45 mg sucrose pellets (n = 45 total subjects; n = 30

controls; n = 15 cocaine). After shaping, all rats received two days of sensory

preconditioning involving auditory cues. Each day, rats were exposed to two blocks of

six trials. A block of trials consisted of presentations of paired auditory cues (A-B and C-

D). Specifically, a 10 s presentation of cue A would be followed immediately by a 10 s

presentation of cue B, in a six-trial block. The second block would consist of six

presentations of the pairing a 10 s presentation of cue C followed immediately by a 10 s

presentation of cue D. The inter-trial intervals varied from 3-6 minutes. Auditory cues A

and C were either white noise or clicker counterbalanced, and auditory cues B and D

were either siren or tone counterbalanced. The order of cue blocks was counter-balanced

across each of the two sensory preconditioning days.

ii) Conditioning:

After two days of preconditioning, rats underwent Pavlovian conditioning for six

consecutive days. Each day rats received a single training session in which a total of 12

trials were presented: six trials of cue B paired with a food reward and six trials of D

paired with no food reward. Auditory cue B was paired with three sucrose pellets that

were delivered into the food cup during the 10 s presentation (one pellet was delivered

after 3, 3 and 4 s time elapsed). Alternatively, auditory cue D was presented alone for 10

s with no sucrose delivered. Trials were counterbalanced including three trials of

71

alternating blocks for a total of six reinforced auditory cue B presentations and six

unreinforced auditory cue D presentations. The inter-trial intervals varied between 3-6

minutes.

iii) Probe Test:

After conditioning, a single probe test day included reminder training of three

trials of B paired with sucrose and three trials of D unpaired. After the presentations of B

and D, rats were exposed to two blocks of counterbalanced auditory cues A and C

presented in extinction (six presentations each). The inter-trial intervals varied between 3-

6 minutes.

F. Inferred Value Blocking:

This procedure consisted of two training phases (Figure III.3) using procedures

identical to those in our prior study (Jones et al., 2012):

i) Blocking:

A subset of rats from the prior experiment (n = 35; n = 22 controls; n = 13

cocaine) underwent additional compound conditioning to test whether cocaine would also

disrupt the normal ability of the inferred value to block new learning. These rats were

trained for two days with preconditioned auditory cues A and C paired with novel light

cues X and Y (houselight; flashing cue light; counterbalanced), with all presentations

reinforced with a food reward. Auditory cues A and C were played together in compound

with visual cues X and Y (i.e. AX and CY) for 10 s each. Each compound cue was

presented in two blocks of three trials, alternating cue blocks, for a total of 12 trials each

day (6 AX and 6 CY, counterbalanced). Each 10 s compound cue was reinforced by the

72

delivery of a sucrose pellet at 3, 6, and 10 s after cue onset. Inter-trial intervals were

between 3-6 minutes.

ii) Probe Test:

After compound conditioning, rats were given a single probe test consisting of

reminder training of three trials each of AX and CY paired with normal sucrose delivery,

followed by two blocks of visual cues X and Y presented alone (four presentations each)

for 10 s each without sucrose delivery. The inter-trial intervals varied between 3-6

minutes. The first three trial presentations of X and Y were analyzed.

G. Response measures:

During self-administration, active and inactive lever pressing was recorded.

During sensory preconditioning, the percentage of time rats spent with their head in the

food cup during cue presentation was measured by an infrared photo beam positioned at

the front of the food cup. The analysis for auditory and visual cues presented alone was

restricted to the last 5 s of cue presentation. Baseline response measures in the

corresponding pre-cue periods were subtracted from the cue-evoked response to account

for individual variation in baseline activity. Rearing behavior to visual stimuli during

inferred value blocking in both the compound conditioning sessions and blocking probe

test was scored from video recordings with a handheld timer. In order to correct for time

spent rearing (Takahashi et al. 2009; Jones et al. 2012), the percentage of time responding

during the visual stimuli, and corresponding baseline periods, was calculated as follows:

% of responding = 100* ([% of time in food cup]/[100 - % of time rearing]).

73

H. Statistical Analyses:

Data were acquired using Coulbourn Graphic State 2 software (Coulbourn

Instruments). Raw data were processed in Matlab to extract lever presses in self-

administration and percentage time spent in the food cup for sensory preconditioning and

inferred value blocking. For all statistical tests, an alpha level of 0.05 was used. The

naïve and surgical control groups did not differ at the end of conditioning or during probe

testing, thus their data were collapsed to form a single control group. All data were

analyzed in Statistica, using repeated measures analysis of variance with Bonferroni post-

hoc testing when appropriate (p < 0.05).

4. Results

A. Self-Administration:

Chronic intravenous jugular catheters were implanted in rats (17 sucrose controls

and 15 cocaine). After recovering from surgery, these rats self-administered either

cocaine or sucrose for 14 days (Figure III.1). Rats acquired cocaine and sucrose self-

administration as demonstrated by a significant interaction between lever and day in the

analysis of active and inactive lever presses for cocaine (F(13,338) = 2.01, p < 0.05) and

sucrose (F(13,403) = 5.99, p < 0.01). In addition, there was a significant effect of training

day for the number of infusions of cocaine (F(13,182) = 15.96, p < 0.01) and number of

sucrose deliveries (F(13,208) = 5.24 , p < 0.01).

74

B. Sensory Preconditioning:

i) Preconditioning:

In preconditioning, rats were taught to associate two pairs of neutral auditory cues

(A→ B and C→ D). Food cup responding during the presentation of each cue was taken

as an index of conditioning. Rats in both groups responded at or near baseline levels to all

cues (Figure III.2. A and B). A 2-factor ANOVA (cue x treatment) comparing the percent

of time the rats spent in the food cup demonstrated no significant main effects or any

interactions (F's < 2.08; p's > 0.16) (Figure III.2A and B).

Figure III.1 Self-administration.

Rats were trained to press the active lever to self-administer a 10% liquid sucrose solution or

cocaine-HCl (0.75 mg/kg/infusion). Figures show the mean number of lever presses on the

active and passive levers for sucrose controls (A) and cocaine rats (B) over 14 days. The

mean number of sucrose or cocaine reward deliveries during each session is also shown. Error

bars = SEM.

* Source of image (Wied et al., 2013)

75

ii) Conditioning:

During the six days of conditioning, the rats learned to discriminate between the

rewarded auditory cue B and the non-rewarded auditory cue D (Figure III.2C and D).

Rats in both groups increased responding to B, but not to D, during these six conditioning

sessions. A 3-factor ANOVA (cue x treatment x session) revealed significant main

effects of cue (F(1,86) = 221.99, p < 0.01) and session (F(5,430) = 27.62, p < 0.01) and

significant interactions between treatment and cue (F(1,86) = 9.88, p < 0.01), treatment

and session (F(5,430) = 2.34, p < 0.05), cue and session (F(5,430) = 13.01, p < 0.01), and cue,

treatment and session (F(5,430) = 2.38, p < 0.05). Effects involving treatment were

primarily due to differences between the two groups in the initial rate of conditioning;

controls acquired the discrimination between B and D faster than cocaine animals (p <

0.05) (session 1 versus session 3). Thus, while control and cocaine-experienced rats

responded similarly to cue D in all sessions (Bonferroni post-hoc correction; p’s > 0.05),

responding to cue B differed in the first session (p < 0.05) but not thereafter (p’s > 0.05).

This likely reflected the prior experience with reward retrieval for controls. Accordingly,

there were no significant differences between groups by the end of conditioning (p’s >

0.05).

iii) Probe Test:

In the probe test, both control and cocaine-experienced rats responded more to B,

the cue previously paired with sucrose, than to D, the non-reinforced cue (Figure III.2E

and F). Controls also exhibited significantly greater responding to the preconditioned cue,

A, which had been previously paired with the rewarded cue, B, than to the preconditioned

control cue, C (Figure III.2E). Cocaine-experienced rats showed no difference in

76

responding to the preconditioned cues A and C (Figure III.2F). A 2-factor ANOVA (cue

x treatment) indicated a significant main effect of cue (F(3,129) = 85.76, p < 0.01) and a

significant interaction between cue and treatment (F(3,129) = 2.90, p < 0.05). Bonferroni

post-hoc testing showed that both groups responded significantly more to B than D (p’s <

0.05), and controls responded significantly more to A than to either C or D (p’s < 0.05),

while cocaine-experienced rats responded similarly to A, C, and D (p’s > 0.05).

77

Figure III.2 Cocaine self-administration disrupts the expression of behavior that

depends upon inferred values.

The experimental timeline is schematized at the bottom of the figure. Figures show the

percentage of time spent in food cup during presentation of the cues during each of the three

phases of training: preconditioning (A,B), conditioning (C,D), and the probe test (E,F).

Control rats are presented in the top panels (A,C,E) and the cocaine-experienced rats are

shown in the bottom panels (B,D,F). *p < 0.05, **p < 0.01 or better. Error bars = SEM.

* Source of image (Wied et al., 2013)

78

C. Inferred Value Blocking:

i) Blocking:

During the two days of compound conditioning, the pre-conditioned cues, A and

C, were presented in compound with new visual cues, X and Y. AX and CY were both

reinforced with the same sucrose reward previously paired with B. A 3-factor ANOVA

(treatment x cue x session) demonstrated significant main effects of session (F(1,66) =

22.69, p < 0.01) and treatment (F(1,66) = 9.62, p < 0.01) and a significant interaction

between cue and session (F(1,66) = 6.35, p < 0.05). While cocaine-experienced rats

exhibited slightly lower responding overall, there were no significant interactions with

treatment and other factors (F's < 0.91, p's > 0.34) (a).

ii) Probe Test:

In the probe test, rats were presented with reinforced AX and CY reminder trials,

followed by probe trials, in which X and Y were presented alone. A 2-factor ANOVA

(treatment x cue) on conditioned responding during the probe trials revealed a significant

interaction between cue and treatment (F(1,33) = 8.83, p < 0.05). Bonferroni post-hoc

testing showed that controls responded significantly more to Y than to X (p < 0.05)

(Figure III.3C), whereas rats exposed to cocaine responded similarly to these two cues (p

> 0.05) (Figure III.3D). Subsequent comparisons of cue-evoked responding to responding

in the baseline period showed that while controls exhibited both a significant main effect

of period (F(1,42) = 19.7, p < 0.01) and a significant period by cue interaction (F(1,42) =

79

7.00, p < 0.05), cocaine-experience rats exhibited only a main effect of period (F(1,24) =

7.68, p < 0.05; other p's > 0.66).

Figure III.3 Cocaine self-administration disrupts learning that depends upon inferred

values.

The experimental timeline is schematized at the bottom of the figure. Figures show the

percentage of time spent in food cup during presentation of the cues during blocking

conditioning (A,B) and the subsequent probe test (C,D). Control rats are presented in the top

panels (A,C) and the cocaine-experienced rats are shown in the bottom panels (B, D). **p <

0.01 or better. Error bars = SEM.

* Source of image (Wied et al., 2013)

80

5. Discussion

Here, we have shown that cocaine self-administration disrupts behavior and

learning that depends upon inferred value. In a sensory preconditioning task, cocaine-

experienced rats were not able to use the relationship between the preconditioned cue and

the subsequently conditioned cue to infer value in order to guide behavior, nor were they

able to mobilize this information in order to modulate learning during the blocking task.

These effects were enduring, appearing during probe tests conducted 5-6 weeks after the

last contact with cocaine. In addition, they were specific inasmuch as the cocaine-

experienced rats responded like controls to the cues that were directly paired with reward.

These data add to prior reports that psychostimulants affect reinforcer devaluation

(Nelson and Killcross, 2006, Schoenbaum and Setlow, 2005). As in the sensory

preconditioning task used here, normal changes in behavior after reinforcer devaluation

requires access to a model-based representation of the causal structure of the task, so that

never before experienced consequences can be mentally simulated. As noted earlier, rats

that have been exposed to psychostimulants show long-lasting deficits in their ability to

inhibit learned Pavlovian and instrumental behaviors after reinforcer devaluation. Our

current results show a similar deficit, but in a setting that requires neither response

inhibition nor the ability to perceive and respond appropriately to aversive information.

In addition, the current data extend these findings and show that these operations are

impacted both for the purpose of guiding behavior appropriately as well as for new

learning derived from this information, since cocaine-experienced rats were also unable

to use the inferred value of the preconditioned cue to block learning. Overall, these

81

results substantially strengthen the idea that exposure to some addictive drugs disrupts the

neural circuits that mediate model-based processing.

Currently, candidates for mediating this model-based processing include

prefrontal regions, such as the medial and orbital prefrontal cortices. Changes in

instrumental responding after reinforcer devaluation as well as extinction learning,

particularly recall, depend critically on input from medial prefrontal cortex to dorsal

striatal regions (Milad and Quirk, 2002, Quirk et al., 2000, Rhodes and Killcross, 2004,

Rhodes and Killcross, 2007, Killcross and Coutureau, 2003, Corbit and Balleine, 2003,

Zapata et al., 2010). Similarly, flexible responding to Pavlovian cues after devaluation

and in other settings requires orbitofrontal output that impacts areas in the dorsal and

ventral striatum (Pickens et al., 2003, Izquierdo et al., 2004, Machado and Bachevalier,

2007, West et al., 2011). Altered prefrontal processing would therefore be expected to

lead to behavioral deficits in these areas. Indeed, orbitofrontal inactivation causes

precisely the deficits reported here after cocaine use in the sensory preconditioning and

inferred value blocking tasks (Jones et al., 2012).

Despite the similarity of our cocaine-induced deficits to those seen in OFC-

inactivated rats (Jones et al., 2012), the current data cannot eliminate the possibility that

the deficit occurs because cocaine-experienced rats are unable to acquire the stimulus-

stimulus association. That is, because cocaine-exposure occurred prior to any

conditioning, it is possible that the probe test deficits are a result of an inability to learn

the stimulus-stimulus contingency in the first phase of the experiment. Such an effect

might reflect an impact of cocaine on hippocampal regions known to be important for

model-based behaviors and stimulus-stimulus learning (Wimmer and Shohamy, 2011,

82

Wimmer and Shohamy, 2012). However we are not aware of evidence that stimulus-

stimulus learning is impaired by cocaine exposure and, given the strong evidence

implicating the prefrontal regions in addiction and in the type of processing isolated in

later phases of the task, we would view this as a more parsimonious explanation of the

current results.

It is also worth noting that although deficits in some model-based behaviors have

been conceived of as reflecting strengthened processing of the countervailing “habits” in

striatal regions (Robbins and Everitt, 1999, Belin and Everitt, 2008), the current results

cannot be easily explained by such a mechanism. This is because stronger encoding of

the cached value or policy associated with the rewarded cue, in concert with an intact

model-based system, should have led to stronger, not weaker, responding to the

preconditioned cue, and better, not worse, blocking. Thus, the only reasonable

explanation that accounts for the current results is alterations in model-based processing.

While this does not preclude a contribution of stronger habits to other aspects of drug-

seeking, it is, in fact, consistent with what we and others have reported, which is that

psychostimulants cause rather modest changes in information processing in dorsal

striatum (Takahashi et al., 2007) while substantially altering that in the orbitofrontal

cortex (Homayoun and Moghaddam, 2006, Stalnaker et al., 2006).

So, what are the implications of these findings for understanding and, more

importantly, addressing behavioral issues in addiction? One implication is that

individuals with a history of psychostimulant use will be less responsive than other

patients to so-called talking therapies, which require insight and mental simulation to

have their effect. This would be consistent with the high rates of relapse after even

83

apparently successful psychotherapeutic interventions (Dutra et al., 2008). A second

implication is that psychostimulant users are likely to be less adept than other patients at

learning when outcomes are not as rewarding as expected, especially to the extent that

these differences are defined by observational and inference based evidence. As noted in

our introduction, we believe this type of deficit is apparent in the diagnostic criteria for

addiction.

Identifying a functional deficit and associated circuit can allow for better insight

into these behaviors and perhaps targeting of affected individuals with pharmacologic

agents known to enhance prefrontal function. For example, imaging research has long

identified frontostriatal, and particularly orbitofrontal, circuits as abnormally active (or

inactive) in psychostimulant users (Goldstein and Volkow, 2011). Abnormal activity

patterns have been associated with reactivity to drug-associated cues, craving, and also

deficits in reward perception. One particularly interesting result is that BOLD signal in

psychostimulant users fails to reflect monetary reward gradients normally (Goldstein et

al., 2007b). That is, while BOLD signal increases generally to a monetary reward in

cocaine users, it does not scale with the amount or magnitude. This result has been

replicated several times and is associated with altered P300 and striatal signaling,

chronicity of drug use, impairments in self-control and even lack of insight (Parvaz et al.,

2012, Moeller et al., 2012a, Konova et al., 2012, Goldstein et al., 2007a). The latter

speculation is particularly intriguing in light of the current results, since insight arguably

depends upon the same process of mental simulation that characterizes model-based

processing. Accordingly, we would suggest that the failure of BOLD signal in frontal

areas to scale with reward likely reflects dysfunctional processing and failure of operation

84

of these model-based systems. This is because the primary difference between a $10 bill

and a $100 bill lies not in its sensory qualities, but rather in the quantity and quality of

goods that one can obtain with each. Fully appreciating this difference requires inference

and mental simulation. Lacking this ability, patients with orbitofrontal BOLD signal fail

to identify any difference between these two green pieces of paper – both are similarly

significant.

If this is correct, then these imaging findings provide a marker for altered

prefrontal function in specific individuals. Such a diagnostic tool might then be combined

with pharmacologic agents known to enhance prefrontal dependent processing. This

effort has been the focus of recent endeavors with methylphenidate (Moeller et al.,

2012b, Goldstein et al., 2010), but additional candidates include nicotinic and alpha-2

adrenergic receptor agents. These have recently been shown to have highly specific

effects on the maintenance of information in working or scratchpad memory in prefrontal

cortex (Wang et al., 2011, Castner et al., 2011). Similarly, serotonergic tone has been

shown to be necessary for reversal learning and devaluation (Clarke et al., 2004,

Schilman et al., 2010, Nonkes et al., 2010). Our results suggest that approaches that

combine these agents with typical psychotherapeutic interventions that appropriately

target individuals who show abnormal processing in orbital or other prefrontal areas

might prove especially beneficial.

85

IV. Conclusions

We make decisions each and every day as a natural consequence of interacting

with our environment. Some of these decisions are easy. Others are more complex or

difficult, particularly those that require us to evaluate actions in novel situations. How,

then, do we make these decisions given these unfamiliar circumstances? And if we make

a "wrong" decision, that is to say where the outcome is adverse or runs contrary to our

predicted outcome, how do we learn to adjust our behavior the next time to produce a

more favorable result? Or more colloquially, how do we learn from our mistakes? The

orbitofrontal cortex (OFC) is critical for making these more complex, in-the-moment

decisions, and for learning to update our expectations when experiences differ from our

assumptions. Making decisions in these unfamiliar situations requires us to use a model-

based representation of our situation and for us to have an idea of the predicted outcome

of our decision. Understanding that the OFC is important for these decisions led me to

wonder: 1) How are inferred values and their predicted outcomes signaled in the OFC?

2) Is this ability to make decisions in more complex situations and the ability to learn

from our mistakes completely seized when people abuse psychostimulant drugs,

specifically cocaine?

The sensory preconditioning task recently used in the Schoenbaum lab

demonstrated further evidence that the OFC is critical for the ability to make decisions in-

the-moment in novel situations or environments based on predicted outcomes. But the

research also suggests that the OFC is not essential for habit-like decisions based off of

repeated and previous experiences. Additionally, this study demonstrated that the OFC

was important for learning when expectations about predicted outcomes were not met.

86

This ability to recognize and compare predictions and outcomes is critical for learning.

This study laid the foundation for my dissertation, as I set out to understand the

neurophysiological mechanism of how the OFC might signal this inferred value. Previous

reports have demonstrated that cocaine disrupts the OFC. In addition, the prevalence of

long term health issues, failed family relationships and repeated incarcerations provide

evidence that addicts face challenges in making good decisions based on long term

outcomes. I hypothesized that rats exposed to cocaine would have difficulties similar to

those seen when the OFC was inactivated. Ultimately, my research aimed to provide

evidence that cocaine affects the ability to infer value, and thus cocaine-exposed rats

subsequently fail to learn when expected outcomes are not met.

In my first study, I hypothesized that the OFC is signaling the inferred value in

the sensory preconditioning task. During the sensory preconditioning phase, the mean

firing rate of the stimulus-stimulus cues were strongly correlated. The strength of this

original relationship decreased throughout conditioning as the cues were paired with

either a reward or no reward. However, there was still a significant correlation at the time

of testing, strongly suggesting that the conditioned cues firing rate predicted the

preconditioned cues firing rate. This significant correlation was maintained and was

enough to make an inference on the probe test when the preconditioned auditory cues

were played. These results are intriguing, and although ultimately incomplete, suggest

that with more neural data we may have a clearer understanding of what the OFC is

signaling. These data were not able to distinguish whether or not the inferred value cue

was predicted by the ability of the OFC to signal this inferred value cue, or if the OFC

may be signaling the meaning or value of the conditioned cue. As more sessions are

87

recorded, with better behavioral outcomes showing stronger inferred value in their

behavior, it will be interesting to explore in greater detail what the OFC neurons are

specifically encoding. Recent studies indicate that the ventral striatum plays a wider role

in learning than can be explained by the signaling of cached values, one that is more

consistent with integration of outcome and value information (McDannald et al., 2011).

Important future studies include recording in downstream projections of the OFC in this

corticolimbic circuit, such as the ventral striatum and ventral tegmental area. It would be

interesting to observe if this integration continues downstream to the midbrain dopamine

neurons as well.

While my findings do not fully articulate the specifics of what the neurons in the

OFC are encoding during the sensory preconditioning task, the results from my study on

the effects of cocaine on the OFC are much clearer. The long-term effects of cocaine

exposure paint a depressing picture as results from the rats who self-administrated

cocaine demonstrated disrupted behavior and learning that depends on inferred value but

not on cached value. These data substantially strengthen the viewpoint that cocaine

disrupts the neural circuits that mediate model-based processing. In the sensory

preconditioning task, 14 days of self-administration of cocaine and 5-6 weeks of

withdrawal led to the inability of rats to use the relationship between the cue and the

subsequently conditioned cue to infer value in order to guide behavior. They were also

unable to mobilize this information in order to modulate learning during a blocking task.

These results, though not altogether unexpected, are extremely discouraging in

terms of some of the common treatments that are emphasized for patients with abuse. In

many therapy situations, patients are asked to discuss their treatments and think about the

88

consequences of their actions, which requires insight to have an effect. But these results

suggest that this type of therapy would not be beneficial for patients as they may be

unable to have this insight about their drug abuse. The potential that addicts may be

unable to learn from the detrimental consequences of their behavior, and that their

behaviors are more stimulus-based due to the inability to compare the predicted outcomes

with reality, make it more difficult for them to stop their behavior of using cocaine. In

order to help addicts, we must think of other therapies. In particular, it would be of great

interest to see if we could reverse these deficits somehow using techniques and strategies

with the potential to enhance prefrontal function. These might include pharmacological

agents or repetitive transcranial magnetic stimulation (rTMS). In addition to experiments

to better understand what the OFC is signaling, it could be compelling to record the

neural signals in the cocaine-exposed animals to see if there are differences compared to

normal controls. Furthermore, it may be of interest to examine if cocaine-exposed rats

show the same strong correlation between stimulus-stimulus signaling during

preconditioning, or if this relationship of the conditioned cue predicting the firing of the

preconditioned cue is completely destroyed by the time of probe test day.

Since Phineas Gage survived the railroad accident back in 1848, which led to an

observed radical transformation of his character, the orbitofrontal cortex (OFC) and its

role in behavior and decision making has been of great interest to the field of

neuroscience. It is through patients like Gage, as well as behavioral and

neurophysiological experiments, that we have a greater understanding of the OFC's role

in behavior, decision making and learning. My research aspires to be a first step in

looking at what the OFC signals during the sensory preconditioning task, and it will be

89

interesting to see how everything unfolds and how, and if, the neurons signal this inferred

value. Additionally, it will be important to see if these signals are demonstrated in

downstream targets of the OFC, such as the ventral striatum and ventral tegmental areas,

including dopamine neurons. The results of the cocaine study further demonstrate that

long term effects of the drug not only affect the ability to predict outcomes, but also

affect further learning. This research demonstrates a great need to figure out therapeutic

targets to reverse these damaging effects of cocaine on the OFC so that addicts have the

potential to overcome these deficits in order to stop abusing drugs and realize the long

term consequences of their actions.

ix

References

ADINOFF, B., DEVOUS, M. D., SR., BEST, S. M., GEORGE, M. S., ALEXANDER,

D. & PAYNE, K. 2001. Limbic responsiveness to procaine in cocaine-addicted

subjects. Am J Psychiatry, 158, 390-8.

APA 2000. Diagnostic and Statistical Manual of Mental Disorders (4th ed., text revision).

Washington DC.

BAXTER, M. G. & MURRAY, E. A. 2002. The amygdala and reward. Nature Reviews

Neuroscience, 3, 563-573.

BECHARA, A., DAMASIO, H., TRANEL, D. & DAMASIO, A. R. 1997. Deciding

advantageously before knowing the advantageous strategy. Science, 275, 1293-

1294.

BELIN, D. & EVERITT, B. J. 2008. Cocaine seeking habits depend upon dopamine-

dependent serial connectivity linking the ventral with the dorsal striatum. Neuron,

57, 432-441.

BISSONETTE, G. B., MARTINS, G. J., FRANZ, T. M., HARPER, E. S.,

SCHOENBAUM, G. & POWELL, E. M. 2008. Double dissociation of the effects

of medial and orbital prefrontal cortical lesions on attentional and affective shifts

in mice. Journal of Neuroscience, 28, 11124-11130.

BROWN, S. & SCHAFER, E. A. 1888. An investigation into the functions of the

occipital and temporal lobes of the monkey's brain. Philosophical Transactions of

the Royal Society of London B, 179, 303-327.

BURKE, K. A., FRANZ, T. M., MILLER, D. N. & SCHOENBAUM, G. 2008. The role

of orbitofrontal cortex in the pursuit of happiness and more specific rewards.

Nature, 454, 340-344.

BUTTER, C. M. 1969. Perseveration and extinction in discrimination reversal tasks

following selective frontal ablations in Macaca mulatta. Physiology and Behavior,

4, 163-171.

CALU, D. J., STALNAKER, T. A., FRANZ, T. M., SINGH, T., SHAHAM, Y. &

SCHOENBAUM, G. 2007. Withdrawal from cocaine self-administration

produces long-lasting deficits in orbitofrontal-dependent reversal learning in rats.

Learn Mem, 14, 325-8.

CARMICHAEL, S. T. & PRICE, J. L. 1994. Architectonic subdivision of the orbital and

medial prefrontal cortex in the macaque monkey. J Comp Neurol, 346, 366-402.

x

CARMICHAEL, S. T. & PRICE, J. L. 1995. Limbic connections of the orbital and

medial prefrontal cortex in macaque monkeys. Journal of Comparative

Neurology, 363, 615-641.

CASTNER, S. A., SMAGIN, G. N., PISER, T. M., WANG, Y., SMITH, J. S.,

CHRISTIAN, E. P., MRZLJAK, L. & WILLIAMS, G. V. 2011. Immediate and

sustained improvements in working memory after selective stimulation of alpha7

nicotinic acetylcholine receptors. Biol Psychiatry, 69, 12-18.

CHUDASAMA, Y. & ROBBINS, T. W. 2003. Dissociable contributions of the

orbitofrontal and infralimbic cortex to pavlovian autoshaping and discrimination

reversal learning: further evidence for the functional heterogeneity of the rodent

frontal cortex. Journal of Neuroscience, 23, 8771-8780.

CLARKE, H. F., DALLEY, J. W., CROFTS, H. S., ROBBINS, T. W. & ROBERTS, A.

C. 2004. Cognitive inflexibility after prefrontal serotonin depletion. Science, 304,

878-880.

CORBIT, L. H. & BALLEINE, B. W. 2003. The role of prelimbic cortex in instrumental

conditioning. Behavioral Brain Research, 146, 145-157.

CROMBAG, H. S., GORNY, G., LI, Y., KOLB, B. & ROBINSON, T. E. 2005. Opposite

effects of amphetamine self-administration experience on dendritic spines in the

medial and orbital prefrontal cortex. Cereb Cortex, 15, 341-8.

DAMASIO, A. R. 1994. Decartes Error, New York, Putnam.

DAVIS, M. 2000. The role of the amygdala in conditioned and unconditioned fear and

anxiety. In: AGGLETON, J. P. (ed.) The Amygdala: A Functional Analysis.

Oxford: Oxford University Press.

DAW, N. D., NIV, Y. & DAYAN, P. 2005. Uncertainty-based competition between

prefrontal and dorsolateral striatal systems for behavioral control. Nature

Neuroscience, 8, 1704-1711.

DELAMATER, A. R. 2007. The role of the orbitofrontal cortex in sensory-specific

encoding of associations in pavlovian and instrumental conditioning. Annals of

the New York Academy of Science, 1121, 152-173.

DEROCHE-GAMONET, V., BELIN, D. & PIAZZA, P. V. 2004. Evidence for addiction-

like behavior in the rat. Science, 305, 951-953.

DUTRA, L., STATHOPOULOU, G., BASDEN, S. L., LEYRO, T. M., POWERS, M. B.

& OTTO, M. W. 2008. A meta-analytic review of psychosocial interventions for

substance use disorders. Am J Psychiatry, 165, 179-187.

xi

EAGLE, D. M., BAUNEZ, C., HUTCHESON, D. M., LEHMANN, O., SHAH, A. P. &

ROBBINS, T. W. 2008. Stop-signal reaction-time task performance: role of

prefrontal cortex and subthalamic nucleus. Cerebral Cortex, 18, 178-188.

ERSCHE, K. D., BARNES, A., JONES, P. S., MOREIN-ZAMIR, S., ROBBINS, T. W.

& BULLMORE, E. T. 2011. Abnormal structure of frontostriatal brain systems is

associated with aspects of impulsivity and compulsivity in cocaine dependence.

Brain, 134, 2013-24.

ERSCHE, K. D., ROISER, J. P., ROBBINS, T. W. & SAHAKIAN, B. J. 2008. Chronic

cocaine but not chronic amphetamine use is associated with perseverative

responding in humans. Psychopharmacology (Berl), 197, 421-31.

EVERITT, B. J. & ROBBINS, T. W. 1992. Amygdala-ventral striatal interactions and

reward-related processes. In: AGGLETON, J. P. (ed.) The Amygdala:

Neurological Aspects of Emotion, Memory, and Mental Dysfunction. Oxford:

John Wiley and Sons.

FELLOWS, L. K. & FARAH, M. J. 2003. Ventromedial frontal cortex mediates affective

shifting in humans: evidence from a reversal learning paradigm. Brain, 126, 1830-

1837.

FERRIER, D. 1876. The Functions of the Brain, New York, GP Putnam's Sons.

FILLMORE, M. T. & RUSH, C. R. 2006. Polydrug abusers display impaired

discrimination-reversal learning in a model of behavioural control. J

Psychopharmacol, 20, 24-32.

FRANKLIN, T. R., ACTON, P. D., MALDJIAN, J. A., GRAY, J. D., CROFT, J. R.,

DACKIS, C. A., O'BRIEN, C. P. & CHILDRESS, A. R. 2002. Decreased gray

matter concentration in the insular, orbitofrontal, cingulate, and temporal cortices

of cocaine patients. Biol Psychiatry, 51, 134-42.

GALLAGHER, M. 2000. The amygdala and associative learning. In: AGGLETON, J. P.

(ed.) The Amygdala: A Functional Analysis. Oxford: Oxford University Press.

GALLAGHER, M., MCMAHAN, R. W. & SCHOENBAUM, G. 1999. Orbitofrontal

cortex and representation of incentive value in associative learning. Journal of

Neuroscience, 19, 6610-6614.

GAWIN, F. H. 1991. Cocaine addiction: psychology and neurophysiology. Science, 251,

1580-6.

GOLDSTEIN, R. Z., ALIA-KLEIN, N., TOMASI, D., ZHANG, L., COTTONE, L. A.,

MALONEY, T., TELANG, F., CAPARELLI, E. C., CHANG, L., ERNST, T.,

SAMARAS, D., SQUIRES, N. K. & VOLKOW, N. D. 2007a. Is decreased

xii

prefrontal cortical sensitivity to monetary reward associated with impaired

motivation and self-control in cocaine addiction? American Journal of Psychiatry,

164, 43-51.

GOLDSTEIN, R. Z., TOMASI, D., ALIA-KLEIN, N., COTTONE, L. A., ZHANG, L.,

TELANG, F. & VOLKOW, N. D. 2007b. Subjective sensitivity to monetary

gradients is associated with frontolimbic activation to reward in cocaine abusers.

Drug and Alcohol Dependence, 87, 233-240.

GOLDSTEIN, R. Z. & VOLKOW, N. D. 2011. Dysfunction of the prefrontal cortex in

addiction: neuroimaging findings and clinical implications. Nature Reviews

Neuroscience, 12, 652-669.

GOLDSTEIN, R. Z., WOICIK, P. A., MALONEY, T., TOMASI, D., ALIA-KLEIN, N.,

SHAN, J., HONORIO, J., SAMARAS, D., WANG, R., TELANG, F., WANG, G.

J. & VOLKOW, N. D. 2010. Oral methylphenidate normalizes cingulate activity

in cocaine addiction during a salient cognitive task. Proc Natl Acad Sci U S A,

107, 16667-72.

GOTTFRIED, J. A., O'DOHERTY, J. & DOLAN, R. J. 2003. Encoding predictive

reward value in human amygdala and orbitofrontal cortex. Science, 301, 1104-

1107.

GROENEWEGEN, H. J. 1988. Organization of the afferent connections of the

mediodorsal thalamic nucleus in the rat, related to the mediodorsal-prefrontal

topography. Neuroscience, 24, 379-431.

GROENEWEGEN, H. J., VERMEULEN-VAN DER ZEE, E., TE KORTSCHOT, A. &

WITTER, M. P. 1987. Organization of the projections from the subiculum to the

ventral striatum in the rat. A study using anterograde transport of Phaseolus

vulgaris leucoagglutinin. Neuroscience, 23, 103-120.

HABER, S. N. & KNUTSON, B. 2010. The reward circuit: linking primate anatomy and

human imaging. Neuropsychopharmacology, 35, 4-26.

HABER, S. N., KUNISHIO, K., MIZOBUCHI, M. & LYND-BALTA, E. 1995. The

orbital and medial prefrontal circuit through the primate basal ganglia. Journal of

Neuroscience, 15, 4851-4867.

HARLOW, J. M. 1868. Recovery after passage of an iron bar through the head.

Publications of the Massachusetts Medical Society, 2, 329-346.

HOLLAND, P. C. 1984. Unblocking in Pavlovian appetitive conditioning. J Exp Psychol

Anim Behav Process, 10, 476-97.

xiii

HOLLAND, P. C. & GALLAGHER, M. 1999. Amygdala circuitry in attentional and

representational processes. Trends in Cognitive Sciences, 3, 65-73.

HOLLAND, P. C. & GALLAGHER, M. 2004. Amygdala-frontal interactions and reward

expectancy. Current Opinion in Neurobiology, 14, 148-155.

HOLLAND, P. C. & RESCORLA, R. A. 1975. The effects of two ways of devaluing the

unconditioned stimulus after first and second-order appetitive conditioning.

Journal of Experimental Psychology: Animal Behavior Processes, 1, 355-363.

HOLLAND, P. C. & STRAUB, J. J. 1979. Differential effects of two ways of devaluing

the unconditioned stimulus after Pavlovian appetitive conditioning. Journal of

Experimental Psychology: Animal Behavior Processes, 5, 65-78.

HOMAYOUN, H. & MOGHADDAM, B. 2006. Progression of cellular adaptations in

medial prefrontal and orbitofrontal cortex in response to repeated amphetamine.

Journal of Neuroscience, 26, 8025-8039.

HORNAK, J., O'DOHERTY, J., BRAMHAM, J., ROLLS, E. T., MORRIS, R. G.,

BULLOCK, P. R. & POLKEY, C. E. 2004. Reward-related reversal learning after

surgical excisions in orbito-frontal or dorsolateral prefrontal cortex in humans.

Journal of Cognitive Neuroscience, 16, 463-478.

IZQUIERDO, A. & JENTSCH, J. D. 2012. Reversal learning as a measure of impulsive

and compulsive behavior in addictions. Psychopharmacology (Berl), 219, 607-20.

IZQUIERDO, A. & MURRAY, E. A. 2010. Functional interaction of medial mediodorsal

thalamic nucleus but not nucleus accumbens with amygdala and orbital prefrontal

cortex is essential for adaptive response selection after reinforcer devaluation.

Journal of Neuroscience, 30, 661-669.

IZQUIERDO, A. D. & MURRAY, E. A. 2004. Combined unilateral lesions of the

amygdala and orbital prefrontal cortex impair affective processing in rhesus

monkeys. Journal of Neurophysiology, 91, 2033-2039.

IZQUIERDO, A. D., SUDA, R. K. & MURRAY, E. A. 2004. Bilateral orbital prefrontal

cortex lesions in rhesus monkeys disrupt choices guided by both reward value and

reward contingency. Journal of Neuroscience, 24, 7540-7548.

JENTSCH, J. D., OLAUSSON, P., DE LA GARZA, R., 2ND & TAYLOR, J. R. 2002.

Impairments of reversal learning and response perseveration after repeated,

intermittent cocaine administrations to monkeys. Neuropsychopharmacology, 26,

183-90.

xiv

JENTSCH, J. D. & TAYLOR, J. R. 1999. Impulsivity resulting from frontostriatal

dysfunction in drug abuse: implications for the control of behavior by reward-

related stimuli. Psychopharmacology, 146, 373-390.

JONES, B. & MISHKIN, M. 1972. Limbic lesions and the problem of stimulus-

reinforcement associations. Experimental Neurology, 36, 362-377.

JONES, J. L., ESBER, G. R., MCDANNALD, M. A., GRUBER, A. J., HERNANDEZ,

A., MIRENZI, A. & SCHOENBAUM, G. 2012. Orbitofrontal cortex supports

behavior and learning using inferred but not cached values. Science, 338, 953-6.

KAMIN, L. J. 1969. Predictability, suprise, attention, and conditioning. In: CAMPBELL,

B. A. & CHURCH, R. M. (eds.) Punishment and Aversive Behavior. New York:

Appleton-Century-Crofts.

KENNERLEY, S. W., BEHRENS, T. E. & WALLIS, J. D. 2011. Double dissociation of

value computations in orbitofrontal and anterior cingulate neurons. Nature

Neuroscience, AOP.

KILLCROSS, S. & COUTUREAU, E. 2003. Coordination of actions and habits in the

medial prefrontal cortex of rats. Cerebral Cortex, 13, 400-408.

KLUVER, H. & BUCY, P. C. 1939. Preliminary analysis of the temporal lobes in

monkeys. Archives of Neurology and Psychiatry, 42, 979-1000.

KOLB, B., PELLIS, S. & ROBINSON, T. E. 2004. Plasticity and functions of the orbital

frontal cortex. Brain Cogn, 55, 104-15.

KONOVA, A. B., MOELLER, S. J., TOMASI, D., PARVAZ, M. A., ALIA-KLEIN, N.,

VOLKOW, N. D. & GOLDSTEIN, R. Z. 2012. Structural and behavioral

correlates of abnormal encoding of money value in the sensorimotor striatum in

cocaine addiction. European Journal of Neuroscience, 36, 2979-2988.

KOOB, G. F. & LE MOAL, M. 2008. Review. Neurobiological mechanisms for

opponent motivational processes in addiction. Philosophical Transactions of the

Royal Society of London B, 363, 3113-3123.

KRETTEK, J. E. & PRICE, J. L. 1977. The cortical projections of the mediodorsal

nucleus and adjacent thalamic nuclei in the rat. Journal of Comparative

Neurology, 171, 157-192.

LEDOUX, J. E. 1996. The emotional brain, New York, Simon and Schuster.

LEONARD, C. M. 1969. The prefrontal cortex of the rat. I. Cortical projections of the

mediodorsal nucleus. II. Efferent connections. Brain Research, 12, 321-343.

xv

LEVY, D. J. & GLIMCHER, P. W. 2011. Comparing apples and oranges: Using reward-

specific and reward-general subjective value representation in the brain. Journal

of Neuroscience, 31, 14693-14707.

LEVY, D. J. & GLIMCHER, P. W. 2012. The root of all value: a neural common

currency for choice. Current Opinion in Neurobiology, 22, 1027-1038.

LONDON, E. D., CASCELLA, N. G., WONG, D. F., PHILLIPS, R. L., DANNALS, R.

F., LINKS, J. M., HERNING, R., GRAYSON, R., JAFFE, J. H. & WAGNER, H.

N., JR. 1990. Cocaine-induced reduction of glucose utilization in human brain. A

study using positron emission tomography and [fluorine 18]-fluorodeoxyglucose.

Arch Gen Psychiatry, 47, 567-74.

LUCANTONIO, F., STALNAKER, T. A., SHAHAM, Y., NIV, Y. & SCHOENBAUM,

G. 2012. The impact of orbitofrontal dysfunction on cocaine addiction. Nat

Neurosci, 15, 358-66.

LUCANTONIO, F., TAKAHASHI, Y., HOFFMAN, A., CHANG, C. Y.,

CHAUDHARY, S., SHAHAM, Y., LUPICA, C. R. & SCHOENBAUM, G. 2014.

Orbitofrontal activation restores insight lost after cocaine use. in review.

LUK, C.-H. & WALLIS, J. D. 2013. Choice coding in frontal cortex during stimulus-

guided or action-guided decision-making. Journal of Neuroscience, 33, 1864-

1871.

LYOO, I. K., STREETER, C. C., AHN, K. H., LEE, H. K., POLLACK, M. H.,

SILVERI, M. M., NASSAR, L., LEVIN, J. M., SARID-SEGAL, O., CIRAULO,

D. A., RENSHAW, P. F. & KAUFMAN, M. J. 2004. White matter

hyperintensities in subjects with cocaine and opiate dependence and healthy

comparison subjects. Psychiatry Res, 131, 135-45.

MACHADO, C. J. & BACHEVALIER, J. 2007. The effects of selective amygdala,

orbital frontal cortex or hippocampal formation lesions on reward assessment in

nonhuman primates. European Journal of Neuroscience, 25, 2885-2904.

MAN, M. S., CLARKE, H. F. & ROBERTS, A. C. 2009. The role of the orbitofrontal

cortex and medial striatum in the regulation of prepotent responses to food

rewards. Cerebral Cortex, 19, 899-906.

MCALONAN, K. & BROWN, V. J. 2003. Orbital prefrontal cortex mediates reversal

learning and not attentional set shifting in the rat. Behavioral Brain Research,

146, 97-130.

MCDANNALD, M. A., LUCANTONIO, F., BURKE, K. A., NIV, Y. &

SCHOENBAUM, G. 2011. Ventral striatum and not orbitofrontal cortex are both

xvi

required for model-based, but not model-free, reinforcement learning. Journal of

Neuroscience, in press.

MCDANNALD, M. A., SADDORIS, M. P., GALLAGHER, M. & HOLLAND, P. C.

2005. Lesions of orbitofrontal cortex impair rats' differential outcome expectancy

learning but not conditioned stimulus-potentiated feeding. Journal of

Neuroscience, 25, 4626-4632.

MCDONALD, A. J. 1991. Organization of the amygdaloid projections to the prefrontal

cortex and associated striatum in the rat. Neuroscience, 44, 1-14.

MENDELSON, J. H. & MELLO, N. K. 1996. Management of cocaine abuse and

dependence. N Engl J Med, 334, 965-72.

MENDEZ, I. A., SIMON, N. W., HART, N., MITCHELL, M. R., NATION, J. R.,

WELLMAN, P. J. & SETLOW, B. 2010. Self-administered cocaine causes long-

lasting increases in impulsive choice in a delay discounting task. Behavioral

Neuroscience, 124, 470-477.

MEUNIER, M., BACHEVALIER, J. & MISHKIN, M. 1997. Effects of orbital frontal

and anterior cingulate lesions on object and spatial memory in rhesus monkeys.

Neuropsychologia, 35, 999-1015.

MILAD, M. R. & QUIRK, G. J. 2002. Neurons in medial prefrontal cortex signal

memory for fear extinction. Nature, 420, 70-74.

MOELLER, S. J., HAJCAK, G., PARVAZ, M. A., DUNNING, J. P., VOLKOW, N. D.

& GOLDSTEIN, R. Z. 2012a. Psychophysiological prediction of choice:

relevance to insight and drug addiction. Brain, 135, 3481-3494.

MOELLER, S. J., HONORIO, J., TOMASI, D., PARVAZ, M. A., WOICIK, P. A.,

VOLKOW, N. D. & GOLDSTEIN, R. Z. 2012b. Methylphenidate enhances

executive function and optimizes prefrontal function in both health and cocaine

addiction. Cereb Cortex.

MONTAGUE, P. R. & BERNS, G. S. 2002. Neural economics and the biological

substrates of valuation. Neuron, 36, 265-284.

MURRAY, E. A., O'DOHERTY, J. & SCHOENBAUM, G. 2007. What we know and do

not know about the functions of the orbitofrontal cortex after 20 years of cross-

species studies. Journal of Neuroscience, 27, 8166-8169.

NELSON, A. & KILLCROSS, S. 2006. Amphetamine exposure enhances habit

formation. Journal of Neuroscience, 26, 3805-3812.

xvii

NIV, Y., JOEL, D. & DAYAN, P. 2006. A normative perspective on motivation. Trends

Cogn Sci, 10, 375-81.

NONKES, L. J., TOMSON, K., MAERTIN, A., DEDEREN, J., MAES, J. H. &

HOMBERG, J. 2010. Orbitofrontal cortex and amygdalar over-activity is

associated with an inability to use the value of expected outcomes to guide

behavior in serotonin transporter knockout rats. Neurobiology of Learning and

Memory, 94, 65-72.

O'DOHERTY, J., DEICHMANN, R., CRITCHLEY, H. D. & DOLAN, R. J. 2002.

Neural responses during anticipation of a primary taste reward. Neuron, 33, 815-

826.

O'DOHERTY, J., ROLLS, E. T., FRANCIS, S., BOWTELL, R., MCGLONE, F.,

KOBAL, G., RENNER, B. & AHNE, G. 2000a. Sensory-specific satiety-related

olfactory activation of the human orbitofrontal cortex. Neuroreport, 11, 893-897.

O'DOHERTY, J., ROLLS, E. T., FRANCIS, S., BOWTELL, R., MCGLONE, F.,

KOBAL, G., RENNER, B. & AHNE, G. 2000b. Sensory-specific satiety-related

olfactory activation of the human orbitofrontal cortex. Neuroreport, 11, 893-7.

O'NEILL, J., CARDENAS, V. A. & MEYERHOFF, D. J. 2001a. Effects of abstinence

on the brain: quantitative magnetic resonance imaging and magnetic resonance

spectroscopic imaging in chronic alcohol abuse. Alcohol Clin Exp Res, 25, 1673-

82.

O'NEILL, J., CARDENAS, V. A. & MEYERHOFF, D. J. 2001b. Separate and

interactive effects of cocaine and alcohol dependence on brain structures and

metabolites: quantitative MRI and proton MR spectroscopic imaging. Addict Biol,

6, 347-361.

ONGUR, D., FERRY, A. T. & PRICE, J. L. 2003. Architectonic subdivision of the

human orbital and medial prefrontal cortex. J Comp Neurol, 460, 425-49.

OSTLUND, S. B. & BALLEINE, B. W. 2007. Orbitofrontal cortex mediates outcome

encoding in Pavlovian but not instrumental learning. Journal of Neuroscience, 27,

4819-4825.

PADOA-SCHIOPPA, C. 2011. Neurobiology of economic choice: a goods-based model.

Annual Review of Neuroscience, 34, 333-359.

PADOA-SCHIOPPA, C. & ASSAD, J. A. 2006. Neurons in orbitofrontal cortex encode

economic value. Nature, 441, 223-226.

xviii

PADOA-SCHIOPPA, C. & ASSAD, J. A. 2008. The representation of economic value in

the orbitofrontal cortex is invariant for changes in menu. Nature Neuroscience,

11, 95-102.

PAIS-VIEIRA, M., LIMA, D. & GALHARDO, V. 2007. Orbitofrontal cortex lesions

disrupt risk assessment in a novel serial decision-making task for rats.

Neuroscience, 145, 225-231.

PARVAZ, M. A., MALONEY, T., MOELLER, S. J., WOICIK, P. A., ALIA-KLEIN, N.,

TELANG, F., WANG, G.-J., SQUIRES, N. K., VOLKOW, N. D. &

GOLDSTEIN, R. Z. 2012. Sensitivity to monetary reward is most severely

compromised in recently abstaining cocaine addicted individuals: A cross-

sectional ERP study. Psychiatry Research: Neuroimaging, 203, 75-82.

PICKENS, C. L., SADDORIS, M. P., GALLAGHER, M. & HOLLAND, P. C. 2005.

Orbitofrontal lesions impair use of cue-outcome associations in a devaluation

task. Behavioral Neuroscience, 119, 317-322.

PICKENS, C. L., SETLOW, B., SADDORIS, M. P., GALLAGHER, M., HOLLAND, P.

C. & SCHOENBAUM, G. 2003. Different roles for orbitofrontal cortex and

basolateral amygdala in a reinforcer devaluation task. Journal of Neuroscience,

23, 11078-11084.

PLASSMANN, H., O'DOHERTY, J. & RANGEL, A. 2007. Orbitofrontal cortex encodes

willingness to pay in everyday economic transactions. Journal of Neuroscience,

27, 9984-9988.

QUIRK, G. J., RUSSO, G. K., BARRON, J. L. & LEBRON, K. 2000. The role of

ventromedial prefrontal cortex in the recovery of extinguished fear. Journal of

Neuroscience, 20, 6225-6231.

RAY, J. P. & PRICE, J. L. 1992. The organization of the thalamocortical connections of

the mediodorsal thalamic nucleus in the rat, related to the ventral forebrain -

prefrontal cortex topography. Journal of Comparative Neurology, 323, 167-197.

REEKIE, Y. L., BRAESICKE, K., MAN, M. S. & ROBERTS, A. C. 2008. Uncoupling

of behavioral and autonomic responses after lesions of the primate orbitofrontal

cortex. Proceedings of the National Academy of Sciences, 105, 9787-9792.

RESCORLA, R. A. 1999. Learning about qualitatively different outcomes during a

blocking procedure. Animal Learning and Behavior, 27, 140-151.

RHODES, S. E. V. & KILLCROSS, S. A. 2004. Lesions of rat infralimbic cortex

enhance recovery and reinstatement of an appetitive Pavlovian response. Learning

and Memory, 11, 611-616.

xix

RHODES, S. E. V. & KILLCROSS, S. A. 2007. Lesions of rat infralimbic cortex

enhance renewal of extinguished Pavlovian responding. European Journal of

Neuroscience, 25, 2498-2503.

ROBBINS, T. W. & EVERITT, B. J. 1999. Drug addiction: bad habits add up. Nature,

398, 567-570.

ROBINSON, T. E. & BERRIDGE, K. C. 2000. The psychology and neurobiology of

addiction: an incentive-sensitization view. Addiction, 95, S91-S117.

ROBINSON, T. E. & BERRIDGE, K. C. 2008. Review. The incentive sensitization

theory of addiction: some current issues. Philosophical Transactions of the Royal

Society of London B, 363, 3137-3146.

ROESCH, M. R. & SCHOENBAUM, G. 2006. From associations to expectancies:

orbitofrontal cortex as gateway between the limbic system and representational

memory. In: ZALD, D. H. & RAUSCH, S. L. (eds.) The Orbitofrontal Cortex.

London: Oxford University Press.

ROESCH, M. R., STALNAKER, T. A. & SCHOENBAUM, G. 2007a. Associative

encoding in anterior piriform cortex versus orbitofrontal cortex during odor

discrimination and reversal learning. Cereb Cortex, 17, 643-52.

ROESCH, M. R., TAKAHASHI, Y., GUGSA, N., BISSONETTE, G. B. &

SCHOENBAUM, G. 2007b. Previous cocaine exposure makes rats hypersensitive

to both delay and reward magnitude. Journal of Neuroscience, 27, 245-250.

ROLLS, E. T., HORNAK, J., WADE, D. & MCGRATH, J. 1994. Emotion-related

learning in patients with social and emotional changes associated with frontal lobe

damage. Journal of Neurology, Neurosurgery, and Psychiatry, 57, 1518-1524.

ROSE, J. E. & WOOLSEY, C. N. 1948. The orbitofrontal cortex and its connections with

the mediodorsal nucleus in rabbit, sheep, and cat. Res Pub Ass Nerv Ment Dis, 27,

210-232.

SCARLET, J., DELAMATER, A. R., CAMPESE, V., FEIN, M. & WHEELER, D. S.

2012. Differential involvement of the basolateral amygdala and orbitofrontal

cortex in the formation of sensory-specific associations in conditioned flavor

preference and magazine approach paradigms. Eur J Neurosci, 35, 1799-809.

SCHILMAN, E. A., KLAVIR, O., WINTER, C., SOHR, R. & JOEL, D. 2010. The role

of the striatum in compulsive behavior in intact and orbitofrontal-cortex-lesioned

rats: possible involvement of the serotonergic system.

Neuropsychopharmacology, 35, 1026-1039.

xx

SCHOENBAUM, G., CHIBA, A. A. & GALLAGHER, M. 1998. Orbitofrontal cortex

and basolateral amygdala encode expected outcomes during learning. Nature

Neuroscience, 1, 155-159.

SCHOENBAUM, G., ROESCH, M. R., STALNAKER, T. A. & TAKAHASHI, Y. K.

2009. A new perspective on the role of the orbitofrontal cortex in adaptive

behaviour. Nature Reviews Neuroscience, 10, 885-892.

SCHOENBAUM, G., SADDORIS, M. P., RAMUS, S. J., SHAHAM, Y. & SETLOW, B.

2004. Cocaine-experienced rats exhibit learning deficits in a task sensitive to

orbitofrontal cortex lesions. Eur J Neurosci, 19, 1997-2002.

SCHOENBAUM, G. & SETLOW, B. 2001. Integrating orbitofrontal cortex into

prefrontal theory: common processing themes across species and subdivision.

Learning and Memory, 8, 134-147.

SCHOENBAUM, G. & SETLOW, B. 2005. Cocaine makes actions insensitive to

outcomes but not extinction: implications for altered orbitofrontal-amygdalar

function. Cerebral Cortex, 15, 1162-1169.

SCHOENBAUM, G., SETLOW, B., NUGENT, S. L., SADDORIS, M. P. &

GALLAGHER, M. 2003. Lesions of orbitofrontal cortex and basolateral

amygdala complex disrupt acquisition of odor-guided discriminations and

reversals. Learn Mem, 10, 129-40.

SCHOENBAUM, G. & SHAHAM, Y. 2008. The role of orbitofrontal cortex in drug

addiction: a review of preclinical studies. Biol Psychiatry, 63, 256-62.

SIMON, N. W., MENDEZ, I. A. & SETLOW, B. 2007. Cocaine exposure causes long-

term increases in impulsive choice. Behavioral Neuroscience, 121, 543-549.

STALNAKER, T. A., ROESCH, M. R., FRANZ, T. M., BURKE, K. A. &

SCHOENBAUM, G. 2006. Abnormal associative encoding in orbitofrontal

neurons in cocaine-experienced rats during decision-making. European Journal of

Neuroscience, 24, 2643-2653.

STEINER, A. P. & REDISH, A. D. 2012. The road not taken: neural correlates of

decision making in orbitofrontal cortex. Frontiers in Neuroscience, 6, 131.

SUL, J. H., KIM, H., HUH, N., LEE, D. & JUNG, M. W. 2010. Distinct roles of rodent

orbitofrontal and medial prefrontal cortex in decision making. Neuron, 66, 449-

460.

TAKAHASHI, Y., ROESCH, M. R., STALNAKER, T. A., HANEY, R. Z., CALU, D. J.,

TAYLOR, A. R., BURKE, K. A. & SCHOENBAUM, G. 2009. The orbitofrontal

xxi

cortex and ventral tegmental area are necessary for learning from unexpected

outcomes. Neuron, 62, 269-280.

TAKAHASHI, Y., ROESCH, M. R., STALNAKER, T. A. & SCHOENBAUM, G. 2007.

Cocaine exposure shifts the balance of associative encoding from ventral to

dorsolateral striatum. Frontiers in Integrative Neuroscience, 1:11. doi:

10.3389/neuro.07/011.2007.

TAKAHASHI, Y. K., CHANG, C. Y., LUCANTONIO, F., HANEY, R. Z., BERG, B.

A., YAU, H. J., BONCI, A. & SCHOENBAUM, G. 2013. Neural estimates of

imagined outcomes in the orbitofrontal cortex drive behavior and learning.

Neuron, 80, 507-18.

TAYLOR, J. R. & HORGER, B. A. 1999. Enhanced responding for conditioned reward

produced by intra-accumbens amphetamine is potentiated after cocaine

sensitization. Psychopharmacology, 142, 31-40.

TAYLOR, J. R. & JENTSCH, J. D. 2001. Repeated intermittent administration of

psychomotor stimulant drugs alters the acquisition of Pavlovian approach

behavior in rats: differential effects of cocaine, d-amphetamine and 3,4-

methylenedioxymethamphetamine ("ecstasy"). Biological Psychiatry, 50, 137-

143.

TEITELBAUM, H. 1964. A comparison of effects of orbitofrontal and hippocampal

lesions upon discrimination learning and reversal in the cat. Experimental

Neurology, 9, 452-462.

THORPE, S. J., ROLLS, E. T. & MADDISON, S. 1983. The orbitofrontal cortex:

neuronal activity in the behaving monkey. Experimental Brain Research, 49, 93-

115.

TREMBLAY, L. & SCHULTZ, W. 1999. Relative reward preference in primate

orbitofrontal cortex. Nature, 398, 704-708.

VANDERSCHUREN, L. J. M. J. & EVERITT, B. J. 2004. Drug seeking becomes

compulsive after prolonged cocaine self-administration. Science, 305, 1017-1019.

VOLKOW, N. D., CHANG, L., WANG, G. J., FOWLER, J. S., DING, Y. S., SEDLER,

M., LOGAN, J., FRANCESCHI, D., GATLEY, J., HITZEMANN, R.,

GIFFORD, A., WONG, C. & PAPPAS, N. 2001. Low level of brain dopamine

D2 receptors in methamphetamine abusers: association with metabolism in the

orbitofrontal cortex. Am J Psychiatry, 158, 2015-21.

VOLKOW, N. D., FOWLER, J. S., WANG, G. J., HITZEMANN, R., LOGAN, J.,

SCHLYER, D. J., DEWEY, S. L. & WOLF, A. P. 1993. Decreased dopamine D2

xxii

receptor availability is associated with reduced frontal metabolism in cocaine

abusers. Synapse, 14, 169-77.

VOLKOW, N. D., FOWLER, J. S., WOLF, A. P. & GILLESPI, H. 1990. Metabolic

studies of drugs of abuse. NIDA Res Monogr, 105, 47-53.

VOLKOW, N. D., FOWLER, J. S., WOLF, A. P., HITZEMANN, R., DEWEY, S.,

BENDRIEM, B., ALPERT, R. & HOFF, A. 1991. Changes in brain glucose

metabolism in cocaine dependence and withdrawal. Am J Psychiatry, 148, 621-6.

VOLKOW, N. D., VALENTINE, A. & KULKARNI, M. 1988. Radiological and

neurological changes in the drug abuse patient: a study with MRI. J Neuroradiol,

15, 288-93.

WALLIS, J. D. 2007. Orbitofrontal cortex and its contribution to decision-making.

Annual Review of Neuroscience, 30, 31-56.

WALLIS, J. D. 2012. Cross-species studies of orbitofrontal cortex and value-based

decision-making. Nat Neurosci, 15, 13-9.

WANG, M., GAMO, N. J., YANG, Y., JIN, L. E., WANG, X. J., LAUBACH, M.,

MAZER, J. A., LEE, D. & ARNSTEN, A. F. 2011. Neuronal basis of age-related

working memory decline. Nature, 476, 210-213.

WEISKRANTZ, L. 1956. Behavioral changes associated with ablations of the

amygdaloid complex in monkeys. Journal of Comparative Physiology and

Psychology, 9, 381-391.

WEST, E. A., DESJARDIN, J. T., GALE, K. & MALKOVA, L. 2011. Transient

inactivation of orbitofrontal cortex blocks reinforcer devaluation in macaques. J

Neurosci, 31, 15128-35.

WIED, H. M., JONES, J. L., COOCH, N. K., BERG, B. A. & SCHOENBAUM, G. 2013.

Disruption of model-based behavior and learning by cocaine self-administration

in rats. Psychopharmacology (Berl), 229, 493-501.

WIMMER, G. E. & SHOHAMY, D. 2011. The striatum and beyond: hippocampal

contributions to decision making. In: DELGADO, M., PHELPS, E. A. &

ROBBINS, T. W. (eds.) Attention and Performance XXII. Oxford, UK: Oxford

University Press.

WIMMER, G. E. & SHOHAMY, D. 2012. Preference by association: How memory

mechanisms in the hippocampus bias decisions. Science, 338, 270-273.

WINSTANLEY, C. A., OLAUSSON, P., TAYLOR, J. R. & JENTSCH, J. D. 2010.

Insight into the relationship between impulsivity and substance abuse from studies

xxiii

using animal models. Alcoholism, Clinical and Experimental Research, 34, 1306-

1318.

WYVELL, C. L. & BERRIDGE, K. C. 2001. Incentive sensitization by previous

amphetamine exposure: increased cue-triggered "wanting" for sucrose reward.

Journal of Neuroscience, 21, 7831-7840.

ZAPATA, A., MINNEY, V. L. & SHIPPENBERG, T. S. 2010. Shift from goal-directed

to habitual cocaine seeking after prolonged experience in rats. Journal of

Neuroscience, 30, 15457-15463.