Abstract
To adapt behavior to a changing environment, one must monitor outcomes of executed actions and adjust subsequent actions accordingly. Involvement of the medial frontal cortex in performance monitoring has been suggested, but little is known about neural processes that link performance monitoring to performance adjustment. Here, we recorded from neurons in the medial prefrontal cortex of monkeys learning arbitrary action-outcome contingencies. Some cells preferentially responded to positive visual feedback stimuli and others to negative feedback stimuli. The magnitude of responses to positive feedback stimuli decreased over the course of behavioral adaptation, in correlation with decreases in the amount of prediction error of action values. Therefore, these responses in medial prefrontal cells may signal the direction and amount of error in prediction of values of executed actions to specify the adjustment in subsequent action selections.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Woodworth, R.S. Dynamics of Behavior (Holt, New York, 1958).
Daw, N.D. & Doya, K. The computational neurobiology of learning and reward. Curr. Opin. Neurobiol. 16, 199–204 (2006).
Matsumoto, K. & Tanaka, K. The role of the medial prefrontal cortex in achieving goals. Curr. Opin. Neurobiol. 14, 178–185 (2004).
Rushworth, M.F., Walton, M.E., Kennerley, S.W. & Bannerman, D.M. Action sets and decisions in the medial frontal cortex. Trends Cogn. Sci. 8, 410–417 (2004).
Falkenstein, M., Hohnsbein, J., Hoormann, J. & Blanke, L. Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. Electroencephalogr. Clin. Neurophysiol. 78, 447–455 (1991).
Gehring, W.J., Goss, B., Coles, M.G.H., Meyer, D.E. & Donchin, E. A neural system for error detection and compensation. Psychol. Sci. 4, 385–390 (1993).
Miltner, W.H.R., Braun, C.H. & Coles, M.G.H. Event-related brain potentials following incorrect feedback in a time-estimation task: evidence for a “generic” neural system for error detection. J. Cogn. Neurosci. 9, 788–798 (1997).
Carter, C.S. et al. Anterior cingulate cortex, error detection, and the online monitoring of performance. Science 280, 747–749 (1998).
Ullsperger, M. & von Cramon, D.Y. Error monitoring using external feedback: specific roles of the habenular complex, the reward system, and the cingulate motor area revealed by functional magnetic resonance imaging. J. Neurosci. 23, 4308–4314 (2003).
Holroyd, C.B. et al. Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nat. Neurosci. 7, 497–498 (2004).
Mars, R.B. et al. Neural dynamics of error processing in medial frontal cortex. Neuroimage 28, 1007–1013 (2005).
Niki, H. & Watanabe, M. Prefrontal and cingulate unit activity during timing behavior in the monkey. Brain Res. 171, 213–224 (1979).
Ito, S., Stuphorn, V., Brown, J.W. & Schall, J.D. Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science 302, 120–122 (2003).
Bartholow, B.D. et al. Strategic control and medial frontal negativity: beyond errors and response conflict. Psychophysiology 42, 33–42 (2005).
Pailing, P.E. & Segalowitz, S.J. The effects of uncertainty in error monitoring on associated ERPs. Brain Cogn. 56, 215–233 (2004).
Vidal, F., Burle, B., Bonnet, M., Grapperon, J. & Hasbroucq, T. Error negativity on correct trials: a reexamination of available data. Biol. Psychol. 64, 265–282 (2003).
Knutson, B., Westdorp, A., Kaiser, E. & Hommer, D. FMRI visualization of brain activity during a monetary incentive delay task. Neuroimage 12, 20–27 (2000).
Walton, M.E., Devlin, J.T. & Rushworth, M.F. Interactions between decision making and performance monitoring within prefrontal cortex. Nat. Neurosci. 7, 1259–1265 (2004).
Holroyd, C.B. & Coles, M.G. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 109, 679–709 (2002).
Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Frank, M.J., Woroch, B.S. & Curran, T. Error-related negativity predicts reinforcement learning and conflict biases. Neuron 47, 495–501 (2005).
Holroyd, C.B., Nieuwenhuis, S., Yeung, N. & Cohen, J.D. Errors in reward prediction are reflected in the event-related brain potential. Neuroreport 14, 2481–2484 (2003).
Amiez, C., Joseph, J.P. & Procyk, E. Anterior cingulate error-related activity is modulated by predicted reward. Eur. J. Neurosci. 21, 3447–3452 (2005).
Gehring, W.J. & Knight, R.T. Prefrontal-cingulate interactions in action monitoring. Nat. Neurosci. 3, 516–520 (2000).
Watkins, C. & Dayan, P. Q-learning. Mach. Learn. 8, 279–292 (1992).
Sutton, R.S. & Barto, A.G. Reinforcement Learning: An Introduction (MIT Press, Cambridge, Massachusetts, 1998).
Yeung, N. & Sanfey, A.G. Independent coding of reward magnitude and valence in the human brain. J. Neurosci. 24, 6258–6264 (2004).
Donkers, F.C., Nieuwenhuis, S. & van Boxtel, G.J. Mediofrontal negativities in the absence of responding. Brain Res. Cogn. Brain Res. 25, 777–787 (2005).
Lewis, D.A., Foote, S.L., Goldstein, M. & Morrison, J.H. The dopaminergic innervation of monkey prefrontal cortex: a tyrosine hydroxylase immunohistochemical study. Brain Res. 449, 225–243 (1988).
Fiorillo, C.D., Tobler, P.N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).
Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).
Daw, N.D., Kakade, S. & Dayan, P. Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).
Robbins, T.W. Chemistry of the mind: neurochemical modulation of prefrontal cortical function. J. Comp. Neurol. 493, 140–146 (2005).
Ullsperger, M. & von Cramon, D.Y. The role of intact frontostriatal circuits in error processing. J. Cogn. Neurosci. 18, 651–664 (2006).
Fantino, E. Choice and rate of reinforcement. J. Exp. Anal. Behav. 12, 723–730 (1969).
Skinner, B.F. Science and Human Behavior (Macmillan, New York, 1953).
Williams, B.A. Conditioned reinforcement: experimental and theoretical issues. Behav. Anal. 17, 261–285 (1994).
Parkinson, J.A. et al. The role of the primate amygdala in conditioned reinforcement. J. Neurosci. 21, 7770–7780 (2001).
Allport, G.W. Pattern and Growth in Personality (Holt, Rinehart and Winston, New York, 1961).
Maslow, A.H. Motivation and Personality (Harper & Row, New York, 1970).
Schultz, W. Multiple reward signals in the brain. Nat. Rev. Neurosci. 1, 199–207 (2000).
Blatter, K. & Schultz, W. Rewarding properties of visual stimuli. Exp. Brain Res. 168, 541–546 (2006).
Carmichael, S.T. & Price, J.L. Architectonic subdivision of the orbital and medial prefrontal cortex in the macaque monkey. J. Comp. Neurol. 346, 366–402 (1994).
Barbas, H. & Pandya, D.N. Architecture and intrinsic connections of the prefrontal cortex in the rhesus monkey. J. Comp. Neurol. 286, 353–375 (1989).
Walker, A.E. A cytoarchitectural study of the prefrontal area of the macaque monkey. J. Comp. Neurol. 73, 59–86 (1940).
Shima, K. et al. Two movement-related foci in the primate cingulate cortex observed in signal-triggered and self-paced forelimb movements. J. Neurophysiol. 65, 188–202 (1991).
Procyk, E., Tanaka, Y.L. & Joseph, J.P. Anterior cingulate activity during routine and non-routine sequential behaviors in macaques. Nat. Neurosci. 3, 502–508 (2000).
Shidara, M. & Richmond, B.J. Anterior cingulate: single neuronal signals related to degree of reward expectancy. Science 296, 1709–1711 (2002).
Nakamura, K., Roesch, M.R. & Olson, C.R. Neuronal activity in macaque SEF and ACC during performance of tasks involving conflict. J. Neurophysiol. 93, 884–908 (2005).
Acknowledgements
This research was partly supported by the Grant-in-Aid for Scientific Research on Priority Areas (17022047) from the Ministry of Education, Culture, Sports, Science and Technology of Japan. We thank W. Schultz for advice about task design, S. Shimamune and K. Murayama for discussion, R. A. Waggoner for taking MRI images, A. Phillips for developing a program for presenting visual stimuli, J. Helen for improving the English, and M. Tomonaga, H. Nakahara and W. Schultz for comments on an early manuscript.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Supplementary information
Supplementary Fig. 1
Recording areas in two monkeys. (PDF 974 kb)
Supplementary Fig. 2
Learning curves of the monkeys and models, in action-learning blocks. (PDF 1181 kb)
Supplementary Fig. 3
Relationship between neuronal responses and prediction errors in each monkey. (PDF 1092 kb)
Supplementary Fig. 4
Distribution of positive-feedback preferring cells and negative-feedback preferring cells in the medial PFC. (PDF 1176 kb)
Supplementary Fig. 5
Comparison of the medial PFC cells' responses between visual blocks and action-learning blocks. (PDF 1055 kb)
Supplementary Fig. 6
Responses determined by the feedback type (positive/negative) but not by stay/shift of actions. (PDF 1539 kb)
Supplementary Table 1
Values of model parameters with which each model best fit the actual performance of monkeys. (PDF 802 kb)
Rights and permissions
About this article
Cite this article
Matsumoto, M., Matsumoto, K., Abe, H. et al. Medial prefrontal cell activity signaling prediction errors of action values. Nat Neurosci 10, 647–656 (2007). https://doi.org/10.1038/nn1890
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn1890
This article is cited by
-
Distributional reinforcement learning in prefrontal cortex
Nature Neuroscience (2024)
-
Prefrontal signals precede striatal signals for biased credit assignment in motivational learning biases
Nature Communications (2024)
-
Asymmetric coding of reward prediction errors in human insula and dorsomedial prefrontal cortex
Nature Communications (2023)
-
Neurons in human pre-supplementary motor area encode key computations for value-based choice
Nature Human Behaviour (2023)
-
Value dynamics affect choice preparation during decision-making
Nature Neuroscience (2023)