Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Synaptic plasticity as Bayesian inference

Nature Neuroscience volume 24, pages 565–571 (2021)Cite this article

Subjects

Abstract

Learning, especially rapid learning, is critical for survival. However, learning is hard; a large number of synaptic weights must be set based on noisy, often ambiguous, sensory information. In such a high-noise regime, keeping track of probability distributions over weights is the optimal strategy. Here we hypothesize that synapses take that strategy; in essence, when they estimate weights, they include error bars. They then use that uncertainty to adjust their learning rates, with more uncertain weights having higher learning rates. We also make a second, independent, hypothesis: synapses communicate their uncertainty by linking it to variability in postsynaptic potential size, with more uncertainty leading to more variability. These two hypotheses cast synaptic plasticity as a problem of Bayesian inference, and thus provide a normative view of learning. They generalize known learning rules, offer an explanation for the large variability in the size of postsynaptic potentials and make falsifiable experimental predictions.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The delta rule is suboptimal.
Fig. 2: Bayesian learning rules track the target weight and estimate uncertainty.
Fig. 3: Bayesian learning rules exhibit lower error than classical ones.
Fig. 4: Recurrent neural network.
Fig. 5: Normalized variability versus presynaptic firing rate as a diagnostic of our theory.

Similar content being viewed by others

Data availability

Data are available for download at: https://github.com/Jegmi/the-bayesian-synapse/releases/tag/v2/.

Code availability

Code is available for download at: https://github.com/Jegmi/the-bayesian-synapse/releases/tag/v2/.

References

  1. Poggio, T. A theory of how the brain might work. Cold Spring Harb. Symp. Quant. Biol. 55, 899–910 (1990).

  2. Knill, D. C. & Richards, W. Perception as Bayesian Inference (Cambridge University Press, 1996).

  3. Pouget, A., Beck, J. M., Ma, W. J. & Latham, P. E. Probabilistic brains: knowns and unknowns. Nat. Neurosci. 16, 1170–1178 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Aitchison, L. Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods. Adv. Neural Inf. Process. Syst. https://proceedings.neurips.cc/paper/2020/file/d33174c464c877fb03e77efdab4ae804-Paper.pdf (2020).

  5. Tripathy, S. J., Burton, S. D., Geramita, M., Gerkin, R. C. & Urban, N. N. Brain-wide analysis of electrophysiological diversity yields novel categorization of mammalian neuron types. J. Neurophysiol. 113, 3474–3489 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  6. Schiess, M., Urbanczik, R. & Senn, W. Somato-dendritic synaptic plasticity and error-backpropagation in active dendrites. PLoS Comput. Biol. 12, e1004638 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Bono, J. & Clopath, C. Modeling somatic and dendritic spike mediated plasticity at the single neuron and network level. Nat. Commun. 8, 706 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  8. Sacramento, J., Ponte Costa, R., Bengio, Y. & Senn, W. Dendritic cortical microcircuits approximate the backpropagation algorithm. Adv. Neural Inf. Process. Syst. 31, 8711 (2018).

    Google Scholar 

  9. Illing, B., Gerstner, W. & Brea, J. Biologically plausible deep learning—but how far can we go with shallow networks? Neural Netw. 118, 90–101 (2019).

    Article  PubMed  Google Scholar 

  10. Akrout, M., Wilson, C., Humphreys, P. C., Lillicrap, T. & Tweed, D. Deep learning without weight transport. Adv. Neural Inf. Process. Syst. 32, 976 (2019).

    Google Scholar 

  11. Ito, M., Sakurai, M. & Tongroach, P. Climbing fibre induced depression of both mossy fibre responsiveness and glutamate sensitivity of cerebellar Purkinje cells. J. Physiol. 324, 113–134 (1982).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Eccles, J., Llinas, R. & Sasaki, K. The excitatory synaptic action of climbing fibres on the purkinje cells of the cerebellum. J. Physiol. 182, 268–296 (1966).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Widrow, B. & Hoff, M. E. Adaptive switching circuits. Technical Report no. 1553-1. https://apps.dtic.mil/dtic/tr/fulltext/u2/241531.pdf (Office of Naval Research, 1960).

  14. Dayan, P. & Abbott, L. F. Theoretical Neuroscience (MIT Press, 2001).

  15. Ko, H. et al. The emergence of functional microcircuits in visual cortex. Nature 496, 96–100 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Thomson, A. M. Presynaptic frequency- and pattern-dependent filtering. J. Comput. Neurosci. 15, 159–202 (2003).

    Article  PubMed  Google Scholar 

  17. Tsodyks, M. V. & Markram, H. The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability. Proc. Natl Acad. Sci. USA 94, 719–723 (1997).

    Article  CAS  PubMed  Google Scholar 

  18. Maffei, A. & Turrigiano, G. G. Multiple modes of network homeostasis in visual cortical layer 2/3. J. Neurosci. 28, 4377–4384 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Hoyer, P. O. & Hyvarinen, A. Interpreting neural response variability as Monte Carlo sampling of the posterior. Adv. Neural Inf. Process. Syst. 15, 293–300 (2002).

    Google Scholar 

  20. Fiser, J., Berkes, P., Orbán, G. & Lengyel, M. Statistically optimal perception and learning: from behavior to neural representations. Trends Cogn. Sci. 14, 119–130 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  21. Berkes, P., Fiser, J., Orbán, G. & Lengyel, M. Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment. Science 331, 83–87 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Orbán, G., Berkes, P., Fiser, J. & Lengyel, M. Neural variability and sampling-based probabilistic representations in the visual cortex. Neuron 92, 530–543 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  23. Haefner, R. M., Berkes, P. & Fiser, J. Perceptual decision-making as probabilistic inference by neural sampling. Neuron 90, 649–660 (2016).

    Article  CAS  PubMed  Google Scholar 

  24. Aitchison, L. & Lengyel, M. The hamiltonian brain: efficient probabilistic inference with excitatory–inhibitory neural circuit dynamics. PLoS Comput. Biol. 12, e1005186 (2016).

  25. Lange, R. D. & Haefner, R. M. Task-induced neural covariability as a signature of approximate bayesian learning and inference. Preprint at bioRxiv https://doi.org/10.1101/081661 (2020).

  26. Ma, W. J., Beck, J. M., Latham, P. E. & Pouget, A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 9, 1432–1438 (2006).

    Article  CAS  PubMed  Google Scholar 

  27. Buntine, W. L. & Weigend, A. S. Bayesian backpropagation. Complex Syst. 5, 603–643 (1991).

    Google Scholar 

  28. MacKay, D. J. A practical bayesian framework for backpropagation networks. Neural Comput. 4, 448–472 (1992).

    Article  Google Scholar 

  29. Blundell, C., Cornebise, J., Kavukcuoglu, K. & Dean, W. Weight uncertainty in neural networks. Proc. Mach. Learn. Res. 37, 1613–1622 (2015).

    Google Scholar 

  30. Kirkpatrick, J. et al. Overcoming catastrophic forgetting in neural networks. Proc. Natl Acad. Sci. USA 106, 10296–10301 (2016).

    Google Scholar 

  31. Dayan, P. & Kakade, S. Explaining away in weight space. Adv. Neural Inf. Process. Syst. 13, 451–457 (2001).

    Google Scholar 

  32. Kappel, D., Habenschuss, S., Legenstein, R. & Maass, W. Network plasticity as bayesian inference. PLoS Comput. Biol. 11, e1004485 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  33. Hiratani, N. & Fukai, T. Redundancy in synaptic connections enables neurons to learn optimally. Proc. Natl Acad. Sci. USA 115, E6871–E6879 (2018).

    Article  CAS  PubMed  Google Scholar 

  34. Drugowitsch, J., Mendonça, A. G., Mainen, Z. F. & Pouget, A. Learning optimal decisions with confidence. Proc. Natl Acad. Sci. USA 116, 24872–24880 (2019).

    Article  CAS  PubMed  Google Scholar 

  35. Pfister, J.-P., Dayan, P. & Lengyel, M. Synapses with short-term plasticity are optimal estimators of presynaptic membrane potentials. Nat. Neurosci. 13, 1271–1275 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Kasai, H., Takahashi, N. & Tokumaru, H. Distinct initial SNARE configurations underlying the diversity of exocytosis. Physiol. Rev. 92, 1915–1964 (2012).

    Article  CAS  PubMed  Google Scholar 

  37. Südhof, T. C. The presynaptic active zone. Neuron 75, 11–25 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  38. Michel, K., Müller, J. A., Oprisoreanu, A.-M. & Schoch, S. The presynaptic active zone: a dynamic scaffold that regulates synaptic efficacy. Exp. Cell Res. 335, 157–164 (2015).

    Article  CAS  PubMed  Google Scholar 

  39. Frey, U. & Morris, R. G. Synaptic tagging and long-term potentiation. Nature 385, 533–536 (1997).

    Article  CAS  Google Scholar 

  40. Redondo, R. L. & Morris, R. G. M. Making memories last: the synaptic tagging and capture hypothesis. Nat. Rev. Neurosci. 12, 17–30 (2011).

    Article  CAS  PubMed  Google Scholar 

  41. Rogerson, T. et al. Synaptic tagging during memory allocation. Nat. Rev. Neurosci. 15, 157–169 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Abraham, W. C. & Bear, M. F. Metaplasticity: the plasticity of synaptic plasticity. Trends Neurosci. 19, 126–130 (1996).

  43. Abraham, W. C. Metaplasticity: tuning synapses and networks for plasticity. Nat. Rev. Neurosci. 9, 387 (2008).

    Article  CAS  PubMed  Google Scholar 

  44. Hulme, S. R., Jones, O. D., Raymond, C. R., Sah, P. & Abraham, W. C. Mechanisms of heterosynaptic metaplasticity. Philos. Trans. R. Soc. Lond. B Biol. Sci. 369, 20130148 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  45. Vogelstein, J. T. et al. Fast nonnegative deconvolution for spike train inference from population calcium imaging. J. Neurophysiol. 104, 3691–3704 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  46. Packer, A. M., Russell, L. E., Dalgleish, H. W. P. & Häusser, M. Simultaneous all-optical manipulation and recording of neural circuit activity with cellular resolution in vivo. Nat. Methods 12, 140–146 (2015).

    Article  CAS  PubMed  Google Scholar 

  47. Loewenstein, Y., Kuras, A. & Rumpel, S. Multiplicative dynamics underlie the emergence of the log-normal distribution of spine sizes in the neocortex in vivo. J. Neurosci. 31, 9481–9488 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Matsuzaki, M., Honkura, N., Ellis-Davies, G. C. & Kasai, H. Structural basis of long-term potentiation in single dendritic spines. Nature 429, 761–766 (2004).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Song, S., Sjöström, P. J., Reigl, M., Nelson, S. & Chklovskii, D. B. Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS Biol. 3, e68 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  50. O’Connor, D. H., Peron, S. P., Huber, D. & Svoboda, K. Neural activity in barrel cortex underlying vibrissa-based object localization in mice. Neuron 67, 1048–1061 (2010).

    Article  PubMed  Google Scholar 

  51. Mizuseki, K. & Buzsáki, G. Preconfigured, skewed distribution of firing rates in the hippocampus and entorhinal cortex. Cell Rep. 4, 1010–1021 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Minka, T. P. A family of algorithms for approximate Bayesian inference. Dissertation, Massachusetts Institute of Technology (2001).

Download references

Acknowledgements

L.A. and P.E.L. were supported by the Gatsby Charitable Foundation. P.E.L. was also supported by the Wellcome Trust (110114/Z/15/Z). J.J. and J.-P.P. were supported by the Swiss National Science Foundation (PP00P3 150637 and 31003A 175644). J.A.M. was supported by University College London (UCL) Graduate Research and UCL Overseas Research Scholarships. A.P. was supported by a grant from the Simons Collaboration for the Global Brain and the Swiss National Foundation (31003A 165831).

Author information

Author notes

  1. These authors contributed equally: Alexandre Pouget, Peter E. Latham.

Authors and Affiliations

  1. Gatsby Computational Neuroscience Unit, University College London, London, UK

    Laurence Aitchison, Jorge Aurelio Menendez, Alexandre Pouget & Peter E. Latham

  2. Department of Computer Science, University of Bristol, Bristol, UK

    Laurence Aitchison

  3. Institute of Neuroinformatics, UZH/ETH Zurich, Zurich, Switzerland

    Jannes Jegminat & Jean-Pascal Pfister

  4. Department of Physiology, University of Bern, Bern, Switzerland

    Jannes Jegminat & Jean-Pascal Pfister

  5. CoMPLEX, University College London, London, UK

    Jorge Aurelio Menendez

  6. Department of Basic Neurosciences, University of Geneva, Geneva, Switzerland

    Alexandre Pouget

Authors
  1. Laurence Aitchison
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jannes Jegminat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jorge Aurelio Menendez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jean-Pascal Pfister
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alexandre Pouget
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Peter E. Latham
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

A.P. and P.E.L. were involved in the initial formulation of the problem. L.A. and P.E.L. conducted the theoretical development. L.A., J.J. and J.A.M. performed the simulations and data analysis. P.E.L., J.-P.P. and A.P. wrote the manuscript.

Corresponding author

Correspondence to Laurence Aitchison.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Neuroscience thanks the anonymous reviewers for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Notes 1–6, Supplementary Figs. 1–7 and Supplementary Tables 1 and 2.

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aitchison, L., Jegminat, J., Menendez, J.A. et al. Synaptic plasticity as Bayesian inference. Nat Neurosci 24, 565–571 (2021). https://doi.org/10.1038/s41593-021-00809-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41593-021-00809-5

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing