Abstract
There is an ever-growing demand for artificial intelligence. Optical processors, which compute with photons instead of electrons, can fundamentally accelerate the development of artificial intelligence by offering substantially improved computing performance. There has been long-term interest in optically constructing the most widely used artificial-intelligence architecture, that is, artificial neural networks, to achieve brain-inspired information processing at the speed of light. However, owing to restrictions in design flexibility and the accumulation of system errors, existing processor architectures are not reconfigurable and have limited model complexity and experimental performance. Here, we propose the reconfigurable diffractive processing unit, an optoelectronic fused computing architecture based on the diffraction of light, which can support different neural networks and achieve a high model complexity with millions of neurons. Along with the developed adaptive training approach to circumvent system errors, we achieved excellent experimental accuracies for high-speed image and video recognition over benchmark datasets and a computing performance superior to that of cutting-edge electronic computing platforms.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41566-021-00796-w/MediaObjects/41566_2021_796_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41566-021-00796-w/MediaObjects/41566_2021_796_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41566-021-00796-w/MediaObjects/41566_2021_796_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41566-021-00796-w/MediaObjects/41566_2021_796_Fig4_HTML.png)
Similar content being viewed by others
Data availability
The data that support the plots within this paper and other findings of this study are available from the corresponding author upon reasonable request.
Code availability
All relevant code is available from the corresponding author upon reasonable request.
References
Moore, G. E. Cramming more components onto integrated circuits. Proc. IEEE 86, 82–85 (1998).
Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (eds Bartlett, P. et al.) 1097–1105 (Curran Associates, 2013).
Zhang, C. et al. Optimizing FPGA-based accelerator design for deep convolutional neural networks. In Proc. 2015 ACM/SIGDA International Symposium on Field-programmable Gate Arrays 161–170 (Association for Computing Machinery, 2015).
Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).
Graves, A., Mohamed, A.-r. & Hinton, G. Speech recognition with deep recurrent neural networks. In Proc. International Conference on Acoustics, Speech and Signal Processing 6645–6649 (IEEE, 2013).
Merolla, P. A. et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673 (2014).
Lukoševičius, M. & Jaeger, H. Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3, 127–149 (2009).
Waldrop, M. M. The chips are down for Moore’s law. Nature 530, 144–147 (2016).
Miller, D. A. B. Attojoule optoelectronics for low-energy information processing and communications. J. Lightwave Technol. 35, 346–396 (2017).
Prucnal, P. R. & Shastri, B. J. Neuromorphic Photonics (CRC, 2017).
Zhang, Q., Yu, H., Barbiero, M., Wang, B. & Gu, M. Artificial neural networks enabled by nanophotonics. Light Sci. Appl. 8, 42 (2019).
Ferrera, M. et al. On-chip CMOS-compatible all-optical integrator. Nat. Commun. 1, 29 (2010).
Zhu, T. et al. Plasmonic computing of spatial differentiation. Nat. Commun. 8, 15391 (2017).
Estakhri, N. M., Edwards, B. & Engheta, N. Inverse-designed metastructures that solve equations. Science 363, 1333–1338 (2019).
Xu, X.-Y. et al. A scalable photonic computer solving the subset sum problem. Sci. Adv. 6, eaay5853 (2020).
Liu, W. et al. A fully reconfigurable photonic integrated signal processor. Nat. Photon. 10, 190–195 (2016).
Kwon, H., Sounas, D., Cordaro, A., Polman, A. & Alù, A. Nonlocal metasurfaces for optical signal processing. Phys. Rev. Lett. 121, 173004 (2018).
Shainline, J. M., Buckley, S. M., Mirin, R. P. & Nam, S. W. Superconducting optoelectronic circuits for neuromorphic computing. Phys. Rev. Appl. 7, 034013 (2016).
Van der Sande, G., Brunner, D. & Soriano, M. C. Advances in photonic reservoir computing. Nanophotonics 6, 561–576 (2017).
Hamerly, R., Bernstein, L., Sludds, A., Soljačić, M. & Englund, D. Large-scale optical neural networks based on photoelectric multiplication. Phys. Rev. X 9, 021032 (2019).
Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).
Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).
Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–447 (2017).
Hughes, T. W., Minkov, M., Shi, Y. & Fan, S. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018).
Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).
Tao, Y. et al. Fourier-space diffractive deep neural network. Phys. Rev. Lett. 123, 023901 (2019).
Zhou, T. et al. In situ optical backpropagation training of diffractive optical neural networks. Photonics Res. 8, 940–953 (2020).
Dou, H. et al. Residual D2NN: training diffractive deep neural networks via learnable light shortcuts. Opt. Lett. 45, 2688–2691 (2020).
Antonik, P., Marsal, N., Brunner, D. & Rontani, D. Human action recognition with a large-scale brain-inspired photonic computer. Nat. Mach. Intell. 1, 530–537 (2019).
Rafayelyan, M., Dong, J., Tan, Y., Krzakala, F. & Gigan, S. Large-scale optical reservoir computing for spatiotemporal chaotic systems prediction. Phys. Rev. X 10, 041037 (2020).
Hughes, T. W., Williamson, I. A. D., Minkov, M. & Fan, S. Wave physics as an analog recurrent neural network. Sci. Adv. 5, eaay6946 (2019).
Bueno, J. et al. Reinforcement learning in a large-scale photonic recurrent neural network. Optica 5, 756–760 (2018).
Maktoobi, S. et al. Diffractive coupling for photonic networks: how big can we go? IEEE J. Sel. Top. Quantum Electron. 26, 1–8 (2019).
Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 12324 (2018).
Luo, Y. et al. Design of task-specific optical systems using broadband diffractive neural networks. Light: Sci. Appl. 8, 112 (2019).
Zuo, Y. et al. All-optical neural network with nonlinear activation functions. Optica 6, 1132–1137 (2019).
LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).
Xiao, H., Rasul, K., & Vollgraf, R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Preprint at https://arxiv.org/abs/1708.07747 (2017).
Blank, M., Gorelick, L., Shechtman, E., Irani, M. & Basri, R. Actions as space-time shapes. In Proc. Tenth International Conference on Computer Vision 1395–1402 (IEEE, 2005).
Schuldt, C., Laptev, I. & Caputo, B. Recognizing human actions: a local SVM approach. In Proc. 17th International Conference on Pattern Recognition 32–36 (IEEE, 2004).
Jhuang, H., Serre, T., Wolf, L. & Poggio, T. A biologically inspired system for action recognition. In Proc. 11th International Conference on Computer Vision 1–8 (IEEE, 2007).
Shu, N., Tang, Q. & Liu, H. A bio-inspired approach modeling spiking neural networks of visual cortex for human action recognition. In 2014 International Joint Conference on Neural Networks 3450–3457 (IEEE, 2014).
Ji, N. Adaptive optical fluorescence microscopy. Nat. Methods 14, 374–380 (2017).
Wang, J. et al. Terabit free-space data transmission employing orbital angular momentum multiplexing. Nat. Photon. 6, 488–496 (2012).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).
Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. Nature 577, 641–646 (2020).
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Computer Vision – ECCV 2018 (eds Ferrari, V. et al.) 833–851 (Springer, 2018).
Larger, L. et al. High-speed photonic reservoir computing using a time-delay-based architecture: million words per second classification. Phys. Rev. X 7, 011015 (2017).
Acknowledgements
We thank T. Zhu and T. Yan for assistance with the software for the prototype system. This work is supported by the Beijing Municipal Science and Technology Commission (No. Z181100003118014), the National Key Research and Development Program of China (No. 2020AAA0130000), the National Natural Science Foundation of China (No. 62088102 and No. 61860206003) and the Tsinghua University Initiative Scientific Research Program.
Author information
Authors and Affiliations
Contributions
Q.D., X.L. and L.F. initiated and supervised the project. X.L. and T.Z. conceived the research and method. X.L. designed the simulations and experiments. T.Z. conducted the experiments. T.Z. and Y.C. performed the simulations and processed the data. T.Z., J.W., Y.C. and H.X. built the experimental system. X.L., T.Z., J.W., H.W., H.X., J.F., Y.L. and L.F. analysed the results. X.L., T.Z., L.F. and Q.D. prepared the manuscript with input from all authors. All authors discussed the research.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Peer review information Nature Photonics thanks Nathan Youngblood and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Supplementary Figs. 1–10, discussion and Tables 1–4.
Supplementary Video 1
Experimental demonstration by using a camera to capture the handwritten digits as the inputs for the system.
Supplementary Video 2
Experimental results of D2NN on the MNIST database.
Supplementary Video 3
Experimental results of D-NIN-1 on the MNIST database.
Supplementary Video 4
Experimental results of D-RNN on the Weizmann database.
Supplementary Video 5
Experimental results of D-RNN on the KTH database.
Rights and permissions
About this article
Cite this article
Zhou, T., Lin, X., Wu, J. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photonics 15, 367–373 (2021). https://doi.org/10.1038/s41566-021-00796-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41566-021-00796-w
This article is cited by
-
Multichannel meta-imagers for accelerating machine vision
Nature Nanotechnology (2024)
-
Orbital angular momentum-mediated machine learning for high-accuracy mode-feature encoding
Light: Science & Applications (2024)
-
Ultrahigh-fidelity spatial mode quantum gates in high-dimensional space by diffractive deep neural networks
Light: Science & Applications (2024)
-
Photonic neuromorphic architecture for tens-of-task lifelong learning
Light: Science & Applications (2024)
-
Analog spatiotemporal feature extraction for cognitive radio-frequency sensing with integrated photonics
Light: Science & Applications (2024)