Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Letter
  • Published:

All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition

Nature Nanotechnology volume 17, pages 27–32 (2022)Cite this article

Subjects

Abstract

With the advent of the Internet of Things era, the detection and recognition of moving objects is becoming increasingly important1. The current motion detection and recognition (MDR) technology based on the complementary metal oxide semiconductor (CMOS) image sensors (CIS) platform contains redundant sensing, transmission conversion, processing and memory modules, rendering the existing systems bulky and inefficient in comparison to the human retina. Until now, non-memory capable vision sensors have only been used for static targets, rather than MDR. Here, we present a retina-inspired two-dimensional (2D) heterostructure based retinomorphic hardware device with all-in-one perception, memory and computing capabilities for the detection and recognition of moving trolleys. The proposed 2D retinomorphic device senses an optical stimulus to generate progressively tuneable positive/negative photoresponses and memorizes it, combined with interframe differencing computations, to achieve 100% separation detection of moving trichromatic trolleys without ghosting. The detected motion images are fed into a conductance mapped neural network to achieve fast trolley recognition in as few as four training epochs at 10% noise level, outperforming previous results from similar customized datasets. The prototype demonstration of a 2D retinomorphic device with integrated perceptual memory and computation provides the possibility of building compact, efficient MDR hardware.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Retina-inspired 2D retinomorphic hardware for MDR.
Fig. 2: Photoconductivity properties of 2D retinomorphic devices for MDR.
Fig. 3: Illustration of motion detection based on the 2D retinomorphic hardware.
Fig. 4: 2D retinomorphic hardware implementation for trichromatic trolley MDR.

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding authors on reasonable request. Source data are provided with this paper.

Code availability

The codes used for the simulation are available from the corresponding author on reasonable request.

References

  1. Choo, K. D. et al. Energy-efficient low-noise CMOS image sensor with capacitor array-assisted charge-injection SAR ADC for motion-triggered low-power IoT application. In Proc. IEEE International Solid-State Circuits Conference (ISSCC) 96–98 (IEEE, 2019).

  2. Chen, T. & Lu, S. Object-level motion detection from moving cameras. IEEE Trans. Circuits Syst. Video Technol. 27, 2333–2343 (2016).

    Article  Google Scholar 

  3. Yang, S., Kim, K., Kim, E., Baek, K. & Kim, S. An ultra low power CMOS motion detector. IEEE Trans. Consum. Electron. 55, 2425–2430 (2009).

    Article  Google Scholar 

  4. Zhao, B., Zhang, X., Chen, S., Low, K. & Zhuang, H. A 64×64 CMOS image sensor with on-chip moving object detection and localization. IEEE Trans. Circuits Syst. Video Technol. 22, 581–588 (2012).

    Article  Google Scholar 

  5. Wang, C.-Y. et al. Gate-tunable van der Waals heterostructure for reconfigurable neural network vision sensor. Sci. Adv. 6, eaba6173 (2020).

    Article  Google Scholar 

  6. Gollisch, T. & Meister, M. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron 65, 150–164 (2010).

    Article  CAS  Google Scholar 

  7. Boahen, K. A retinomorphic chip with parallel pathways: encoding INCREASING, ON, DECREASING, and OFF visual signals. AICSP 30, 121–135 (2002).

    Google Scholar 

  8. Trujillo Herrera, C. & Labram, J. G. A perovskite retinomorphic sensor. Appl. Phys. Lett. 117, 233501 (2020).

    Article  CAS  Google Scholar 

  9. Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).

    Article  CAS  Google Scholar 

  10. Choi, C. et al. Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system. Nat. Commun. 11, 5934 (2020).

    Article  CAS  Google Scholar 

  11. Wang, S. et al. A MoS2/PTCDA hybrid heterojunction synapse with efficient photoelectric dual modulation and versatility. Adv. Mater. 31, 1806227 (2019).

    Article  Google Scholar 

  12. Hou, X. et al. A logic-memory transistor with the integration of visible information sensing-memory-processing. Adv. Sci. 7, 2002072 (2020).

    Article  CAS  Google Scholar 

  13. Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).

    Article  CAS  Google Scholar 

  14. Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B. & Delbruck, T. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc. IEEE 102, 1470–1484 (2014).

    Article  Google Scholar 

  15. Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2015).

    Article  CAS  Google Scholar 

  16. Masland, R. H. The neuronal organization of the retina. Neuron 76, 266–280 (2012).

    Article  CAS  Google Scholar 

  17. Dowling, J. & Boycott, B. B. Organization of the primate retina: electron microscopy. Proc. R. Soc. Lond. Ser. B. Biol. Sci. 166, 80–111 (1966).

    CAS  Google Scholar 

  18. Demb, J. B. Cellular mechanisms for direction selectivity in the retina. Neuron 55, 179–186 (2007).

    Article  CAS  Google Scholar 

  19. Liu, C. et al. Two-dimensional materials for next-generation computing technologies. Nat. Nanotechnol. 15, 545–557 (2020).

    Article  CAS  Google Scholar 

  20. Wang, S. et al. Two-dimensional ferroelectric channel transistors integrating ultra-fast memory and neural computing. Nat. Commun. 12, 53 (2021).

    Article  CAS  Google Scholar 

  21. Ölveczky, B. P., Baccus, S. A. & Meister, M. Segregation of object and background motion in the retina. Nature 423, 401–408 (2003).

    Article  Google Scholar 

  22. Uyttendaele, M., Eden, A. & Skeliski, R. Eliminating ghosting and exposure artifacts in image mosaics. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) Vol. 2, II–II (IEEE, 2001).

  23. Yan, B., Liu, C., Liu, X., Chen, Y. & Li, H. Understanding the trade-offs of device, circuit and application in ReRAM-based neuromorphic computing systems. In Proc. IEEE International Electron Devices Meeting (IEDM) 11.4.1–11.4.4 (IEEE, 2017).

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (grant nos. 61925402, 61851402, 61725505, 11734016 and 62090032), Science and Technology Commission of Shanghai Municipality (grant nos. 19JC1416600 and 21JC1406100) and Shanghai Education Development Foundation and Shanghai Municipal Education Commission Shuguang Program (grant no. 18SG01).

Author information

Author notes

  1. These authors contributed equally: Zhenhan Zhang, Shuiyuan Wang, Chunsen Liu.

Authors and Affiliations

  1. State Key Laboratory of ASIC and Systems, School of Microelectronics, Fudan University, Shanghai, China

    Zhenhan Zhang, Shuiyuan Wang, Chunsen Liu & Peng Zhou

  2. State Key Laboratory of Infrared Physics, Shanghai Institute of Technical Physics, Chinese Academy of Sciences, Shanghai, China

    Zhenhan Zhang, Runzhang Xie & Weida Hu

  3. Frontier Institute of Chip and System, Shanghai Frontier Base of Intelligent Optoelectronics and Perception, Institute of Optoelectronics, Fudan University, Shanghai, China

    Chunsen Liu & Peng Zhou

Authors
  1. Zhenhan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuiyuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chunsen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Runzhang Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weida Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Peng Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Z.Z. and S.W. designed and conducted the experiments. P.Z. and W.H. conceived the idea. R.X. supported the characterization of materials. C.L. provided assistance with mechanism analysis and discussion. S.W. and Z.Z. wrote the manuscript and all authors contributed to the revision of the paper.

Corresponding authors

Correspondence to Weida Hu or Peng Zhou.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Nanotechnology thanks Han Zhang and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Sections 1–11 containing Supplementary Figs. 1–11 and corresponding discussions.

Source data

Source Data Fig. 2

Source data of Fig. 2.

Source Data Fig. 3

Source data of Fig. 3.

Source Data Fig. 4

Source data of Fig. 4.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Wang, S., Liu, C. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17, 27–32 (2022). https://doi.org/10.1038/s41565-021-01003-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41565-021-01003-1

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing