Abstract
Diagnostic pathology is the foundation and gold standard for identifying carcinomas. However, high inter-observer variability substantially affects productivity in routine pathology and is especially ubiquitous in diagnostician-deficient medical centres. Despite rapid growth in computer-aided diagnosis (CAD), the application of whole-slide pathology diagnosis remains impractical. Here, we present a novel pathology whole-slide diagnosis method, powered by artificial intelligence, to address the lack of interpretable diagnosis. The proposed method masters the ability to automate the human-like diagnostic reasoning process and translate gigapixels directly to a series of interpretable predictions, providing second opinions and thereby encouraging consensus in clinics. Moreover, using 913 collected examples of whole-slide data representing patients with bladder cancer, we show that our method matches the performance of 17 pathologists in the diagnosis of urothelial carcinoma. We believe that our method provides an innovative and reliable means for making diagnostic suggestions and can be deployed at low cost as next-generation, artificial intelligence-enhanced CAD technology for use in diagnostic pathology.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data that support the findings of this study are available from Figshare: https://figshare.com/projects/nmi-wsi-diagnosis/61973.
Code availability
Source code are available from the Github repository: https://github.com/zizhaozhang/nmi-wsi-diagnosis.
Change history
17 July 2019
An amendment to this paper has been published and can be accessed via a link at the top of the paper.
17 May 2019
An amendment to this paper has been published and can be accessed via a link at the top of the paper
References
Brimo, F., Schultz, L. & Epstein, J. I. The value of mandatory second opinion pathology review of prostate needle biopsy interpretation before radical prostatectomy. J. Urol. 184, 126–130 (2010).
Elmore, J. G. et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA 313, 1122–1132 (2015).
Djuric, U., Zadeh, G., Aldape, K. & Diamandis, P. Precision histology: how deep learning is poised to revitalize histomorphology for personalized cancer care. npj Precis. Oncol. 1, 22 (2017).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Poplin, R. et al. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng. 2, 158–164 (2018).
Bejnordi, B. E. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).
Litjens, G. et al. Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Sci. Rep. 6, 26286 (2016).
Araújo, T. et al. Classification of breast cancer histology images using convolutional neural networks. PloS ONE 12, e0177544 (2017).
Xu, Y. et al. Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features. BMC Bioinformatics 18, 281 (2017).
Yoshida, H. et al. Automated histological classification of whole slide images of colorectal biopsy specimens. Oncotarget 8, 90719 (2017).
Han, Z. et al. Breast cancer multi-classification from histopathological images with structured deep learning model. Sci. Rep. 7, 4172 (2017).
Hou, L. et al. Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2424–2433 (IEEE, 2016).
Holzinger, A., Biemann, C., Pattichis, C. S. & Kell, D. B. What do we need to build explainable AI systems for the medical domain? Preprint at https://arxiv.org/abs/1712.09923 (2017).
Lipton, Z. C. The mythos of model interpretability. Queue. 16, 30 (2018).
Pasin, E., Josephson, D. Y., Mitra, A. P., Cote, R. J. & Stein, J. P. Superficial bladder cancer: an update on etiology, molecular development, classification, and natural history. Rev. Urol. 10, 31–43 (2008).
Zhou, M. & Magi-Galluzzi, C. Genitourinary Pathology (Foundations in Diagnostic Pathology, Saunders, 2015).
Humphrey, P. A., Moch, H., Cubilla, A. L., Ulbright, T. M. & Reuter, V. E. The 2016 WHO classification of tumours of the urinary system and male genital organs—Part B: Prostate and bladder tumours. Eur. Urol. 70, 106–119 (2016).
Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics 311–318 (Association for Computational Linguistics, 2002).
Vedantam, R., Lawrence Zitnick, C. & Parikh, D. CIDEr: Consensus-based Image Description Evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 4566–4575 (IEEE, 2015).
Karpathy, A. & Fei-Fei, L. Deep visual–semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3128–3137 (IEEE, 2015).
Maaten, Lvd & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).
Miyamoto, H. et al. Non-invasive papillary urothelial neoplasms: the 2004 WHO/ISUP classification system. Pathol. Int. 60, 1–8 (2010).
Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention 234–241 (Springer, 2015).
Xu, K. et al. Show, attend and tell: neural image caption generation with visual attention. In International Conference on Machine Learning, 2048–2057 (JMLR, 2015).
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2818–2826 (IEEE, 2016).
Deng, J. et al. Imagenet: a large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 248–255 (IEEE, 2009).
Krause, J., Johnson, J., Krishna, R. & Fei-Fei, L. A hierarchical approach for generating descriptive image paragraphs. Preprint at https://arxiv.org/abs/1611.06607 (2016).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. Preprint at https://arxiv.org/abs/1409.0473 (2016).
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. & Torralba, A. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2921–2929 (IEEE, 2016).
Abadi, M. et al. Tensorflow: a system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation Vol. 16 265–283 (USENIX Association, 2016).
Acknowledgements
The authors thank the Department of Pathology, University of Florida (UF), and UF Health Shands Hospital for support with data collection. The authors also thank members of the Moffitt Cancer Center and the Department of Pathology, the First Affiliated Hospital of Xi’an Jiaotong University, for their participation in this research, and thank all participating pathologists for their valuable suggestions and active involvement. Thanks also go to Y. Cai for assistance with figure production. The research reported in this publication was supported by the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under award no. 5R01AR065479-05. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Author information
Authors and Affiliations
Contributions
Z.Z. led the development and evaluation. Z.Z., C.W. and L.Y. designed the research. Z.Z. implemented the algorithm. Z.Z., P.C., M.M. and M.S. collected and cleaned the data and developed the annotation software. L.Y. and M.B. recruited pathologists for annotation and machine–human comparison. L.C. and P.C. managed the machine–human competition. J.D., N.A., F.K.K. and S.I.D. participated in the competition. Z.Z. wrote the manuscript. M.M., F.X., Y.X., X.S., F.L., H.S. and J.C. provided valuable comments on the algorithm design and the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
About this article
Cite this article
Zhang, Z., Chen, P., McGough, M. et al. Pathologist-level interpretable whole-slide cancer diagnosis with deep learning. Nat Mach Intell 1, 236–245 (2019). https://doi.org/10.1038/s42256-019-0052-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s42256-019-0052-1
This article is cited by
-
Artificial intelligence applications in histopathology
Nature Reviews Electrical Engineering (2024)
-
Which data subset should be augmented for deep learning? a simulation study using urothelial cell carcinoma histopathology images
BMC Bioinformatics (2023)
-
Hierarchical AI enables global interpretation of culture plates in the era of digital microbiology
Nature Communications (2023)
-
Colorectal cancer lymph node metastasis prediction with weakly supervised transformer-based multi-instance learning
Medical & Biological Engineering & Computing (2023)
-
Survey on Explainable AI: From Approaches, Limitations and Applications Aspects
Human-Centric Intelligent Systems (2023)