Abstract
Background and aims
Computed tomography (CT) scan is frequently used to detect hepatocellular carcinoma (HCC) in routine clinical practice. The aim of this study is to develop a deep-learning AI system to improve the diagnostic accuracy of HCC by analysing liver CT imaging data.
Methods
We developed a deep-learning AI system by training on CT images from 7512 patients at Henan Provincial Peoples’ Hospital. Its performance was validated on one internal test set (Henan Provincial Peoples’ Hospital, n = 385) and one external test set (Henan Provincial Cancer Hospital, n = 556). The area under the receiver-operating characteristic curve (AUROC) was used as the primary classification metric. Accuracy, sensitivity, specificity, precision, negative predictive value and F1 metric were used to measure the performance of AI systems and radiologists.
Results
AI system achieved high performance in identifying HCC patients, with AUROC of 0.887 (95% CI 0.855–0.919) on the internal test set and 0.883 (95% CI 0.855–0.911) on the external test set. For internal test set, accuracy was 81.0% (76.8–84.8%), sensitivity was 78.4% (72.4–83.7%), specificity was 84.4% (78.0–89.6%) and F1 (harmonic average of precision and recall rate) was 0.824. For external test set, accuracy was 81.3% (77.8–84.5%), sensitivity was 89.4% (85.0–92.8%), specificity was 74.0% (68.5–78.9%) and F1 was 0.819. Compared with radiologists, AI system achieved comparable accuracy and F1 metric on internal test set (0.853 versus 0.818, P = 0.107; 0.863 vs. 0.824, P = 0.082) and external test set (0.805 vs. 0.793, P = 0.663; 0.810 vs. 0.814, P = 0.866). The predicted HCC risk scores by AI system in HCC patients with multiple tumours and high fibrosis stage were higher than those with solitary tumour and low fibrosis stage (tumour number: 0.197 vs. 0.138, P = 0.006; fibrosis stage: 0.183 vs. 0.127, P < 0.001). Radiologists’ review showed that the accuracy of saliency heatmaps predicted by algorithms was 92.1% (95% CI: 89.2–95.0%).
Conclusions
AI system achieved high performance in the detection of HCC compared with a group of specialised radiologists. Further investigation by prospective clinical trials was necessitated to verify this model.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 24 print issues and online access
$259.00 per year
only $10.79 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The original CT data that consist of the training sets and test sets are available on request from the corresponding author. They are not publicly available due to the privacy of research participants.
Code availability
The code used in this study to train AI models is freely accessed and uploaded in supplementary files.
References
Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2018;68:394–424. https://doi.org/10.3322/caac.21492.
El-Serag HB. Hepatocellular carcinoma. N Engl J Med. 2011;365:1118–27. https://doi.org/10.1056/NEJMra1001683.
Lim JH, Kim MJ, Park CK, Kang SS, Lee WJ, Lim HK. Dysplastic nodules in liver cirrhosis: detection with triple phase helical dynamic CT. Br. J Radiol. 2004;77:911–6. https://doi.org/10.1259/bjr/56713551.
Choi JY, Lee JM, Sirlin CB. CT and MR imaging diagnosis and staging of hepatocellular carcinoma: part I. Development, growth, and spread: key pathologic and imaging aspects. Radiology. 2014;272:635–54. https://doi.org/10.1148/radiol.14132361.
Laghi A, Iannaccone R, Rossi P, Carbone L, Ferrari R, Mangiapane F, et al. Hepatocellular carcinoma: detection with triple-phase multi-detector row helical CT in patients with chronic hepatitis. Radiology. 2003;226:543–9. https://doi.org/10.1148/radiol.2262012043.
Ippolito D, Sironi S, Pozzi M, Antolini L, Invernizzi F, Ratti L, et al. Perfusion CT in cirrhotic patients with early stage hepatocellular carcinoma: assessment of tumor-related vascularization. Eur J Radiol. 2010;73:148–52. https://doi.org/10.1016/j.ejrad.2008.10.014.
Sahani DV, Holalkere NS, Mueller PR, Zhu AX. Advanced hepatocellular carcinoma: CT perfusion of liver and tumor tissue-initial experience. Radiology. 2007;243:736–43. https://doi.org/10.1148/radiol.2433052020.
Khalili K, Kim TK, Jang HJ, Haider MA, Khan L, Guindi M, et al. Optimization of imaging diagnosis of 1-2 cm hepatocellular carcinoma: an analysis of diagnostic performance and resource utilization. J Hepatol. 2011;54:723–8. https://doi.org/10.1016/j.jhep.2010.07.025.
Bruix J, Sherman M. Management of hepatocellular carcinoma: an update. American Association for the Study of Liver Diseases. Hepatology. 2011;53:1020–2. https://doi.org/10.1002/hep.24199.
European Association For The Study Of The Liver; European Organisation For Research And Treatment Of Cancer. EASL-EORTC clinical practice guidelines: management of hepatocellular carcinoma. J Hepatol. 2012;56:908–43. https://doi.org/10.1016/j.jhep.2011.12.001.
Omata M, Lesmana LA, Tateishi R, Chen PJ, Lin SM, Yoshida H, et al. Asian Pacific Association for the Study of the Liver consensus recommendations on hepatocellular carcinoma. Hepatol Int. 2010;4:439–74. https://doi.org/10.1007/s12072-010-9165-7.
Tang A, Bashir MR, Corwin MT, Cruite I, Dietrich CF, Do RKG, et al. Evidence supporting LI-RADS major features for CT- and MR imaging-based diagnosis of hepatocellular carcinoma: a systematic review. Radiology. 2018;286:29–48. https://doi.org/10.1148/radiol.2017170554.
Bruix J, Sherman M. Management of hepatocellular carcinoma. Practice Guidelines Committee, American Association for the Study of Liver Diseases. Hepatology. 2005;42:1208–36. https://doi.org/10.1002/hep.20933.
Heimbach JK, Kulik LM, Finn RS, Sirlin CB, Abecassis MM, Roberts LR, et al. AASLD guidelines for the treatment of hepatocellular carcinoma. Hepatology. 2018;67:358–80. https://doi.org/10.1002/hep.29086.
Russo FP, Imondi A, Lynch EN, Farinati F. When and how should we perform a biopsy for HCC in patients with liver cirrhosis in 2018? A review. Dig Liver Dis. 2018;50:640–6. https://doi.org/10.1016/j.dld.2018.03.014.
Roberts LR, Sirlin CB, Zaiem F, Almasri J, Prokop LJ, Heimbach JK, et al. Imaging for the diagnosis of hepatocellular carcinoma: a systematic review and meta-analysis. Hepatology. 2018;67:401–21. https://doi.org/10.1002/hep.29487.
Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL. Artificial intelligence in radiology. Nat Rev Cancer. 2018;18:500–10. https://doi.org/10.1038/s41568-018-0016-5.
Bi WL, Hosny A, Schabath MB, Giger ML, Birkbak NJ, Mehrtash A, et al. Artificial intelligence in cancer imaging: Clinical challenges and applications. CA Cancer J Clin. 2019;69:127–57. https://doi.org/10.3322/caac.21552.
Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med. 2019;380:1347–58. https://doi.org/10.1056/NEJMra1814259.
Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J Am Med Assoc. 2016;316:2402–10. https://doi.org/10.1001/jama.2016.17216.
Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–8. https://doi.org/10.1038/nature21056.
Li XC, Zhang S, Zhang Q, Wei X, Pan Y, Zhao J, et al. Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol. 2019;20:193–201. https://doi.org/10.1016/S1470-2045(18)30762-9.
Chilamkurthy S, Ghosh R, Tanamala S, Biviji M, Campeau NG, Venugopal VK, et al. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet. 2018;392:2388–96. https://doi.org/10.1016/S0140-6736(18)31645-3.
Ardila D, Kiraly AP, Bharadwaj S, Choi B, Reicher JJ, Peng L, et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat Med. 2019;25:954–61. https://doi.org/10.1038/s41591-019-0447-x.
He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. IEEE conference on computer vision and pattern recognition, 2016. Las Vegas, NV, USA; 2016. p. 770–8. https://doi.org/10.1109/CVPR.2016.90
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, et al. ImageNet large scale visual recognition challenge. Int J Comput Vis. 2015;115:211–52.
Zhang HY, Cisse M, Dauphin YN, Lopez-Paz D. Mixup: beyond empirical risk minimization. International Conference on Learning Representations, 2018. arXiv:1710.09412v2. 2018. Available from: https://arxiv.org/abs/1710.09412
Mundhenk TN, Chen B, Friedland G. Efficient saliency maps for explainable AI. International Conference on Learning Representations, 2020. arXiv:1911.11293v2. 2019. Available from: https://arxiv.org/abs/1911.11293
Bedossa P, Poynard T. An algorithm for the grading of activity in chronic hepatitis C. The METAVIR Cooperative Study Group. Hepatology. 1996;24:289–93. https://doi.org/10.1002/hep.510240201.
Hannun AY, Rajpurkar P, Haghpanahi M, Tison GH, Bourn C, Turakhia MP, et al. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat Med. 2019;25:65–69. https://doi.org/10.1038/s41591-018-0268-3.
Zhou DJ, Tian F, Tian XD, Sun L, Huang XH, Zhao F, et al. Diagnostic evaluation of a deep learning model for optical diagnosis of colorectal cancer. Nat Commun. 2020;11:2961 https://doi.org/10.1038/s41467-020-16777-6.
Ahmad OF, Soares AS, Mazomenos E, Brandao P, Vega R, Seward E, et al. Artificial intelligence and computer-aided diagnosis in colonoscopy: current evidence and future directions. Lancet Gastroenterol Hepatol. 2019;4:71–80. https://doi.org/10.1016/S2468-1253(18)30282-6.
Ehteshami Bejnordi B, Veta M, Johannes van Diest P, Ginneken BV, Karssemeijer N, Litjens G, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. J Am Med Assoc. 2017;318:2199–210.
Byrne MF, Chapados N, Soudan F, Oertel C, Pérez ML, Kelly R, et al. Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut. 2019;68:94–100. https://doi.org/10.1136/gutjnl-2017-314547.
Sun R, Limkin EJ, Vakalopoulou M, Dercle L, Champiat S, Han SR, et al. A radiomics approach to assess tumor-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study. Lancet Oncol. 2018;19:1180–91. https://doi.org/10.1016/S1470-2045(18)30413-3.
Silva MA, Hegab B, Hyde C, Guo B, Buckels JAC, Mirza DF. Needle track seeding following biopsy of liver lesions in the diagnosis of hepatocellular cancer: a systematic review and meta-analysis. Gut. 2008;57:1592–6. https://doi.org/10.1136/gut.2008.149062.
Khalili K, Kim TK, Jang HJ, Yazdi LK, Guindi M, Sherman M. Indeterminate 1-2-cm nodules found on hepatocellular carcinoma surveillance: biopsy for all, some, or none? Hepatology. 2011;54:2048–54. https://doi.org/10.1002/hep.24638.
Chernyak V, Fowler KJ, Kamaya A, Kielar AZ, Elsayes KM, Bashir MR, et al. Liver imaging reporting and data system (LI-RADS) version 2018: imaging of hepatocellular carcinoma in at-risk patients. Radiology. 2018;289:816–30. https://doi.org/10.1148/radiol.2018181494.
Funding
This work was supported by the natural science foundation of Tianjin Education Committee (Project No. 2018KJ072), Tianjin Science and Technology Committee (Project No. 18JCQNJC80800), Henan Provincial Science and Technology Research Projects (Project No. 212102310689) and National Natural Science Foundation of China (Project No. 31801117).
Author information
Authors and Affiliations
Contributions
MW, XL and FT take full responsibility for the integrity of the data and the accuracy of the analysis. Concept and design were contributed by FT, XL and DK. Drafting of the manuscript is attributed to FT, XL and MW. Clinical record data were obtained and reviewed by FF, BZ, YB and XM. Statistical analysis is attributed to YY and HS. FF, BZ, YB, QW, JW and XM read and interpreted CT images. LS, QL and ML extracted and reviewed pathological data. Technical and material support is attributed to PY, XL and YY. We authors thank Genevieve Nemeth of Harvard Medical School for editing the language of this manuscript.
Corresponding authors
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent to publish
Not applicable.
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
About this article
Cite this article
Wang, M., Fu, F., Zheng, B. et al. Development of an AI system for accurately diagnose hepatocellular carcinoma from computed tomography imaging data. Br J Cancer 125, 1111–1121 (2021). https://doi.org/10.1038/s41416-021-01511-w
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41416-021-01511-w
This article is cited by
-
Diagnostic Performance of Artificial Intelligence in Detection of Hepatocellular Carcinoma: A Meta-analysis
Journal of Imaging Informatics in Medicine (2024)
-
Performance and clinical applicability of machine learning in liver computed tomography imaging: a systematic review
European Radiology (2023)
-
Development of a deep-learning model for classification of LI-RADS major features by using subtraction images of MRI: a preliminary study
Abdominal Radiology (2023)
-
Liver function classification based on local direction number and non-local binary pattern
Multimedia Tools and Applications (2022)