Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Matters Arising
  • Published:

Reply to: ‘IMPRES does not reproducibly predict response to immune checkpoint blockade therapy in metastatic melanoma’

Nature Medicine volume 25, pages 1836–1838 (2019)Cite this article

Subjects

The Original Article was published on 05 December 2019

replying to Carter, J. A. et al. Nature Medicine https://doi.org/10.1038/s41591-019-0671-4 (2019)

Carter et al.1 have raised several criticisms regarding our recent publication2. In a nutshell, we find these criticisms to be flawed and misleading. In our response, we show (i) that the process employed in our original publication to generate the immuno-predictive score (IMPRES) is appropriate and unbiased and (ii) that the approach presented by Carter et al. is inappropriate and results in data overfitting.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: IMPRES training set is unbiased and reproducibly predicts response to immune checkpoint blockade therapy.

Data availability

All data analyzed in this manuscript are publicly available, and the accessions or web links to each dataset are provided either in the original publications or in the relevant parts of this manuscript.

Code availability

The feature selection from the original training set and all melanoma ICB response prediction codes are available from the original publication: https://github.com/noamaus/IMPRES-codes. The additional code to sample similar training sets is provided at https://github.com/noamaus/IMPRES-codes/tree/master/MAIN_CODES/Feature_selection/Sample_training_set.

References

  1. Carter, J. A. et al. IMPRES does not reproducibly predict response to immune checkpoint blockade therapy in metastatic melanoma. Nat. Med. https://doi.org/10.1038/s41591-019-0671-4 (2019).

  2. Auslander, N. et al. Robust prediction of response to immune checkpoint blockade therapy in metastatic melanoma.Nat. Med. 24, 1545–1549 (2018).

    Article  CAS  Google Scholar 

  3. Rudlof, S. & Köppen, M. Stochastic hill climbing with learning by vectors of normal distributions. in Proceedings of the First Online Workshop on Soft Computing (WSC1) 60–70 (1996).

  4. Patrick, K. Comparison of simulated annealing and hill climbing in the course timetabling problem. Afr. J. Math. Comput. Sci. Res. 5, 176–178 (2012).

    Article  Google Scholar 

  5. Jacobson, S. H. & Yücesan, E. Analyzing the performance of generalized hill climbing algorithms. J. Heuristics 10, 387–405 (2004).

    Article  Google Scholar 

  6. Vinh, N. X., Chetty, M., Coppel, R. & Wangikar, P. P. Globalmit: learning globally optimal dynamic Bayesian network with the mutual information test criterion. Bioinformatics 27, 2765–2766 (2011).

    Article  CAS  Google Scholar 

  7. Zhang, Y., Lei, T., Barzilay, R. & Jaakkola, T. Greed is good if randomized: new inference for dependency parsing. in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing 1013–1024 (Association for Computational Linguistics, 2015).

  8. Burke, E. K. & Bykov, Y. The late acceptance hill-climbing heuristic. Eur. J. Oper. Res. 258, 70–78 (2017).

    Article  Google Scholar 

  9. Efron, B. Introduction to the Bootstrap (Statistical Science, 2003).

  10. Fu, W. J., Carroll, R. J. & Wang, S. Estimating misclassification error with small samples via bootstrap cross-validation. Bioinformatics 21, 1979–1986 (2005).

    Article  CAS  Google Scholar 

  11. Popovici, V. et al. Effect of training-sample size and classification difficulty on the accuracy of genomic predictors. Breast Cancer Res. 12, R5 (2010).

    Article  Google Scholar 

  12. Pang, S., Havukkala, I., Hu, Y. & Kasabov, N. Classification consistency analysis for bootstrapping gene selection. Neural Comput. Appl. 16, 527–539 (2007).

    Article  Google Scholar 

  13. Zhang, Y. Q. & Rajapakse, J. C. (eds) Machine Learning in Bioinformatics (Wiley, 2008).

  14. Huang, A. C. et al. A single dose of neoadjuvant PD-1 blockade predicts clinical outcomes in resectable melanoma. Nat. Med. 25, 454–461 (2019).

    Article  CAS  Google Scholar 

  15. Wagner, G. P., Kin, K. & Lynch, V. J. Measurement of mRNA abundance using RNA-seq data: RPKM measure is inconsistent among samples. Theory Biosci. 131, 281–285 (2012).

  16. Van Allen, E. M. et al. Genomic correlates of response to CTLA-4 blockade in metastatic melanoma. Science 350, 207–211 (2015).

  17. Wu, D. C., Yao, J., Ho, K. S., Lambowitz, A. M. & Wilke, C. O. Limitations of alignment-free tools in total RNA-seq quantification. BMC Genomics 19, 510 (2018).

  18. Riaz, N. et al. Tumor and microenvironment evolution during immunotherapy with nivolumab. Cell 171, 934–949 (2017).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Center for Biotechnology Information, National Library of Medicine, US National Institutes of Health, Bethesda, MD, USA

    Noam Auslander

  2. Cancer Data Science Lab (CDSL), National Cancer Institute, US National Institutes of Health, Bethesda, MD, USA

    Joo Sang Lee & Eytan Ruppin

  3. Samsung Medical Center, Sungkyunkwan University School of Medicine, Suwon, Republic of Korea

    Joo Sang Lee

Authors
  1. Noam Auslander
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joo Sang Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eytan Ruppin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

N.A. performed the additional analysis and provided the additional code; N.A. and E.R. designed experiments; N.A., E.R. and J.S.L wrote the manuscript.

Corresponding author

Correspondence to Eytan Ruppin.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Saheli Sadanand was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 The performance of scores selected with different sampled training sets, and survival prediction for an independent NB dataset.

(a) Boxplots showing the area under the ROC curve (AUC) for responder/non-responder classification obtained with 200 feature sets selected over the NB data, for each of the melanoma datasets (b) The fraction of features recurring in these 200 feature sets (Original IMPRES features are colored in red). (c) IMPRES score predicts survival for an independent NB dataset from TARGET, where the top 15 features reported by Carter et al. fail to do that (a strong indication of overfitting). The NB data is RPKM values downloaded from cBioPortal spanning 39 NB patients of age 1.5 years or less, as described originally. We predicted overall survival using the original IMPRES features (with the same Kaplan-Meier codes used in Figure 2C in our original publication). The P-values are for two-sided log-rank test.

Extended Data Fig. 2 IMPRES performance on TPM and FPKM data.

(a) ROC curves quantifying the accuracy of IMPRES response prediction in two different TPM generated datasets generated by the Van Allen group (https://github.com/vanallenlab/VanAllen_CTLA4_Science_RNASeq_TPM). (b) Boxplots showing IMPRES performance when calculated on FPKM values of the Riaz et al. dataset, for progressive diseases (PD), stable disease (SD), partial response (PR) and complete response (CR). The P-values are for one-sided rank-sum test, comparing IMPRES scores of each non-PD group (SD/PR/CR) to that of PD.

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Auslander, N., Lee, J.S. & Ruppin, E. Reply to: ‘IMPRES does not reproducibly predict response to immune checkpoint blockade therapy in metastatic melanoma’. Nat Med 25, 1836–1838 (2019). https://doi.org/10.1038/s41591-019-0646-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-019-0646-5

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Cancer

Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

Get what matters in cancer research, free to your inbox weekly. Sign up for Nature Briefing: Cancer