Proteomic Profiling in Ovarian Cancer: Is it Plausible?
Proteomic Profiling in Ovarian Cancer: Is it Plausible?
Abstract & Commentary
Robert L. Coleman, MD, Associate Professor, University of Texas; M.D. Anderson Cancer Center, Houston.
Synopsis: The reproducibility of the proteomic profiling approach has yet to be established.
Source: Baggerly KA, et al. J Natl Cancer Inst. 2005;97: 307-309.
Reproducibility of complex proteomic signatures in detecting early stage ovarian cancer has proven to be a significant challenge in the field of biomarker discovery. Although early reports of proteomic profiling demonstrated near 100% sensitivity and specificity in discriminating ovarian cancer from non-cancer, the ability to reproduce these results across datasets has been difficult. Baggerly and colleagues examined 2 proteomic spectral datasets available in the public domain with the intent of testing the published classification method and assessing reproducibility. They then calculated the probability that classification could occur by chance alone. Using statistical methodology similar to the original reports then followed by random allocation of spectral clusters, Baggerly et al report that the predictive spectral peaks were less accurate (less than 80%) than originally reported. Applying a separate methodology (jack-knife approach) to test the original peaks to a second dataset produced high accuracy (98.4%). However, this level of accuracy was met or exceeded in 6% of the simulations using completely random values. On the basis of these 2 findings, Baggerly et al concluded that proteomic signature profiling to accurately and reproducibly discriminate cancer from non-cancer was not plausible and may be the effect of procedural bias.
Comment by Robert L. Coleman, MD
News of accurate discrimination of cancer patients from unaffected patients via evaluation of blood samples was a welcomed and widely publicized advance in the field of ovarian cancer screening research. Although difficult to understand, the technology of developing and interpreting proteomic signatures from banked blood of both cancer patients, patients with non-cancerous lesions, and normal controls, appeared to offer renewed hope in the ability to accurately classify patients with disease, particularly those with early stage disease, where survival is the best. While the data being produced were patterns (spectral peaks) recognizable only by sophisticated statistical algorithms sensitivity, specificity and accuracy were very high and far better than our best (and current) biomarker methodology. However, almost as soon as the data were reported, news of the inability to accurately reproduce the data permeated the clinical community who were already baited with the promise of a commercial product awaiting approval. The current report, on the surface, seems to dampen that enthusiasm even more where the results of 2 independent data sets were found to not be reproducible and could be explained by chance—that is by overfitting. Overfitting errors are encountered when a multivariate model is used to fit a very large number of possible predictors. Indeed, Baggerly et al demonstrated that simply choosing random spectral peak values for a prediction model met or exceeding the predictive accuracy of the historical model in 6% of their samplings. This would imply that the plausibility for prediction based on the public domain datasets was low and confounded.
The Baggerly article is followed by 2 commentaries in the Journal of the National Cancer Institute that help to put this apparent conundrum into perspective.1 We find from the original authors of one of the sentinel articles that the datasets placed into the public domain was generated from experimental settings where one or more parameters were purposefully altered to study the effects on output spectra. In this situation, the datasets would likely not be reproducible as the baseline between sets is intentionally and artificially adjusted. In addition, it was raised that issues of reproducibility are premature as the technology is a moving target and continuously updated. However, as is discussed in the second commentary, reproducibility is the lifeblood of moving this complex promising field (termed "-omics") forward. It is further complicated by the dichotomy of discovery; on the one hand, you have a test based on pattern recognition alone, where the results are based on unknown and un-described proteins, and on the other, you have biomarkers development, which would take on the tedious process of describing these proteins, showing their relevance (biological impact) to ovarian cancer and then develop a measurable antibody in the patient’s sera. Calls for a new trial have been made, where reproducibility is the mission, in a single data set with careful attention to statistical methodology.2,3
What is clear from the 10 pages of commentary that accompanies the 3-page brief communication is that this field holds tremendous promise in promoting our ability to identify at-risk patients, but must be embarked upon carefully. In addition, free interchange and discussion through publicly disseminated data will help perfect this important task.
Dr. Coleman is Associate Professor, University of Texas; M.D. Anderson Cancer Center, Houston.
References
1. Ransohoff DF. J Natl Cancer Inst. 2005;97:315-319.
2. Zhang Z, et al. Cancer Res. 2004;64:5882-5890.
3. Zhu W, et al. Proc Natl Acad Sci USA. 2003;100: 14666-14671.
Reproducibility of complex proteomic signatures in detecting early stage ovarian cancer has proven to be a significant challenge in the field of biomarker discovery. Although early reports of proteomic profiling demonstrated near 100% sensitivity and specificity in discriminating ovarian cancer from non-cancer, the ability to reproduce these results across datasets has been difficult.Subscribe Now for Access
You have reached your article limit for the month. We hope you found our articles both enjoyable and insightful. For information on new subscriptions, product trials, alternative billing arrangements or group and site discounts please call 800-688-2421. We look forward to having you as a long-term member of the Relias Media community.