Journals | Policy | Permission

World Journal of Oncology

Original Article

Volume 2, Number 4, August 2011, pages 181-190

Validation of a Predictive Model for Survival in Patients With Advanced Cancer: Secondary Analysis of RTOG 9714

Survival prediction of patients with advanced cancer is one of the most difficult and least welcomed tasks clinicians have to face. However such an estimate is very important in end-of-life care [1]. Inaccurate prediction of survival often results in inadequate access to palliative care. Overly optimistic prediction may deter patients from being involved in palliative programs at an earlier stage.

Patient preferences about the trade-off between the risks and benefits associated with treatment strategies are often based on perceptions of prognosis. Inaccurate perceptions can lead to unrealistic expectations [2]. Weeks and colleagues found that patients with a misperceived optimistic prognosis often request medical therapies that most physicians would consider futile. These same patients were 8.5 times more likely to favor receiving aggressive, life-extending medical care than were patients with more accurate estimates of their 6-month survival. More disturbingly, those with overly optimistic prognoses were more likely to die in hospital on mechanical ventilation than were those patients with more realistic estimates of their survival potential. The authors conclude that terminal cancer patients’ miscalibrated optimistic prognosis may lead them to choose highly aggressive, invasive and ultimately futile medical care rather than palliative care [3].

The accurate classification of patients with advanced cancer into groups with similar and predictable survival has the potential to improve delivery of care and minimize undertreatment or overtreatment [2, 4]. In validating a classification model, there is a hierarchy of increasing stringent validation strategies [5]: 1) Internal – evaluation restricted to a single training data set; 2) Temporal – evaluation on a second data set at a different time point from the same center; 3) External – evaluation on data from a different center, perhaps by different investigators.

We previously developed a predictive model for patients with advanced cancer by employing three readily available parameters: primary cancer site, site of metastases and Karnofsky performance score (KPS). The initial classification model was developed with a training data set (n = 395) comprised of patients treated at Sunnybrook Odette Cancer Center’s Rapid Response Radiotherapy Program (RRRP) in 1999. Three risk groups–low, intermediate, high–were determined by partial score method and number of risk factors (NRF) method (details in materials and methods section) (Table 1) [6, 7]. A temporal validation set (n = 445) of patients treated in the RRRP in 2000 and an external validation set (n = 467) of patients treated at Princess Margaret Hospital’s Palliative Radiation Oncology Program in 2002 were used to successfully evaluate the initial model.

Click to view

Table 1. Risk Groups Previously Identified

The objective of this secondary study was to use Radiation Therapy Oncology Group (RTOG) 9714 data to provide an additional external validation set to the two methods comprised of patients treated among multiple institutions in the United States and Canada.

The Radiation Therapy and Oncology Group (RTOG) and the North Central Cancer Treatment Group conducted a randomized Phase III trial with breast or prostate cancer patients with bone metastases (RTOG 9714) [8]. Eligible patients had moderate or severe pain, as indicated by a Brief Pain Inventory (BPI) worst pain score of 5 and above or narcotic medication with a daily oral morphine equivalent dose of at least 60 mg if pain scores were < 5. Patients were randomized between treatment with a single fraction of 8 Gy and 30 Gy in 10 fractions. The Karnofsky performance status of the enrolled patients was at least 40, with an estimated life expectancy of at least 3 months. Patients were excluded if there was prior radiation therapy or palliative surgery to the planned radiation treatment site, pathologic or impending fracture, or compression of the spinal cord or cauda equina [8]. The survival status and the date of death from any cause or last follow up of the patients were current as of February 2009.

A prognostic score (partial score) based on the regression coefficients of the Cox regression model was assigned to the three factors [primary cancer site (breast/prostate/lung/others), site of metastases (bone/others), and KPS (> 60/≤ 60)] as in the previous work (Table 1) [6]. The survival predictive score (SPS) for a given patient was obtained by adding together his/her partial scores for the three factors [6, 9]. Patients were classified into risk groups based on their SPS score. Three risk groups A, B and C were established.

The patients were also grouped according to the total number of risk factors (NRF) that they possessed [6]. The three risk factors are as follows: (1) non-breast (i.e., prostate), (2) site of metastases other than bone only, and (3) KPS ≤ 60. Three risk groups I, II and III were established.

The logrank test was used to determine differences in overall survival in the resultant risk groups. The resultant median survival estimates in each risk group were also compared to those of the corresponding risk group in the previous datasets for similarity. Additional methods were then used to evaluate model discrimination. The C index of predictive discrimination measures the proportion of correct predictions based on observed responses. The C index ranges from 0 to 1 with values of 0.5 indicating no predictive discrimination (random classification) between patients with different outcomes and values close to 1 indicating perfect discrimination (proper classification) [10]. The Royston and Sauerbrei D statistic is also a measure of discrimination of the survival model based on its ability to separate survival estimates between risk groups. Higher values indicate a better degree of model separation [11].

The RTOG external validation set comprised of 908 patients treated at 66 different radiation facilities from 1998 to 2002. Canadian patients represented only 4% of the patient population (Table 2). The mean age was 65 years (SD 12). There were slightly more females (51% vs. 49%). Although geographic variability was obtained, RTOG 9714 had a fairly homogenous population in regards to prognostic variables. All patients had bone metastases and had either a breast (51%) or a prostate (49%) primary tumor, as these disease specifications were trial eligibility criteria. Only twenty-four percent of patients had KPS ≤ 60 as a life expectancy of at least 3 months was also a trial eligibility criterion [8].

Click to view

Table 2. Patient Characteristics From RTOG Validation Set (N = 908)

The SPS method classified all patients into the low-risk Group A with scores from 0 - 4 (Table 3). The 3 month survival estimate for patients in Group A was 84% which was comparable to that of the training set at 82% (Table 4). Given that there were no identified intermediate/high-risk patients, the C index of discrimination was 0.96, indicating near perfect model discrimination.

Click to view

Table 3. Derivation of Risk Groups From RTOG Validation Set

Click to view

Table 4. Summary of Model Performance and Survival Estimates: Partial Score Method

Based on the NRF, two risk groups were identified. Prostate patients with KPS ≤ 60 were classified into the intermediate-risk group II. Eleven percent of patients had those 2 risk factors. Prostate patients with KPS > 60 and all breast patients were classified into the low-risk Group I. Eighty-nine percent of patients had those 0 - 1 risk factors. The 3 month survival estimates for patients in Group I and Group II were 85% and 72%, respectively, which was comparable to that of the training set at 80% and 73%, respectively (Table 5). Given that there were no identified high-risk patients, the C index of discrimination was 0.94, indicating near perfect model discrimination. The Royston and Sauerbrei D statistic was 0.82, indicating good model discrimination.

Click to view

Table 5. Summary of Model Performance and Survival Estimates: Number of Risk Factors Method

Based on these results, the NRF method is preferable to the SPS method in that it gives a more accurate classification of patients and requires simpler calculation. The NRF method is able to further distinguish intermediate-risk patients that the SPS method classifies into a low-risk group only. The difference in survival estimates of the two classification groups was statistically significant (P < 0.0001) (Fig. 1).

Click for large image

Figure 1. Survival estimates and risk classification group.

Physicians are often requested to predict patient survival at times of referral to hospice programs and enrollment into clinical trials [2]. However, clinicians are often overly optimistic in the survival prediction of terminally ill cancer patients [12, 13]. Parkes, in a commentary, aired his disappointment that doctors are still no better at predicting the length of survival of terminally ill patients than they were 27 years ago. He also stated that if all predictions had been divided by two, they would have been marginally more accurate. He urged that prognoses should be based on proven indexes and not intuition. Physicians need to stop guessing, and when predictions are needed, they should make use of the available predictive instruments [14].

The inaccuracy of survival prediction by palliative radiation oncologists has been well documented. After consultations of cancer patients with metastatic disease for referral of palliative radiotherapy, six radiation oncologists provided survival estimates for 739 patients. These were compared with the actual dates of death obtained from the Cancer Death Registry. The prediction of survival by palliative radiation oncologists was inaccurate and tended to be overly optimistic [15]. Hartsell et al reported the physician prediction of the survival of patients in RTOG 9714. Again the survival prediction was optimistic compared to actual survival by an average of 3 months. The median survival of the 618 expired patients was 6.5 months and the median physician prediction of survival was 12 months [16].

Reviews on survival prediction were conducted by Glare et al [17] and the European Association for Palliative Care [18]. In a systematic review of physicians’ survival prediction in terminally ill cancer patients, Glare et al evaluated 8 published studies providing 1563 individual prediction-survival dyads. Clinical prediction of survival (CPS) was generally overoptimistic, with the median CPS 42 days and the median actual survival (AS) 29 days. The CPS was correct to within one week in only 25% of cases and overestimated survival by at least four weeks in 27%. The survival of patients was typically 30% shorter than predicted.

The Steering Committee of the European Association for Palliative Care (EAPC) published their evidence-based clinical recommendations on the prognostic factors in advanced cancer patients. In their analysis of the 16 eligible studies, the correlation coefficient of the CPS and AS varied between 0.2 and 0.65. CPS was more than twice as likely to be overoptimistic versus overpessimistic and to overestimate the length of actual survival by a factor of between 3 and 5. The committee recommends clinicians should consider using CPS in combination with other prognostic factors or scores to improve the accuracy of their predictions [18].

The first study investigating a prognostic model in 395 patients contained six significant covariates: primary cancer, site, site of metastases, KPS, fatigue, appetite, and shortness of breath [6]. The discrimination C index for this model was 0.73, 0.75 and 0.81 for discrimination between patients surviving past 3, 6, and 12 months, respectively.

The subsequent study attempted to simplify this six variable model to include only three factors: primary cancer site, site of metastases, and KPS [7]. Utilizing the SPS method, a training set, temporal validation set and external validation set had C indices of 0.66, 0.65 and 0.63, respectively. Using the NRF method, D statistics were listed as 0.65, 0.66, and 0.63 for the same above three sets, respectively. The patient characteristics can be seen in Table 6 for comparison with the current validation set.

Click to view

Table 6. Summary of Patient Characteristics From Two Previous Survival Prediction Investigations

This present analysis is limited by the eligibility criteria of RTOG 9714 allowing only breast/prostate primary tumors and KPS > 40, thus limiting the classification ability of the original model. We have validated the low risk group in SPS model and the low/intermediate risk groups in the NRF model but cannot draw conclusions about the high risk patients. We encourage other investigators to validate especially the high risk groups. Until then, as Parkes encourages, when predictions are needed, physicians should make use of the available validated predictive instruments such as the current one.

Studies have been conducted to determine the optimal way to present medical information to patients and their families. It has been shown that survival curves are a potentially powerful tool to communicate information about health and treatment outcomes [19]. Survival curves provide a graphic presentation of the risk of an outcome over time, as they include a large amount of information that is difficult to convey with numbers alone [20, 21]. Furthermore, use of survival curves avoids the problem of having to select the time points to present–this is important as such a selection has been shown to influence treatment choice [22]. Figure 1 may be of help to clinicians when counseling on survival.

There has been no study examining whether an accurate prediction of survival can improve actual clinical care, nor investigating if the models improve the decision making in the care of this group of patients. Future studies should also evaluate the inception criteria and define common inception points for the accrual of patients in studies of advanced disease. Quality of life assessment including self-rated health may assist in the selection of homogeneous cohorts of patients with terminal cancer and fine-tune the prognostic models.

Acknowledgments

We thank Stacy Yuen for the secretarial assistance.

Grant

This project was supported by RTOG grant U10 CA21661, CCOP grant U10 CA37422 from the National Cancer Institute (NCI). This manuscript’s contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Cancer Institute.

Conflict of Interest

None

1. Maher EJ. How long have I got doctor?. Eur J Cancer. 1994;30A(3):283-284.
   pubmed doi
2. Lamont EB, Christakis NA. Some elements of prognosis in terminal cancer. Oncology (Williston Park). 1999;13(8):1165-1170, discussion 1172-1164, 1179-1180.
   pubmed
3. Weeks JC, Cook EF, O'Day SJ, Peterson LM, Wenger N, Reding D, Harrell FE, et al. Relationship between cancer patients' predictions of prognosis and their treatment preferences. JAMA. 1998;279(21):1709-1714.
   pubmed doi
4. Christakis NA, Escarce JJ. Survival of Medicare patients after enrollment in hospice programs. N Engl J Med. 1996;335(3):172-178.
   pubmed doi
5. Altman DG, Royston P. What do we mean by validating a prognostic model?. Stat Med. 2000;19(4):453-473.
   pubmed doi
6. Chow E, Abdolell M, Panzarella T, Harris K, Bezjak A, Warde P, Tannock I. Predictive model for survival in patients with advanced cancer. J Clin Oncol. 2008;26(36):5863-5869.
   pubmed doi
7. Chow E, Fung K, Panzarella T, Bezjak A, Danjoux C, Tannock I. A predictive model for survival in metastatic cancer patients attending an outpatient palliative radiotherapy clinic. Int J Radiat Oncol Biol Phys. 2002;53(5):1291-1302.
   pubmed doi
8. Hartsell WF, Scott CB, Bruner DW, Scarantino CW, Ivker RA, Roach M 3rd, Suh JH, et al. Randomized trial of short- versus long-course radiotherapy for palliation of painful bone metastases. J Natl Cancer Inst. 2005;97(11):798-804.
   pubmed doi
9. Pirovano M, Maltoni M, Nanni O, Marinari M, Indelli M, Zaninetta G, Petrella V, et al. A new palliative prognostic score: a first step for the staging of terminally ill cancer patients. Italian Multicenter and Study Group on Palliative Care. J Pain Symptom Manage. 1999;17(4):231-239.
   pubmed doi
10. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361-387.
    pubmed doi
11. Royston P, Sauerbrei W. A new measure of prognostic separation in survival data. Stat Med. 2004;23(5):723-748.
    pubmed doi
12. Parkes CM. Accuracy of predictions of survival in later stages of cancer. Br Med J. 1972;2(5804):29-31.
    pubmed doi
13. Heyse-Moore LH, Johnson-Bell VE. Can doctors accurately predict the life expectancy of patients with terminal cancer?. Palliative Med. 1987;1:165-166.
    doi
14. Parkes CM. Commentary: prognoses should be based on proved indices not intuition. BMJ. 2000;320(7233):473.
    pubmed
15. Chow E, Davis L, Panzarella T, Hayter C, Szumacher E, Loblaw A, Wong R, et al. Accuracy of survival prediction by palliative radiation oncologists. Int J Radiat Oncol Biol Phys. 2005;61(3):870-873.
    pubmed doi
16. Hartsell WF, Desilvio M, Bruner DW, Scarantino C, Ivker R, Roach M 3rd, Suh J, et al. Can physicians accurately predict survival time in patients with metastatic cancer? Analysis of RTOG 97-14. J Palliat Med. 2008;11(5):723-728.
    pubmed doi
17. Glare P, Virik K, Jones M, Hudson M, Eychmuller S, Simes J, Christakis N. A systematic review of physicians' survival predictions in terminally ill cancer patients. BMJ. 2003;327(7408):195-198.
    pubmed doi
18. Maltoni M, Caraceni A, Brunelli C, Broeckaert B, Christakis N, Eychmueller S, Glare P, et al. Prognostic factors in advanced cancer patients: evidence-based clinical recommendations—a study by the Steering Committee of the European Association for Palliative Care. J Clin Oncol. 2005;23(25):6240-6248.
    pubmed doi
19. Mazur DJ, Hickam DH. Interpretation of graphic data by patients in a general medicine clinic. J Gen Intern Med. 1990;5(5):402-405.
    pubmed doi
20. Miller GA. The magical number seven plus or minus two: some limits on our capacity for processing information. Psychol Rev. 1956;63(2):81-97.
    pubmed doi
21. Baron J. Thinking and Deciding. New York: Cambridge University Press, 1994.
22. Mazur DJ, Hickam DH. The effect of physician's explanations on patients' treatment preferences: five-year survival data. Med Decis Making. 1994;14(3):255-258.
    pubmed doi

This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

World Journal of Oncology is published by Elmer Press Inc.