Comparison of three mathematical prediction models in patients with a solitary pulmonary nodule

From Guangdong Lung Cancer Institute, Guangdong Provincial Key Laboratory of Translational Medicine in Lung Cancer, Guangdong General Hospital & Guangdong Academy of Medical Sciences, Guangzhou 510080, China.

＊This author is currently a medical postgraduate at Jinan University, the 2nd Clinical College, Shenzhen people’s hospital, Guangzhou 510632, China

Correspondence to: Dr. Xue-Ning Yang, MD. Guangdong Lung Cancer Institute, Guangdong Provincial Key Laboratory of Translational Medicine in Lung Cancer, Guangdong General hospital & Guangdong Academy of Medical Sciences, No. 106 Zhongshan 2nd Road, Guangzhou 510080, China. Email: yangxuening@gmail.com.

Comparison of three mathematical prediction models in patients with a solitary pulmonary nodule

Xuan Zhang＊, Hong-Hong Yan, Jun-Tao Lin, Ze-Hua Wu, Jia Liu, Xu-Wei Cao, Xue-Ning Yang

From Guangdong Lung Cancer Institute, Guangdong Provincial Key Laboratory of Translational Medicine in Lung Cancer, Guangdong General Hospital & Guangdong Academy of Medical Sciences, Guangzhou 510080, China.

＊This author is currently a medical postgraduate at Jinan University, the 2nd Clinical College, Shenzhen people’s hospital, Guangzhou 510632, China

Correspondence to: Dr. Xue-Ning Yang, MD. Guangdong Lung Cancer Institute, Guangdong Provincial Key Laboratory of Translational Medicine in Lung Cancer, Guangdong General hospital & Guangdong Academy of Medical Sciences, No. 106 Zhongshan 2nd Road, Guangzhou 510080, China. Email: yangxuening@gmail.com.

Background:Effective methods for managing patients with solitary pulmonary nodules (SPNs) depend critically on the predictive probability of malignancy.

Methods:Between July 2009 and June 2011, data on gender, age, cancer history, tumor familial history, smoking status, tumor location, nodule size, spiculation, calcification, the tumor border, and the final pathological diagnosis were collected retrospectively from 154 surgical patients with an SPN measuring 3-30 mm. Each fnal diagnosis was compared with the probability calculated by three predicted models—the Mayo, VA, and Peking University (PU) models. The accuracy of each model was assessed using area under the receiver operating characteristics (ROC) and calibration curves.

Results:The area under the ROC curve of the PU model [0.800; 95% confdence interval (CI): 0.708-0.891] was higher than that of the Mayo model (0.753; 95% CI: 0.650-0.857) or VA model (0.728; 95% CI: 0.623-0.833); however, this fnding was not statistically signifcant. To varying degrees, calibration curves showed that all three models overestimated malignancy.

Conclusions:The three predicted models have similar accuracy for prediction of SPN malignancy, although the accuracy is not suffcient. For Chinese patients, the PU model may has greater predictive power.

Solitary pulmonary nodule (SPN); benign and malignant; model; comparison

View this article at:http://dx.doi.org/10.3978/j.issn.1000-9604.2014.11.02chest radiograph or CT scan. When the probability was＜3%, watchful waiting was preferred. When the probability was 3% to 68%, needle biopsy was preferred. When the probability was ＞68%, surgery was preferred.

The VA model is defined by the equations: Pre-test probability of a malignant SPN = ex/(1+ex), x= —8.404+ (2.061× smoking history) + (0.779× age10) + (0.112× diameter) — (0.567× yearsquit10), e is the natural logarithm, and 1 for yes and 0 for no in the smoking history; age10 indicates age in years at the time of nodule identifcation, divided by 10; diameter indicates the largest nodule measurement (in mm) reported on initial chest radiograph or CT scan; and yearsquit10 indicates the number of years since quitting smoking, divided by 10 (0 indicates not applicable). When the probability was ＜20%, watchful waiting was preferred. When the probability was 20% to 69%, needle biopsy was preferred. When the probability was ＞69%, surgery was preferred.

The PU model is defined by the equations: Pre-test probability of a malignant SPN = ex/ (1+ex), x= —4.496+ (0.07× age) + (0.676× diameter) + (0.736× spiculation) + (1.267× family history of cancer) — (1.615× calcifcation) —(1.408× border), e is the natural logarithm, and 1 for yes and 0 for no in the last four elements. Diameter indicates the largest nodule measurement (in cm) reported on initial chest radiograph or CT scan. When the probability was＜46.3%, the nodule should be considered benign. When the probability was ＞46.3%, the nodule should be considered malignant.

This study aimed to evaluate the sensitivity of the three models in verifying a malignant SPN using a set of data using calibration and receiver operating characteristics (ROC) curves.

Materials and methods

We retrospectively reviewed the medical records of 160 patients with SPNs who underwent surgical resection between July 2009 and June 2011. The present study recruited patients who underwent computed tomography (CT) scans at Guangdong General Hospital and without a previous diagnosis of lung cancer. Patients with extrathoracic cancer within 5 years of nodule identifcation were excluded (n=6). A total of 154 patients were finally enrolled in our study, comprising 88 males and 66 females, mainly from the south of China. This study was approved by the hospital ethics committee.

Clinical data were collected, including age, gender, smoking history, history of a previous cancer diagnosis, family history of cancer, pathological diagnosis, and radiology characteristics.

All patients underwent surgical resection of SPNs and pathological diagnoses were obtained. Pulmonary wedge resection and lobectomy were performed in these patients.

Clinical data of 154 patients were calculated by three models separately. The calculation outcomes of the Mayo, VA, and PU models were compared. Analyses were performed using commercially available software packages (SPSS for Windows version 17.0 and Medcalc for Windows version 11.4.2.0). Values of P＜0.05 were deemed to indicate statistical significance. ROC curves were created, and the areas under the curve (AUCs) were calculated and compared using the Medcalc software (version 11.4.2.0).

According to the predicted probabilities, we divided the entire dataset into five equal groups. By plotting the median probability of each group, a calibration curve that described the distinction between the predicted probability and observed frequency of malignancy for that group was created.

Results

Pathological results

Of all SPN patients, 29 (18.8%) cases were benign, including 5 tuberculomas, 6 hamartomas, 11 infammatory pseudotumors, 3 pulmonary sclerosing hemangiomas, 2 atypical adenomatous hyperplasia, 1 fibrosis nodule, and 1 lung adenoma. One hundred and twenty-five (81.2%) cases were malignant, including 104 adenocarcinomas, 14 squamous carcinomas, 2 small-cell carcinomas, 4 large-cell carcinomas, and 1 neuroendocrine carcinoma.

Comparison of the malignant and benign groups

Validation of the Mayo model

Based on the applicable conditions of the Mayo model, weexcluded six patients who had a previous cancer history of more than 5 years. Of the remaining 154 patients, we constructed the ROC curve and calculated the AUC to be 0.753 [95% confdence interval (CI): 0.650-0.857].

Table 1Clinical data of the malignant and benign groups

Validation of the VA model

The applicable conditions of the VA model included patients with a previous cancer history of more than 5 years. These SPNs tend to be malignant and were suggested to undergo surgical resection, so we excluded these six cases. The area under the ROC curve was 0.728 (95% CI: 0.623-0.833). If the six cases were included, the area under the ROC curve was 0.736 (95% CI: 0.635-0.838). If only male patients were included, the area under the ROC curve of the 88 male patients was 0.707 (95% CI: 0.580-0.834). Ifonly smokers were included, the area under the ROC curve of the 49 smoker patients was 0.701 (95% CI: 0.513-0.889).

Figure 1Receiver operating characteristics (ROC) curves of the three models. VA, Department of Veterans Affairs; PU, Peking University.

Validation of the PU model

According to the applicable conditions of the PU model, we also excluded six patients who had a previous cancer history of more than 5 years. For the remaining 154 patients, the area under the ROC curve was 0.800 (95% CI: 0.708-0.891).

Comparison of areas under the ROC curve

采用感官分析比较低度与高度五粮液风格特征的结果表明，低度五粮液具有色泽透明、香气优雅、酒体醇和协调、绵甜、爽净等特点，其酒体风格基本保持了高度五粮液的风格。色谱骨架成分的分析结果表明，低度与高度五粮液中骨架香味成分的相对比例较一致，低度白酒在己酸乙酯等骨架成分上含量低于高度五粮液。Osme分析表明，果香类骨架物质（己酸乙酯、丁酸乙酯等）的香气强度顺序基本一致，能使低度与高度五粮液在气味风格上保持一致，而酸类物质和芳香族类物质在低度与高度五粮液中优势物质的排列顺序不同，尤其是部分呈显著药材香和甜香味物质（苯乙酸乙酯、4-甲基愈创木酚等）则可能是低度五粮液香气特征区别于高度五粮液的原因。

Using the Medcalc software, we compared the AUCs among the three models. The p values were 0.82 between the Mayo and VA models, 0.49 between the Mayo and PU models, and 0.35 between the VA and PU models. Therefore, the AUCs of the three models did not demonstrate statistically significant differences. Figure 1 shows the ROC curves of the three models.

Predicted probabilities of the three models

Table 2 shows the interval distribution of each model according to the predicted results, recommended strategies, and pathological diagnosis.

Calibration curves of the three models

The calibration curves for the Mayo, VA and PU model, describing the distinction between the predicted probabilityand observed frequency of malignancy, is shown in Figures 2-4.

Table 2Predicted results and interval distribution of the three models

Discussion

The Mayo, VA, and PU models were validated in terms of the accuracy of SPN diagnosis by collecting clinical and imaging information. Our study reveals that the accuracies of the three models were not signifcantly different.

The Mayo model, published in 1997, was the first to focus on SPN. Between 1984 and 1986, Swensen et al. at the Mayo Clinic retrospectively reviewed the medical records and imaging tests of 629 SPN patients (51% male) newly discovered with a malignancy rate of 23%. This group of patients lived locally for at least 20 years; thus, limitations in the data existed. However, in our results, the area under the ROC curve of the Mayo model was 0.753, which was higher than that of the VA model. With a classifed process of three intervals, the false-positive (8%) and false-negative (0%) rates were low, providing reliable suggestions to clinicians. The calibration curve showed that the Mayo model underestimated the probability of malignancy in lowrisk patients.

Figure 2The calibration curve for the Mayo model.

Figure 3The calibration curve for the VA model. VA, Department of Veterans Affairs.

The VA model was established in 2007 using the records of 375 SPN cases were collected from the Department of Veterans Affairs. The data were unbalanced because 98% of patients were male, and 94% had a smoking history. Thus, we found that smoking history has heavy weightings in the VA model, indicating that if a patient does not smoke, the calculated probability of the model may change considerably. This explains why the VA model showed poor accuracy. Additionally, a large proportion (68.9%) of patients was in the ＜20% probability interval, producing malignant pathological results. The area under the ROC curve did not improve in the male or smoker group. The calibration curve also showed that the Mayo model underestimated the probability of malignancy in low-risk patients, different from the actual situation.

Figure 4The calibration curve for the PU model. PU, Peking University.

The PU model was presented in 2011, and comprised 371 SPN patients who underwent surgery at the People’s Hospital of PU. The area under the ROC curve (0.800) was higher than those of the other two models; however, with the judgment of a spot, half of the patients had undergone unnecessary surgeries. We suggested dividing the entire region into three probability intervals; i.e., malignant, benign, and further examination. The calibration curve showed that the fve groups of the PU model matched more closely the true circumstance than the other two models, excluding the second group. High-resolution CT should be performed on patients in the PU model because imaging characteristics are necessary factors in the calculation.

Although medical records with inferior radiological technology were collected during 1984 to 1986 in Mayo model, it revealed high predictive value in this study. We suggested that Mayo model could be applied in some undeveloped area or some hospital without high-resolution CT scans. However, PU model, which collected medical records including high-resolution CT scans, showed the highest AUC and the best calibration curve. Accordingly, PU model set higher requirements to the doctors and equipments in medical image center.

As a high percentage (81.2%) of malignant SPNpatients existed in our population, the calibration curve did not perform well. Because the observed probability of malignancy of the fve groups was higher than the predicted values, particularly for low-risk patients, the curve tended to underestimate malignant probability.

Schultz and colleagues (7) calculated an AUC of 0.80 (95% CI: 0.72-0.88) for the Mayo model and 0.73 (95% CI: 0.64-0.82) for the VA model. Data characteristics of this study were similar to those of the VA model, with 96% male patients and 89% smokers. The difference between the two models was not statistically signifcant.

In conclusion, clinicians should use an SPN predictive model based on the local epidemiology of lung cancer. Another study showed that if the SPN malignancy rate of the hospital was similar to that of one of the selected models, a higher diagnostic accuracy would result (8). The AUC of the Peking model was the highest of the three models, and the calibration curve fit better with our results. Thus, the PU model should be applied in the south of China.

In conclusion, clinicians should choose an SPN predictive model based on the local epidemiology of lung cancer. And, the PU model should be applied in the China and similar area.

Acknowledgements

Guarantor: Xue-Ning Yang takes responsibility for the content of the manuscript, including the data and analysis. Authors’ contributions: Hong-Hong Yan directed and confirmed the statistical methods. Xuan Zhang collected data and wrote the manuscript. Jun-Tao Lin, Ze-Hua Wu, Jia Liu and Xu-Wei Cao collected data. Xue-Ning Yang designed the study and confrmed that the study objectives and procedures are honestly disclosed.

Funding: This research was granted by the initiative research scheme for college student, Guangdong, China (No. 1212110046).

Disclosure: The authors declare no confict of interest.

1. Tan BB, Flaherty KR, Kazerooni EA, et al. The solitary pulmonary nodule. Chest 2003;123:89S-96S.

2. Ost D, Fein AM, Feinsilver SH. Clinical practice. The solitary pulmonary nodule. N Engl J Med 2003;348:2535-42.

3. Gould MK, Sanders GD, Barnett PG, et al. Costeffectiveness of alternative management strategies for patients with solitary pulmonary nodules. Ann Intern Med 2003;138:724-35.

4. Swensen SJ, Silverstein MD, Ilstrup DM, et al. The probability of malignancy in solitary pulmonary nodules. Application to small radiologically indeterminate nodules. Arch Intern Med 1997;157:849-55.

5. Schultz EM, Sanders GD, Trotter PR, et al. Validation of two models to estimate the probability of malignancy in patients with solitary pulmonary nodules. Thorax 2008;63:335-41.

6. Li Y, Chen KZ, Wang J. Development and validation of a clinical prediction model to estimate the probability of malignancy in solitary pulmonary nodules in Chinese people. Clin Lung Cancer 2011;12:313-9.

7. Schultz EM, Sanders GD, Trotter PR, et al. Validation of two models to estimate the probability of malignancy in patients with solitary pulmonary nodules. Thorax 2008;63:335-41.

8. Isbell JM, Deppen S, Putnam JB Jr, et al. Existing general population models inaccurately predict lung cancer risk in patients referred for surgical evaluation. Ann Thorac Surg 2011;91:227-33; discussion 233.

Cite this article as:Zhang X, Yan HH, Lin JT, Wu ZH, Liu J, Cao XW, Yang XN. Comparison of three mathematical prediction models in patients with a solitary pulmonary nodule. Chin J Cancer Res 2014;26(6):647-652. doi: 10.3978/ j.issn.1000-9604.2014.11.02

10.3978/j.issn.1000-9604.2014.11.02

Submitted Oct 05, 2014. Accepted for publication Nov 12, 2014.