Validity of studies suggesting preoperative chemotherapy for resectable thoracic esophageal cancer: A critical appraisal of randomized trials

Giulia Manzini, Ursula Klotz, Doris Henne-Bruns, Michael Kremer

Giulia Manzini, Ursula Klotz, Doris Henne-Bruns, Michael Kremer, Department of General and Visceral Surgery, University of Ulm, Ulm 89081, Germany

Michael Kremer, Department of General and Visceral Surgery, Hospital of Aarau, Aarau 5000,Switzerland

Abstract

Key words: Esophageal cancer; Assessment of validity; Meta-analysis; CONSORT;Overall survival

INTRODUCTION

Esophageal cancer is the eighth most common cancer in the world and the sixth most common cause of death from cancer with an overall ratio of mortality of 0.88[1].Although it accounts for only 3.2% of all cancers, the incidence of esophageal cancer is increasing with an incidence of 572/100000 new cases/year in 2018[2].

Surgery is the treatment of choice for localized esophageal cancer[3-4]with a potential to provide loco-regional control, as well as long-term survival[5]. Curative resection is possible in only 15% to 39% of the cases[6-9]. Surgery is the only curative treatment, but it alone often fails to overcome the natural history of the disease owing to the presence of occult micrometastases, and fatal distant and loco-regional disease relapse is common. Median survival after esophagectomy with curative intention is 15 to 18 mo with a 5-year survival rate of 20% to 25%[5]. Therefore, clinicians are now inclined to use some form of multidisciplinary treatment including surgery as a standard of care for locally advanced esophageal cancer, which is defined as disease restricted to the esophagus or resectable periesophageal tissue (T2-T4) and/or lymphnode involvement (N1-N3) in the absence of distant metastasis[10].

The optimal multimodality treatment is still controversial. Potential contentious issues exist regarding the (1) ideal preoperative, perioperative or postoperative approach and (2) ideal combination of radiotherapy (RTx), chemotherapy (CTx) or concurrent chemoradiation. Various randomized and non-randomized trials and several meta-analyses have been conducted to address this topic, but established standard guidelines still vary considerably or even fail to propose a specific treatment regime[11]. Preoperative (radio-)chemotherapy aims to exterminate micro-metastases,enhance resectability by down-staging the tumour, improve loco-regional control and provide relief of dysphagia[11,12].

Several studies have investigated whether preoperative CTx leads to improved cure rates, but reports remain conflicting. The initial Cochrane review of preoperative CTx for resectable esophageal cancer[13]concluded that no survival advantage was associated with CTx. The same result was found by Urschelet al[14]after inclusion of 11 randomized trials in a meta-analysis. Ychouet al[15], Boonstraet al[16]and MRC Allumet al[17]subsequently reported a survival benefit for patients receiving neoadjuvant CTx.After inclusion of these last three studies, the updated Cochrane Review and metaanalysis by Kidaneet al[18]on the same topic found an improvement in overall survival(OS) [hazard ratio (HR): 0.88; 95% confidence interval (CI): 0.80 to 0.96] for patients receiving preoperative CTx. A total of ten randomized controlled studies with OS as the primary endpoint were included in this meta-analysis.

The aim of our study was to assess the validity of the studies by Ychouet al[15],Boonstraet al[16]and MRC Allumet al[17]included in the updated Cochrane Review and the meta-analysis by Kidaneet al[18], which confirmed the benefit of preoperative CTx on survival for resectable thoracic esophageal cancer with the intention to invite everyone to critically interpret not only the results, but also the methodology by which the results were achieved. We performed a variety of meta-analyses excluding or including studies depending on their validity and attributed power and discuss those findings in regard to current recommendations of esophageal carcinoma guidelines.

MATERIALS AND METHODS

The meta-analysis by Kidaneet al[18]included a total of ten studies. Four (40%) (Ychouet al[15], Boonstraet al[16], MRC Allumet al[17], and Lawet al[19]) found a statistically significant advantage in survival in patients after preoperative CTx for resectable thoracic esophageal cancer (HR: ＜ 1 with a significant 95%CI). All the other six included studies (60%) were not statistically significant[20-25].

In the first part of the results section we assessed the validity of the three most powerful studies included in the Cochrane review by Kidaneet al[18], which found a statistically significant advantage in survival in patients receiving preoperative CTx before resection for thoracic esophageal cancer. These studies were those of Ychouet al[15], Boonstraet al[16]and MRC Allumet al[17].

In the second part of the results section, we performed a new meta-analysis without these aforementioned three studies. Among the three analysed studies, Boonstraet al[16]had the higher validity, so we performed another meta-analysis assuming that this study is valid enough to be included in the meta-analysis.

Finally, we present the results of the meta-analysis excluding the four statistically significant studies confirming the survival advantage for patients treated with preoperative CTx. In this last case, only statistically non-significant studies were included in the meta-analysis.

Selection of the studies and assessment of their validity

We used the same methodology as described in our previous publication[26]to analyse the validity of the Cochrane review. From the several endpoints investigated in the Cochrane review by Kidaneet al[18], we identified OS as a major endpoint of interest.Among the ten studies identified by the authors of the Cochrane review investigating OS, we selected the three most powerful studies as weighted by the review’s authors which support the advantage of preoperative CTx: Ychouet al[15], Boonstraet al[16]and MRC Allumet al[17]. The weights assigned to these three studies by the authors of the systematic review according to their sample size, precision of the estimates and width of the confidence intervals were 24.5%, 24.1% and 20.5%, respectively. We then assessed the validity of these studies using the CONSORT checklist 2010[27], which is a validated instrument for the evaluation of randomized controlled trials (RCTs) and contains a total of 37 items. The checklist with all items and their precise description is available in the Appendix of our previous publication[26]. We then asked whether the positive result in the Cochrane review is supported by sufficient validity. Figure 1 illustrates our methodology. Two independent review authors (UK and GM) assessed the validity of each of the three publications.

中国在线自助游市场持续探索多样化盈利模式来增加营收，目前主要包括企业端（TOB）和用户端（TOC）两种营收模式。企业端（TOB）盈利模式主要包括佣金、付费广告、差价和竞价排名，其中差价为OTA（在线旅行社）核心营收来源，而佣金为非OTA（第三方平台）核心营收来源。从平均佣金率范围区间来看，现阶段酒店佣金率最高为10-15%，其次景区门票及索道8-12%，再次才是交通运输（以机票为主）为1-4%。用户端（TOC）盈利模式则主要包括自营产品销售、金融服务、定制服务和会员费。

Meta-analysis

We repeated the meta-analysis without the three analysed studies (n= 7) and compared the result with the original meta-analysis comprising ten studies. Since Boonstraet al[16]has the higher validity among the analysed studies, we then conducted a second meta-analysis only excluding Ychouet al[15]and MRC Allumet al[17]. In a next step, we assumed that all single studies with a statistically significant benefit of preoperative CTx for thoracic resectable esophageal cancer (n= 4) were not valid enough and performed a second meta-analysis with the remaining six studies.The results were compared with the original meta-analysis (n= 10 studies). The metaanalyses were performed with R, version 3.2.0, with the package “meta” (http://www.r-project.org/foundation).

Figure 1 Four steps to the analysis of validity of a systematic review according to our previous work[26].

RESULTS

Assessment of the validity of the studies

Table 1 presents a summary of the three analysed papers. The results are reported for each of the three included studies. Table 2 summarizes all the items present in the CONSORT checklist showing how the studies dealt with them. In this section, we describe the problems of each study. Eleven of the 34 validity criteria (32.4%) were not met in the study by Ychouet al[15]. Three items were not applicable. The randomisation occurred by phone call through a centralized randomisation system, and then the assignment was stratified according to centre, performance status, and tumour site using the minimisation procedure. Due to the use of the minimisation method,allocation concealment was not maintained. Blinding was not possible in this study,as the control group did not receive any preoperative treatment. Inclusion of untreated controls limits the interpretation of the study. Specifically, the difference between the intervention and control group may be caused by a non-specific effect,such as a placebo effect. Moreover, 50% of the patients in the intervention group also received postoperative CTx. Regarding sample size, in the methods section the authors described that 250 patients (178 deaths) were required to achieve the needed power. The trial was closed earlier due to difficulties in patient recruitment. At the closure time, a total of 224 patients (156 deaths) had been included, raising the question of whether the power was sufficient. Moreover, patients with stomach adenocarcinoma were also included in the study at a later time after changing the inclusion criteria. Taken together, these issues lead to insufficient validity of the report; therefore the described effect cannot be considered as clinically relevant.

In Boonstraet al[16], we identified poor validity in 8 of the 33 validity criteria (24.2%).Four items were not applicable. Again, as in the previous study, the use of untreated controls limits the interpretation of the study. Blinding was not possible in this work either, as the control group did not receive any preoperative treatment. Central randomisation was performed, but the process is not clearly described. Therefore, it is not possible to ascertain whether allocation concealment was maintained or not.Additionally, random assignment was stratified by age, gender, weight loss and tumour length. As also pointed out by the authors of the Cochrane review, it is unclear whether an intention-to-treat (ITT) analysis had been performed, as information on withdrawals was missing or unclear.

Table 1 Summary of the three analysed studies

As the validity of the report is not sufficient, the described effect cannot be considered as clinically relevant.

MRC Allumet al[17]described the long-term results of a previously published study by the same group in 2002. If information was not found in the last studies, we checked if the needed information was available in the first publication[28]. Taking this into consideration, 11 of the 33 validity criteria were not met (33.3%) by MRC Allumet al[17]. Four items were not applicable. As in the previous study, the use of untreated controls limits the interpretation of the study. Blinding was also not possible because the control group did not receive any preoperative treatment. Due to the use of the minimisation method, allocation concealment is not maintained. A power calculation is missing.

In the previous publication by the same group in 2002[28], the sponsor appointed the writing committee, which interpreted data, wrote the report and submitted it for publication. The risk profiles of the two groups are slightly different with a certain probability of unbalanced risk distribution in favour of the intervention group regarding age and degree of dysphagia.

As the validity of the report is not sufficient, the described effect cannot be considered as clinically relevant.

Meta-analyses

Figure 2 shows the result of the meta-analysis when the three analysed studies were excluded. A total of seven studies were included. One study (Lawet al[19]) showed a positive and statistically significant result in favour of the use of preoperative CTx before resection of thoracic esophageal cancer. Six of the included studies were not statistically significant by themselves. The new meta-analysis estimate had a HR of 0.94 with a 95%CI (0.81; 1.09) under assumption of a fixed-effect model and a HR of 0.92 with a 95%CI (0.75; 1.13) under assumption of a mixed-effect model. Regardless of the assumed model, the new estimate does not confirm the advantage of preoperative CTx for resectable thoracic esophageal cancer. The estimate of the original meta-analysis was 0.88 with a 95%CI (0.80; 0.96). The exclusion of the three studies completely changed the result of the meta-analysis. In Boonstraet al[16], only 24.2% of the items on the CONSORT checklist were inappropriately answered, so we assumed that the validity of this study was enough to be included in the metaanalysis. We performed a new meta-analysis excluding only Ychouet al[15]and MRC Allumet al[17](Figure 3). We found a HR of 0.90 with a 95%CI (0.81; 1.00) under assumption of a fixed-effect model and a HR of 0.90 with a 95%CI (0.78; 1.05) under assumption of a mixed-effect model. Again, regardless of the assumed model, the new estimate does not confirm the advantage of preoperative CTx for resectable thoracic esophageal cancer.

Table 2 Assessment of validity of the three analysed studies according to the CONSORT checklist (REF)

Finally, we performed a second meta-analysis (Figure 4) also excluding Lawet al[19],which found a positive and statistically significant result as well. After the exclusion of all four studies with positive and statistically significant results, the new metaanalysis consisted of only six statistically non-significant studies. The new metaanalysis estimate was HR 1.04 with a 95%CI (0.88; 1.22), confirming the lack of a survival advantage for patients undergoing preoperative CTx before resection of the thoracic esophageal cancer.

Figure 2 Meta-analysis of seven studies after excluding the three analysed studies.

DISCUSSION

In the present manuscript, we assessed the validity of three studies included in the meta-analysis by Kidaneet al[18], which supports the results of improved survival in patients treated with preoperative CTx for resectable thoracic esophageal cancer. It is important to identify possible bias in the three studies which support the result of the meta-analysis because bias jeopardizes validity. We demonstrated that these three studies are not valid enough to be included in a Cochrane review. When excluded from the meta-analysis, the overall result of the meta-analysis is no longer significant.

We will first illustrate the problems we discovered in the three mathematically most influential studies supporting the conclusions and, in a second step, discuss our findings after performing the new meta-analyses.

Common problems in all studies

The lack of a placebo-controlled and blinded study affects the validity of the three studies and, consequently, the validity of the review. As discussed in our previous work[26], without a placebo control, it is impossible to differentiate between specific pharmacological and placebo effects. A placebo effect is defined as the “…response of a subject to a substance or any procedure known to be without specific therapeutic effect for the condition being treated[29].” Several studies demonstrated that perceptual characteristics of drugs[30], the route of administration[31], laboratory tests[32],diagnosis[33]and the doctor-patient relationship play an important role in the outcome of an illness[34-37]. Information regarding treatment or no treatment alone is sufficient to elicit a placebo effect[38]. Moreover, patients’ and doctors’ preferences could also have influenced the results in an open study[39]. Patients assigned to the control group feel disadvantaged because they expect to be treated. Furthermore, when there is no concealment of treatment allocation, the randomisation procedure is compromised because of conscious or subconscious bias[40]. It is important to perform an ITT analysis to maintain the balance distribution of risk factors between groups achieved by a randomisation procedure. A correct ITT analysis was only conducted in the studies by MRC Allumet al[17]and Ychouet al[15]. These aspects collectively affect the validity of the reports and, therefore, the described effects cannot be considered as clinically relevant.

Specific problems of the study by Ychou et al[15]

In the study by Ychouet al[15], a minimisation method is used. Minimisation[41-44], a type of dynamic allocation, is gaining popularity especially in clinical cancer trials. In this design, the new subject’s treatment assignment is determined by evaluating the potential covariate imbalance that would result if he or she were assigned to the treatment or to the control group[45]. Minimisation aims at achieving balance over a large number of prespecified prognostic factors simultaneously. We raise concerns over this design, as it compromises adequate generation of an allocation sequence and concealment in this study. Investigators using minimisation can actually determine the group to which a prospective subject would be allocated and then decide whether this is positive or negative in terms of creating an imbalance in some key predictor of outcome not considered in the imbalance function. Despite adding randomisation, so that the treatment that minimises the imbalance function for a given patient is not necessarily allocated to that patient, there is a high probability of this being the case[46].The European Medicines Agency’s Committee[47]states that “dynamic allocation is strongly discouraged”.

Figure 3 Meta-analysis of eight studies after excluding the studies by Ychou and MRC Allum.

Specific problems of the study by Boonstra et al[16]

In this study, as in the study by Ychouet al[15], the randomisation process is not exhaustively described; they only mentioned that a central randomisation took place.A description of the randomisation process is completely lacking. Aside from this problem, which is extremely relevant, we find that this study was conducted well in comparison to the other two.

Specific problems of the study by MRC Allum et al[17]

This study reports long follow-up results of a previously published study by the same authors (2002)[28]. As in the study by Ychouet al[15], minimisation was used, raising the same concerns as previously described. A power calculation is completely missing.Finally, a sponsor-related conflict of interest was identified by our analysis.

As recently shown by Shnieret al[48], financial conflicts of interest and relationships between guideline authors and drug companies are common and represent a source of bias in studies. As authoritative value is assigned to guidelines, it is important to develop formal policies to limit the potential influence of any conflict of interest on guideline recommendations. This is the only way to improve the quality of medical publications. Only valid studies are reliable studies. For an expert pool aiming to publish guidelines, it is necessary to scrutinise the validity of single studies and of meta-analyses as well, as low-quality studies can lead to a distortion of the summaryeffect estimate[49].

In the second part of our analysis we performed the meta-analysis first without the three analysed studies and showed that the result of the meta-analysis is no longer significant. This result coincides with previous big studies and the original metaanalysis by Malthaneret al[13]. Moreover, as we find that the study by Boonstraet al[16]was quite well done in comparison to the other two, we performed a new metaanalysis excluding only the studies of Ychouet al[15]and MRC Allumet al[17]. The estimate also showed no benefit of preoperative CTx before surgical resection. As expected, when all studies with positive results are eliminated from the meta-analysis,the estimate is not significant.

Implications for practice

According to the results of the Cochrane review, preoperative CTx should be used for patients with resectable thoracic esophageal cancer. However, it is important to note that some of the included trials contain limitations so that definitive assessments of this topic should be delayed until future trials are properly developed. The three analysed studies that were chosen because of their attributed weights are not sufficiently valid to be included in a meta-analysis, which is also true for most of the other studies included.

Figure 4 Meta-analysis of six studies after excluding all studies which found a statistically significant survival advantage in the experimental group.

Despite finding several inconsistencies and substantial deficits in the included high-power studies, the aim of this work is not primarily to identify the best therapeutic treatment for esophageal cancer, but to increase awareness of the quality of studies and their impact on medical treatment when used in meta-analyses or Cochrane reviews. Especially studies that were performed before implementation of the CONSORT checklist show a variety of inconsistencies that would exclude publication according to current quality standards. High-quality RCTs decrease the risk of inherent bias and therefore receive higher attributed weight in meta-analyses.The inclusion of several low-power studies with serious deficits can overpower well conducted studies and change the outcome.

The analysed Cochrane review was published in 2015; only three included studies were performed after 2009, but seven before 2001, some even dating back to before the 1990s. At that time, no standardised reporting procedure, like the CONSORT checklist, existed. Therefore, the findings are quite heterogeneous. The three most powerful studies were the last ones published and still show a substantial lack in standardisation according to the CONSORT checklist, which was first published in 1996 and revised in 2001 and 2010.

As the incidence of esophageal carcinoma is relatively low, studies often include adenocarinoma and squamous-cell carcinoma without discrimination. Even worse, in some of the studies adjuvant treatment was not only CTx, but sometimes also RCTx for squamous-cell carcinoma. Both inherently different carcinoma types with different neoadjuvant treatment regimens were included in a single group. To analyse the role of neoadjuvant CTx in this context, two groups needed to be established: RTx alonevsRCTx as neoadjuvant therapy as performed by Herskovicet al[50]. In this paper,adenocarcinoma and squamous-cell carcinoma of the esophagus were also put into one group.

Multimodale therapy in patients with esophageal cancer is now the standard treatment in most centres today and is recommended in several national guidelines[51-52].

In Germany, S3 guidelines for esophageal carcinoma were updated in 2018[51].Several newer publications, usually multicentric randomised controlled studies, were taken into account.

The evaluated Cochrane review by Kidane is not mentioned in the current German S3 guideline for the standardised treatment of esophageal carcinoma. However, the analysed studies by Ychouet al[15], Boonstraet al[16]and MRC Allumet al[17]with observed inconsistencies are mentioned and included. Thanks to the authors of the German S3 guideline, the current data is critically presented and not all study results are included in the recommendation for standardised treatment: “In squamous cell carcinoma, no consistent increase in survival after CTx alone - despite the positive study by Boonstra - could be observed by metaanalyses.” (page 101 German S3 guidelines AWMF-Registernummer: 021/023OL).

In conclusion, multimodal therapy of advanced esophageal carcinoma represents the current gold standard for treatment. We observed several deficits of the analysed studies in the Cochrane review by Kidane. Interestingly, this review was not taken into account in the current german S3 guideline for treatment of esophageal carcinoma, and the analyzed single studies are there critically reviewed and set in context with similar research papers. Well performed (multicentric) randomized controlled studies are needed to be analysed together in a meta-analyse. High-quality single studies are required, as they determine the outcome of meta-analyses that can influence the recommendations of national guidelines.