Title: Impact of peer review in reducing uncertainty in the definition of the lung target volume amongst trainee oncologists.

Abstract:

Purpose:

Evaluate the impact of peer review and contouring workshops on reducing uncertainty in target volume delineation for lung cancer radiotherapy.

Methods:

Data from two lung cancer target volume delineation courses were analysed. A total of 22 trainees in clinical oncology trainees working across different UK centres attended these courses with priori experience in lung cancer radiotherapy. The courses were made up of short presentations and contouring practice sessions. The participants were divided into two groups and asked to first individually (IND) delineate and then to individually peer review (IPR) the contours of another participant. The contours were discussed with an expert panel consisting of 2 consultant clinical oncologists and a consultant radiologist. Contours were analysed quantitatively by measuring the volume and local distance standard deviation (localSD) from the reference expert consensus contour and qualitatively through visual analysis. Feedback from the participants was obtained through the use of a questionnaire.

Results:

All participants applied minor editing to the contours during IPR leading to a non-statistically significant reduction in the mean delineated volume (IND=140.92cc, IPR=125.26cc,p=0.211). The overall interobserver variation was similar with a localSD of 0.33cm and 0.38cm for the IND and IPR respectively (p=0.848). Six participants (29%) performed correct major changes by either including tumour or excluding healthy tissue. One participant (5%) performed an incorrect edit by excluding parts of the tumour while another observer failed to identify a major contour error. The participants’ level of confidence in target volume delineation increased following the course and identified the discussions with radiologist and colleagues as the most important highlights of the course.

Conclusion:

IPR could improve target volume delineations quality amongst trainee oncologists by identifying most major contour errors. However, errors were also introduced after individual peer review suggesting the need to further discuss major changes with a multidisciplinary team.

1 | Page

Introduction:

Large interobserver variation in the definition of the gross target volume (GTV) has been reported in numerous studies for all cancer types but in particularly for Non-Small Cell Lung cancer (NSCLC)[1–4]. These tumours are often surrounded by soft tissue and interstitial lung changes making it difficult to distinguish tumour boundaries from normal tissue. Failure to accurately define the GTV will influence the accuracy of all other downstream processes in the radiotherapy chain [5]. Various interventions have been proposed to reduce the interobserver variation in the definition of the lung GTV including; training and the use of established protocols [6–10]. Knowledge of normal anatomy is extremely important in order to be able to detect pathology and therefore several anatomical atlases were developed to facilitate the delineation of the primary GTV (GTVp) and lymph nodes (GTVln) [11–13]. However, although training and protocols have been found to reduce interobserver variation in the delineation of the GTV, errors can still occur highlighting the need to review the target volumes. Studies have shown that peer review of GTV contours can identify errors in 17% to 45% of cases not only for lung cancer but also for other sites[14–17]. Gross errors in contouring can be considered a protocol violation. Non-compliance with radiotherapy protocols has been associated with increased risks of treatment failure and overall survival detriment as per randomised controlled trials and meta-analysis [18–20]. The overall survival in the UK for both early and advanced stage lung cancer cases has been found to be poor when compared to other western countries even though the populations evaluated had similar performance status[21,22]. Timely access to treatment and specialised staff was attributed to the poor outcomes highlighting the need to increase resources and improve multidisciplinary collaboration.

Despite the importance of this process, the definition of the GTV is not always comprehensively peer reviewed due to limitations of the institution quality assurance (QA) program [5]. In view of this, the Royal College of Radiologists (RCR) issued guidelines to establish minimum standards for peer review in target volume definition as part of the UK radiotherapy department’s QA processes [23]. The guidelines highlight the importance of integrating peer review as part of the training of clinical oncologists.

In 2018, we performed a survey of practice to evaluate peer review practices across 24 UK centres. Sixty-two per-cent of centres stated that due to time constraints and staff shortages not all cases were reviewed[24]. Another important finding of this survey was that the majority of centres (54%) do not have a formal in-house contouring training programme for trainees and therefore contouring skills are often acquired opportunistically in an informal and unstructured manner. Furthermore current training contouring programmes offered by professional bodies [8,10,25] have not included training on how to review contours and provide feedback to colleagues. To address this need, we initiated a delineation course in which peer review plays a central role.

The aim of this work was to evaluate the impact of peer reviewed learning in reducing uncertainty in target volume delineation for lung cancer based on the outcome of this course. The objectives of this study were to:

2 | Page

• Develop a one-day course on lung cancer contouring and peer review;• Enable the participants to make use of anatomical, clinical and multimodality diagnostic

imaging information to accurately define the lung GTV;• Evaluate the impact of peer review in reducing uncertainty in target volume definition for

lung cancer amongst clinical oncology trainees;• Evaluate the participants’ perception on the impact of the course and peer review on their

clinical practice.

Methods:

Course Development

Two one-day courses on lung cancer contouring were developed and run by a multidisciplinary team (MDT) including 2 consultant clinical oncologists, a radiology consultant, a radiographer and a medical physicist; all of which are working at The Christie NHS Foundation Trust (Manchester, UK). The courses were open to specialist trainee clinical oncologists working across UK radiotherapy centres.

The course was based on the RCR speciality training curriculum[26] and consisted of short lectures on thoracic anatomy and radiology, delineation and peer review guidelines followed by practice sessions including individual delineation (IND), individual peer review (IPR) practice sessions and group discussions to simulate a MDT peer review. The first course was conducted as a pilot study and the second course was provided to a new cohort of participants.

Case Selection

Two cases of patients diagnosed with stage 3 Non-Small Cell Lung Cancer (NSCLC) were selected (Refer to supplementary data 1 for the case descriptions). These cases were specifically included as they were deemed to be challenging to delineate due to extensive lymph node involvement and the tumour being surrounded by atelectasis as described in Fig.1 and Fig.2.

A planning respiratory correlated computed tomography (4DCT) was reconstructed using the Maximum intensity projection (MIP)[27] and Mid-position scan[28] and registered with a breath-hold CT scan with IV contrast acquired prior to the planning 4DCT, a diagnostic CT, 4DCT cine view and diagnostic 3D 18-fluorodeoxyglucose positron emission tomography (not in the treatment position) (FDG PET-CT). The registered images were loaded on the Big Brother[29] training contouring software developed at The Netherlands Cancer Institute and University of Manchester. During the practice sessions the participants were provided with a case description and instructions on how to use the delineation software and delineation guidelines [11,13] (Supplementary data 1). The delineations had to be performed on the MIP images according to local protocol, while all other images had to be used as reference to facilitate delineations.

Definition of reference contour

Two consultant clinical oncologists individually delineated one case each during the practice session. These delineations were then reviewed together with the radiologists until a consensus contour was

3 | Page

reached. The consensus contour was used as a reference gold standard to analyse the student contours. Uncertain regions were also identified as shown in Fig.1 and Fig.2.

Practice session set-up

For both courses the participants were divided into 2 groups whereby the participants in groups 1 and 2 were asked to individually delineate (IND) case 1 and case 2 respectively. For the second practice session the participants were then asked to individually peer review (IPR) the delineations of another participant in the other group as indicated in Table 1. The participants were asked to modify the delineation if required and to classify the reviewed contour as “major change required”, “minor change required” or uncertain (if not sure) as explained in Supplementary Data 2. The reviewed delineations were first discussed between the participants and then centrally as a group together with the teachers on a big screen.

Questionnaire

Feedback following the course was sought through the use of a questionnaire whereby the participants were asked to rate their level of confidence in performing target volume delineations and peer review before and after the course through the use of a Likert scale [30]. Participants were also asked to score the impact of presentations and practice sessions. Open-ended questions were included whereby participants were asked to indicate the highlights of the course and to provide recommendations for future courses. For instance, following feedback obtained from participants and teachers’ faculty, the structure of the second course was amended so that the radiology training was conducted before the first workshop.

Data Analysis

The IND and IPR contours were analysed quantitatively using the expert consensus contour as a gold standard by measuring the delineated volume, the volume inside (missing) and outside the expert consensus (excess) delineations, general conformity index (CI) and local standard deviation (localSD).

The CI was used to measure the degree of overlap with the expert consensus contour [31]. A CI of 1 indicates perfect overlap (i.e. no interobserver variation) while a CI of 0 indicates no overlap. The localSD was calculated by measuring the perpendicular distances at numerous arbitrary points from the expert contour to the participant’s delineations and the overall SD in the distance is calculated. The root mean distance square was used to calculate the average localSD per delineation (localSD)[29]. A larger localSD indicates a larger interobserver variation. The reference contour surface area was normalised to 100% and the mean distribution of SD over this surface was plotted (SD/area) to identify outliers.

The Wilcoxon-signed-rank test was used to identify any statistical differences in the volume, CI, local SD and SD/area curve between IND and IPR contours[30]. Statistical analysis was performed using SPSS version 20.0 and a p-value below 0.05 was deemed to be statistically significant.

All contours were also analysed qualitatively by the researcher by visually comparing them with the expert consensus contours. This method is often used in quality assurance of clinical trials in

4 | Page

conjunction with other quantitative metrics[32]. Any changes performed following IPR were classified based on the RCR peer review target volume delineation guidelines[23] as:

● “Minor change” for small modifications that enable better coverage or reduce healthy tissue

with the implication that original contour would still be clinically acceptable.

● “Major correct change” for modification performed to include tumour or pathological lymph

node and exclude healthy tissue.

● “Major incorrect change” for any editing performed that excluded tumour or included

healthy tissue.

● “Uncertain” for modification performed within the same region of uncertainty identified by

the experts. (The latter category is not part of the RCR recommendations and was included since the cases presented were particularly difficult to delineate and some areas remained uncertain even after expert discussion).

The questionnaire was analysed using descriptive statistics and the level of confidence in contouring before and after the courses compared using the Wilcoxon-sign-rank test. Open-ended questions were analysed using content analysis[30].

Results

Description of participants

A total of 12 and 10 participants participated in the first and second courses respectively. All participants had experience in lung cancer delineation with 45% and 40% of the participants in course 1 and course 2 respectively having less than 3 months experience in lung cancer delineation while the rest of the participants had more than 3 months.

Expert delineations

The final agreed consensus delineation had a volume of was 195.76cc for Case 1 and 68.15cc for Case 1 and Case 2 respectively. After comparing the consultant oncologist’s contour with the final consensus contour 5 difficult regions were identified for case 1 (Fig.1), including lymph node involvement at stations 5 and 6 (Both EBUS and FDG PET-CT positive), FDG PET-CT positive and EBUS negative left hiliar lymph node, a region with post- obstructive pneumonitis with low FDG uptake that, a suspicious lung nodule excluded following discussion with the radiologists and a region with tumour located close to the left pulmonary artery. For case 2, the variation between consultant and consensus contours was mainly attributed to a highly mobile tumour (2.3cm in the cranio-caudal direction) on the right lower lung lobe surrounded by lung collapse making it difficult to define the tumour boundary at this point. Following discussions with the radiologist this region was edited so as to identify the optimal trade-off between excluding lung collapse while ensuring that all the tumour motion was included (Fig.2).

5 | Page

Interobserver variation: Quantitative Analysis

Although not statistically significant, IPR lead to an overall reduction in the mean target volume delineated for both case 1 (-7.37% p=0.575) and case 2 (-13.36% p=0.239) (Table 2). A reduction in the target volume was observed in 64% of reviewed delineations (Fig.3). The excess volume was reduced following IPR by 43.45% and 8.07% for case 1 (p=0.203) and case 2 respectively (p=0.583). The missing volume remained approximately the same in case 1 following review but increased by 54% in case 2 (p=0.182) (Table 2).

When evaluating the images quantitatively there was a reduction in the interobserver variation following IPR for Case 1 as indicated by the higher CI (IND: 0.72, IPR: 0.76 p=0.059) and statistically significant lower localSD (IND: 0.32cm, IPR: 0.28cm p=0.046) (Table 2). A lower local SD was noted in 4 out of 10 cases evaluated and remained unchanged in the rest of the cases (Fig.4) However, the same pattern was not observed for the second case whereby IPR lead to an overall increase in the interobserver variation as indicated by the lower CI (IND: 0.48, IPR: 0.44 p=0.286) and statistically significant higher localSD (IND: 0.33cm and IPR 0.48cm p=0.040). When comparing the 2 courses using the SD/area histogram these differences were only statistically significant for the first course (Fig.4).

Interobserver Variation: Qualitative analysis

The editing performed following IPR is summarised in supplementary data 3. All participants performed some element of editing to the contours provided. Forty-three per-cent of the participants performed editing of the contours in regions identified as uncertain by the experts including the uncertain lung nodule and EBUS negative and PET positive left hilar lymph node in case 1 and region with lung collapse in case 2. However, the decision to include or exclude these regions was not consistent, following IPR (Fig.1 and 2).

Following IPR, 6 participants (29%) performed major correct changes by including a larger portion of the expert consensus tumour contour or by excluding the heart or infiltrative lung changes (Fig.1). However, not all the required changes were performed. For example, in the first course, one observer did not edit the contour to include tumour located close to the left pulmonary artery for Case 1, while another observer excluded the right GTV lymph node and parts of the primary GTV in Case 2. An interesting observation was that in 4 out of 6 clinically relevant major editing did not result in a major change in the overall localSD or CI probably because these changes only affected a small part of the tumour. An example of this is provided in supplementary data 4.

Participants Annotation Analysis

Seven participants (33%) did not provide a classification for their contouring editing while 48% (n=7), 10% and 10% (n=2) classified their editing as uncertain, minor and major changes respectively (Supplementary data 5). The annotations were in line with the expert consensus with the exception

6 | Page

of 3 (14%) of the annotations that were classified as uncertain by the participants and major according to the experts, while 1 (5%) participant classified a contour as major change required as opposed to uncertain according to the expert contour.

Questionnaire Analysis

Both courses improved the participants’ reported level of confidence in contouring in target volume delineation and peer review for lung cancer (Supplementary Data 6). The mean score for all lectures and practice session was 4 out of 5 for both courses indicating that the participants believe that the course will have a major impact on their clinical practice. Analysis of the free text questions showed that the participants had overall positive feedback on the course (Supplementary data 7). Improved knowledge on use and interpretation of FDG PET-CT, more awareness about peer review and radiology input were the 3 main highlights in each course. Suggested improvements in both courses included more individual expert feedback on own contours and to extend courses with more cases for other sites. In the second course one participant also indicated to include more discussion on margins, motion management and the use of auto-segmentation tools.

Discussion

Despite the importance of accuracy in target volume definition, the number of delineation courses offered by professional bodies is limited in the UK. Training and assessment is generally provided in the workplace which may vary significantly across centres [33]. Numerous international courses have been organised by international organisations [8,10,25]. These courses generally consist of a mixture of lectures with practical contouring sessions delivered either live or online but do not provide training on peer review. In view of this we have developed and delivered 2 contouring courses that incorporate for the first time training on how to perform peer review of target volumes in lung cancer. We aimed to evaluate the impact of peer learning on improving the quality of target volume definition and stress the importance of performing QA checks on the target volumes.

Two difficult cases of patients diagnosed with stage III lung cancer were provided. The level of interobserver variation was also high amongst experts with numerous regions of uncertainties being identified. Following discussions with the radiologists the oncologists were more confident to exclude uncertain regions such as the lung nodule in case 1 but remained cautious about excluding areas where it is difficult to distinguish the tumour boundary from surrounding normal tissue, as in the region with lung collapse in case 2 (Fig.1 and Fig.2). These issues generated a lot of discussion during the course and the participants appreciated the input of the radiologist in resolving radiological conundrums. An important aspect that came out from these discussions is that not all issues will be resolved. Knowledge on how to estimate the risk of lymph node involvement based on FDG PET-CT and biopsy results is required in order to be able to take a clinical decision [13].

During IPR all participants performed some minor editing to the contours. However, most of the changes performed were small and unlikely to have any clinical impact. It is important to acknowledge that this may not reflect clinical practice as the participants may have felt observed and pressurised to perform changes since this was an educational teaching session. Forty-three per-cent of the editing was performed within the same regions identified as uncertain by the experts indicating that peer review will not resolve all uncertainty issues. In addition, the consensus contour 7 | Page

might not necessarily be the optimal contour especially in challenging cases. Therefore, it is important to create a database with difficult cases that can be used for training purposes and to re-assess the consensus contours regularly in light of new knowledge and developments. Long term follow-up is also essential to identify tumour imaging features prone to local recurrence.

Nevertheless, IPR reduced the excess delineated volume in both cases leading to a contour closer to the expert consensus. However, the mean delineated volume was not always smaller than the expert contours as found in other studies[31,34] indicating that when uncertain most participants (8/11) preferred to delineate larger volumes as shown in case 2.

Major errors were correctly identified in 29% of the reviewed contours. Previous studies reported errors 17-18% of GTV delineations for lung cancer treated with SBRT that tend to be easier to delineate while a larger number of GTV errors were reported for more advanced cases and other tumour sites [5,14–16]. Not all errors were identified in our study while one observer introduced a major error following review. This suggests that any major changes must to be reviewed by a radiotherapy MDT including radiologist, physicists and oncologist before being approved by the treating oncologist so as to minimise the risk of errors. However, due to time constraint and lack of personnel some cases might not be reviewed thoroughly during MDT meetings [5,24]. IPR can be used to identify cases that would require more attention and therefore these should be discussed more closely during MDT meeting. Further research is required to identify the optimal peer review workflows.

It is important to acknowledge that the sample in our study was small and review was being performed by trainees rather than experienced oncologists. However, it is interesting to note that the majority of the participants’ annotations were in-line with the expert consensus. On the other hand, 33% of the participants did not provide a classification for their editing while 14% of the participants classified the editing as uncertain instead of major as per expert consensus. These findings suggest that the participants may have not been confident enough to provide feedback to peers in writing or the guidelines provided were not sufficiently clear. Continuous evaluation of peer review is necessary to improve delineation guidelines. Huo et al[35], reported that following the introduction of peer review the number of changes decreased over time. This implies that although initially peer review can be time consuming, meaningful collaboration can eventually result in a reduction in the number of errors eventually improving efficiency. Peer review organisation needs to encourage meaningful discussion without fear of reprimand whereby irrespective of seniority or hierarchy all participants can have an equal opportunity to provide input.

The interobserver variation was larger for the individual delineations in the first course while peer review had less impact in reducing interobserver variation in the second course. This could probably be due to the fact that the radiology training was provided after the individual delineation practice session in the first course. In addition, there was a larger proportion of more experienced trainees in the second course. Radiology training and PET-CT interpretation were identified by the participants as key highlights of the course. Dimigen et al, [36] reported that involving a radiologist in weekly MDT meetings resulted in a significant change in management in 6% of cases.

Another important aspect of the learning process is the receiving of feedback on one’s own contours. This can be provided both qualitatively and quantitatively [23]. In this study qualitative

8 | Page

feedback was provided by colleagues through the use of annotations and contours were discussed at the end of the course. No quantitative feedback was provided. Various software packages have been developed to provide quantitative feedback to participants by either measuring degree of overlap or distance in relation to a reference gold standard [37,38]. The limitation of this approach is that in practice there is no 100% gold standard and there will always be (clinically acceptable) differences between centres or professionals. Therefore for future training programmes we recommend that the gold standard should be defined by an expert panel based on local practice and clinical guidelines. Input from a radiologist is also essential in particularly for difficult cases. The larger the number of experts involved the more accurate is the reference contour. Hence clinical trial benchmark cases provide an optimal platform to develop these gold standards. On the other hand, for teaching purposes, apart from the need of defining the consensus contour, it is also important to define the acceptable deviation from the consensus. The latter is even more challenging to define. Currently this is being done through the use of quantitative metrics. However, the accuracy of these methods in quantifying this deviation varies and is often case dependent as also noted in our study. Furthermore the score obtained may not reflect the clinical impact of this deviation. This makes the development of an automated scoring system for examination, QA and feedback purposes quite challenging[33]. Therefore in our study we classified the contours as acceptable and unacceptable based on visual analysis of contours, a method often used in clinical trials[32]. However, we do acknowledge that this approach has a number of limitations. First of all this method is time consuming and cannot be used to provide automatic feedback to students. On the other hand, it can also be subject to interpretational differences[16]. This problem can potentially be overcome by studying the link between observer variation and clinical outcome.

An important limitation of the 2 courses was that the impact of this training on the participants’ clinical practice could not be tested. Another course could be organised whereby the participants could be asked to discuss a difficult case encountered during their practice to ensure continuous learning. In the future oncologists will be required to do less manual contouring and more editing of contours as auto-segmentation tools become more accurate particularly for organs at risk[39]. Therefore, future training should focus more on review and editing of contours rather than manual contouring. Ironically, the accuracy of new auto-segmentation algorithms such as deep learning is highly dependent on the quality of the contouring data used to train these algorithms[40] and thus peer review will be essential to develop a database of high quality contours that can be used to improve these algorithms.

Conclusion

Training on radiological normal and abnormal imaging anatomy is essential to reduce image interpretational differences and improve participants’ confidence in target volume delineation. Individual peer review can improve contouring by identifying regions of uncertainty and is recommended as a triage to identify cases that would benefit the most from a discussion within a MDT or with more experienced oncologists. The findings of the study indicate that although peer review overall improved the definition of the target volume, it will not resolve all uncertainties.

9 | Page

Furthermore, it can also introduce errors. The findings suggest that major changes need to be reviewed within a MDT and approved by the treating oncologist prior treatment.

Table 1: Description of the tasks performed by the different participants during the practice sessions.

10 | Page

Practicesession

Group Participants Case Task performed

1 1 1-6 1 Individual GTV delineation (IND)2 7-12 2 Individual GTV delineation (IND)

2 1 1-6 2 Individual peer review (IPR)e.g. observer 1 reviewed GTV delineation of observer 7

2 7-12 1 Individual peer review case (IPR) 1e.g. observer 7 reviewed GTV delineation of observer 1

Table 2: Comparison between the mean delineated volume, missing and excess volume, mean CI and mean local SD for the IND and IPR delineations.

11 | Page

Case IND IPR % change p-Value

Consensus Vol (cc) 1 195.76Mean Vol (cc) 1 195.76 181.34 -7.37% 0.575Missing Vol (cc) 1 32.87 32.83 -0.12% 0.646Excess Vol (cc) 1 33.35 18.86 -43.45% 0.203CI 1 0.72 0.76 5.56% 0.059LocalSD (cm) 1 0.32 0.28 -12.50% 0.046

Consensus Vol (cc) 68.15Mean Vol (cc) 2 90.65 78.54 -13.36% 0.239Missing Vol (cc) 2 16.61 25.58 54.00% 0.182Excess Vol (cc) 2 38.89 35.75 -8.07% 0.583CI 2 0.48 0.44 -8.33% 0.286LocalSD (cm) 2 0.33 0.48 45.45% 0.040

Consensus Vol (cc) 1+2 263.91Mean Vol (cc) 1+2 140.92 125.26 -11.11% 0.211Missing Vol (cc) 1+2 24.38 28.88 18.46% 0.426Excess Vol (cc) 1+2 36.24 28.07 -22.54% 0.249CI 1+2 0.59 0.58 -1.69% 0.507LocalSD (cm) 1+2 0.33 0.38 15.15% 0.848

Vol: Volume, CI: Conformity Index, LocalSD: Local distance standard deviation , IND: Individual Delineation, IPR: Individual Peer Review

Figures and Legends

Fig.1: FDG PET-CT and planning MIP images for case 1 showing a patient diagnosed with a pT3N2M0 NSCLC: The contours of the consultant oncologist and final expert consensus are compared with the individual (IND) and peer review (IPR) delineations performed by the participants in course 1 and 2. The % number of participants incorporating a specific region is indicated using a hot and cold map whereby red indicates 100% agreement and blue indicates low agreement.

Arrows (A) to (D) indicate difficult regions to delineate; (A) shows lymph node stations 5 and 6 with experts disagreeing on the medial extent of contour, (B) shows region with post obstructive pneumonitis with low uptake on FDG PET-CT, (C): suspicious lung nodule, (D): EBUS negative and PET-CT positive hiliar lymph node (E): tumour located close to the left pulmonary artery. Following IPR region (A) was sometimes extended to include lymph node 4L, region (B) was correctly excluded following review in both courses, region (C) remained uncertain with some participants including and some participants excluding, and region (E) was correctly extended to include the infiltration in

12 | Page

course 2 but not course 1. Arrow (F) shows how part of the heart was correctly excluded following IPR to match with the consensus delineations.

Fig.2: FDG PET-CT and planning MIP images for case 2 showing a patient diagnosed with a pT1 tumour in the left lower lung lobe (A) and a pT2a tumour in the right lower lung lobe (B) with EBUS positive right hiliar lymph node involvement (C). The contours of the consultant oncologist and final expert consensus are compared with the individual (IND) and peer review (IPR) delineations performed by the participants in course 1 and 2. The percentage number of participants incorporating a specific region is indicated using a hot and cold map whereby red indicates 100% agreement and blue indicates low agreement. Two difficult regions were identified including an area with low FDG uptake and possible lung collapse labelled (D) and difficulty to determine the inferior boundary of the right lower lobe tumour due to lung collapse, high tumour motion in this region and mis-registration with the FDG PET-CT (E).

13 | Page

Fig.3: Volume delineated during the individual delineation (IND) and following individual peer review (IPR) for course 1 (C1) and course 2 (C2) for each case respectively. The expert consensus volume is highlighted by the dashed line. Note that course 1 and course 2 had 12 and 10 participants respectively and these were not the same participants. Participant 6 in the first course did not submit the delineation for the IPR workshop.

14 | Page

Fig.4: Illustrates the localSD for the IND and IPR delineations for course 1 (C1) and course 2 (C2). The histogram illustrates the distribution of the localSD over the tumour surface area. As the curve is shifted towards the right the interobserver variation increases. For case 1 a decrease in the localSD is noted in 4 out of 10 pairs reviewed with the decrease being more prominent for the first course. For the second case the interobserver variation increased following IPR as indicated by the higher localSD for 8 out of the 11 pairs reviewed.

15 | Page

References

[1] Mercieca S, Belderbos JSA, De Jaeger K, Schinagl DAX, van der Voort Van Zijp N, Pomp J, et al. Interobserver variability in the delineation of the primary lung cancer and lymph nodes on different four-dimensional computed tomography reconstructions. Radiother Oncol 2017;126:325–32. doi:10.1016/j.radonc.2017.11.020.

[2] Fitton I, Steenbakkers RJHM, Gilhuijs K, Duppen JC, Nowak PJCM, van Herk M, et al. Impact of Anatomical Location on Value of CT–PET Co-Registration for Delineation of Lung Tumors. Int J Radiat Oncol 2008;70:1403–7. doi:http://dx.doi.org/10.1016/j.ijrobp.2007.08.063.

[3] Steenbakkers RJHM, Duppen JC, Fitton I, Deurloo KEI, Zijp LJ, Comans EFI, et al. Reduction of observer variation using matched CT-PET for lung cancer delineation: A three-dimensional analysis. Int J Radiat Oncol 2006;64:435–48. doi:10.1016/j.ijrobp.2005.06.034.

[4] Iqbal MS, Hothi A, Evans ES, Gilson D, Laws K, Saleem W, et al. An Evaluation Report on Radiotherapy Contouring Workshops During the Royal College of Radiologists’ Annual Meeting 2018. Clin Oncol 2019. doi:10.1016/j.clon.2019.09.060.

[5] Chang ATY, Tan LT, Duke S, Ng W-T. Challenges for Quality Assurance of Target Volume Delineation in Clinical Trials. Front Oncol 2017;7:221. doi:10.3389/fonc.2017.00221.

[6] Vinod S, Min M, Jameson M, C Holloway L. A review of interventions to reduce inter-observer variability in volume delineation in radiation oncology. J Med Imaging Radiat Oncol 2016;60:1–14. doi:10.1111/1754-9485.12462.

[7] van Baardwijk A, Bosmans G, Boersma L, Buijsen J, Wanders S, Hochstenbag M, et al. PET-CT–Based Auto-Contouring in Non–Small-Cell Lung Cancer Correlates With Pathology and Reduces Interobserver Variability in the Delineation of the Primary Tumor and Involved Nodal Volumes. Int J Radiat Oncol 2007;68:771–8. doi:10.1016/j.ijrobp.2006.12.067.

[8] Konert T, Vogel W V., Everitt S, MacManus MP, Thorwarth D, Fidarova E, et al. Multiple training interventions significantly improve reproducibility of PET/CT-based lung cancer radiotherapy target volume delineation using an IAEA study protocol. Radiother Oncol 2016;121:39–45. doi:10.1016/j.radonc.2016.09.002.

[9] Mercieca S, Belderbos J, van Baardwijk A, Delorme S, van Herk M. The impact of training and professional collaboration on the interobserver variation of lung cancer delineations: a multi-institutional study. Acta Oncol (Madr) 2018;58:1–9. doi:10.1080/0284186X.2018.1529422.

[10] De Bari B, Dahele M, Palmu M, Kaylor S, Schiappacasse L, Guckenberger M, et al. Short interactive workshops reduce variability in contouring treatment volumes for spine stereotactic body radiation therapy: Experience with the ESTRO FALCON programme and EduCaseTM training tool. Radiother Oncol 2017;127:150–3. doi:10.1016/j.radonc.2017.10.038.

[11] Smithius R, International Association for the Study of Lung Cancer (IASLC). The Radiology Assistant: Mediastinum lymph node map 2009. http://www.radiologyassistant.nl/en/p4646f1278c26f/mediastinum-lymph-node-map.html (accessed 29 September 2018).

[12] Radiotherapy Oncology Group. Lung Atlas: RTOG 1106 Target Atlas 2019. https://www.rtog.org/CoreLab/ContouringAtlases/LungAtlas.aspx (accessed 22 March 2018).

16 | Page

[13] Nestle U, De Ruysscher D, Ricardi U, Geets X, Belderbos J, Pöttgen C, et al. ESTRO ACROP guidelines for target volume definition in the treatment of locally advanced non-small cell lung cancer. Radiother Oncol 2018;127:1–5. doi:10.1016/j.radonc.2018.02.023.

[14] Rooney KP, McAleese J, Crockett C, Harney J, Eakin RL, Young VAL, et al. The Impact of Colleague Peer Review on the Radiotherapy Treatment Planning Process in the Radical Treatment of Lung Cancer. Clin Oncol 2015;27:514–8. doi:10.1016/j.clon.2015.05.010.

[15] Rasch C, Belderbos J, van Giersbergen A, De Kok I, Laura T, Boer M, et al. The Influence of a Multi-disciplinary Meeting for Quality Assurance on Target Delineation in Radiotherapy Treatment Preparation. Int J Radiat Oncol 2009;75:S452–3. doi:10.1016/j.ijrobp.2009.07.1035.

[16] Lo AC, Liu M, Chan E, Lund C, Truong PT, Loewen S, et al. The Impact of Peer Review of Volume Delineation in Stereotactic Body Radiation Therapy Planning for Primary Lung Cancer: A Multicenter Quality Assurance Study. J Thorac Oncol 2014;9:527–33. doi:10.1097/JTO.0000000000000119.

[17] Samuel R, Thomas E, Gilson D, Prestwich RJD. Quality Assurance Peer Review for Radiotherapy for Haematological Malignancies. Clin Oncol 2019;31:e1–8. doi:10.1016/j.clon.2019.06.010.

[18] Ohri N, Shen X, Dicker AP, Doyle LA, Harrison AS, Showalter TN. Radiotherapy protocol deviations and clinical outcomes: a meta-analysis of cooperative group clinical trials. J Natl Cancer Inst 2013;105:387–93. doi:10.1093/jnci/djt001.

[19] Peters LJ, O’Sullivan B, Giralt J, Fitzgerald TJ, Trotti A, Bernier J, et al. Critical Impact of Radiotherapy Protocol Compliance and Quality in the Treatment of Advanced Head and Neck Cancer: Results From TROG 02.02. J Clin Oncol 2010;28:2996–3001. doi:10.1200/JCO.2009.27.4498.

[20] Groom N, Wilson E, Faivre-Finn C. OA 01.05 Analysis of Radiotherapy Quality Assurance Data for the Convert Trial - Does Non-Compliance to Protocol Affect Survival? J Thorac Oncol 2017;12:S1745. doi:10.1016/J.JTHO.2017.09.325.

[21] Phillips I, Sandhu S, Lüchtenborg M, Harden S. Stereotactic Ablative Body Radiotherapy Versus Radical Radiotherapy: Comparing Real-World Outcomes in Stage I Lung Cancer. Clin Oncol 2019;31:681–7. doi:10.1016/j.clon.2019.07.013.

[22] Adizie JB, Khakwani A, Beckett P, Navani N, West D, Woolhouse I, et al. Stage III Non-small Cell Lung Cancer Management in England. Clin Oncol 2019;31:688–96. doi:10.1016/J.CLON.2019.07.020.

[23] The Royal College of Radiologists. Radiotherapy target volume definition and peer review RCR guidance 2017:1–35. https://www.rcr.ac.uk/system/files/publication/field_publication_files/bfco172_peer_review_outlining.pdf (accessed 15 May 2018).

[24] Mercieca S, Belderbos J, Gilson D, Dickson J, Pan S, van Herk M. Implementing the Royal College of Radiologists’ Radiotherapy Target Volume Definition and Peer Review Guidelines: More Still To Do? Clin Oncol (R Coll Radiol) 2019;31:706–10. doi:10.1016/j.clon.2019.07.021.

17 | Page

[25] American Society for Radiation Oncology (ASTRO). eContouring 2019. https://academy.astro.org/course-format/econtouring (accessed 25 August 2019).

[26] The Royal College of Radiologists. Clinical Oncology Specialty Training Curriculum 2016. https://www.rcr.ac.uk/clinical-oncology/specialty-training/curriculum/2007-clinical-oncology-specialty-training-curriculum (accessed 22 January 2019).

[27] Underberg RW, Lagerwaard FJ, Slotman BJ, Cuijpers JP, Senan S. Use of maximum intensity projections (MIP) for target volume generation in 4DCT scans for lung cancer. Int J Radiat Oncol Biol Phys 2005;63:253–60. doi:10.1016/j.ijrobp.2005.05.045.

[28] Wolthaus JWH, Schneider C, Sonke J-J, van Herk M, Belderbos JSA, Rossi MMG, et al. Mid-ventilation CT scan construction from four-dimensional respiration-correlated CT scans for radiotherapy planning of lung cancer patients. Int J Radiat Oncol 2006;65:1560–71. doi:http://dx.doi.org/10.1016/j.ijrobp.2006.04.031.

[29] Steenbakkers RJHM, Duppen JC, Fitton I, Deurloo KEI, Zijp L, Uitterhoeve ALJ, et al. Observer variation in target volume delineation of lung cancer related to radiation oncologist–computer interaction: A ‘Big Brother’ evaluation. Radiother Oncol 2005;77:182–90. doi:http://dx.doi.org.ejournals.um.edu.mt/10.1016/j.radonc.2005.09.017.

[30] Bowling A. Research methods in health : investigating health and health services. 2nd ed. Buckingham ; Philadelphia: Open University Press; 2002.

[31] Vinod SK, Jameson MG, Min M, Holloway LC. Uncertainties in volume delineation in radiation oncology: A systematic review and recommendations for future studies. Radiother Oncol 2016;121:169–79. doi:http://dx.doi.org/10.1016/j.radonc.2016.09.009.

[32] Cox S, Cleves A, Clementel E, Miles E, Staffurth J, Gwynne S. Impact of deviations in target volume delineation – Time for a new RTQA approach? Radiother Oncol 2019;137:1–8. doi:10.1016/J.RADONC.2019.04.012.

[33] Gwynne S, Gilson D, Dickson J, McAleer S, Radhakrishna G. Evaluating Target Volume Delineation in the Era of Precision Radiotherapy: FRCR, Revalidation and Beyond. Clin Oncol (R Coll Radiol) 2017;29:436–8. doi:10.1016/j.clon.2017.01.045.

[34] Giraud P, Elles S, Helfre S, De Rycke Y, Servois V, Carette MF, et al. Conformal radiotherapy for lung cancer: Different delineation of the gross tumor volume (GTV) by radiologists and radiation oncologists. Radiother Oncol 2002;62:27–36. doi:10.1016/S0167-8140(01)00444-3.

[35] Huo M, Gorayski P, Poulsen M, Thompson K, Pinkham MB. Evidence-based Peer Review for Radiation Therapy – Updated Review of the Literature with a Focus on Tumour Subsite and Treatment Modality. Clin Oncol 2017;29:680–8. doi:10.1016/j.clon.2017.04.038.

[36] Dimigen M, Vinod SK, Lim K. Incorporating a Radiologist in a Radiation Oncology Department: A New Model of Care? Clin Oncol 2014;26:630–5. doi:http://dx.doi.org/10.1016/j.clon.2014.04.030.

[37] Eriksen JG, Salembier C, Rivera S, De Bari B, Berger D, Mantello G, et al. Four years with FALCON – An ESTRO educational project: Achievements and perspectives. Radiother Oncol 2014;112:145–9. doi:10.1016/j.radonc.2014.06.017.

18 | Page

[38] ProKnow. Radiation Oncology Residency Programs 2018. https://proknowsystems.com/benefits/educators-rorp?content=proknow (accessed 9 January 2019).

[39] Lustberg T, van Soest J, Gooding M, Peressutti D, Aljabar P, van der Stoep J, et al. Clinical evaluation of atlas and deep learning based automatic contouring for lung cancer. Radiother Oncol 2018;126:312–7. doi:10.1016/J.RADONC.2017.11.012.

[40] Hatt M, Lee JA, Schmidtlein CR, Naqa I El, Caldwell C, De Bernardi E, et al. Classification and evaluation strategies of auto-segmentation approaches for PET: Report of AAPM task group No. 211. Med Phys 2017;44:e1–42. doi:10.1002/mp.12124.

19 | Page

Supplementary Data 1: Instructions to participants

Case Description Case 1:

1. 84 year old man presented with several weeks history of haemoptysis. 2. He is a fit gentleman with no co-morbidities. 3. Retired engineer. Lives with his wife. PS 1. Frailty score of 1. Non-smoker. Drinks a glass of sherry

every night.

Investigations Results

Histology Biopsy from the hilar lymph node on the left (10L) – no malignant cells.Biopsy from the left lung lesion – squamous cell carcinoma.

PETCT 1. Left upper lobe lung lesion.2. Intense FDG activity in the left hilar lymph nodes (SUV max 21.4). Activity

present in the sub-aortic and para-aortic lymph nodes (station 5 and 6).

Staging pT3N2M0

Case Description 2

1. 70 year old smoker with history of 3 months persistent cough.2. He has COPD, on Salbutamol and Seretide inhalers. Exercise tolerance of 200 yards. 3. Lives alone and independent in activities of daily living. PS 1. Frailty score of 3. 75 pack year

history – smokes 10/day. Minimal alcohol intake. Retired coal miner.

Investigations Results

Histology Biopsy from the hilar lymph node on the right (10R) – adenocarcinoma.Biopsy from lesion on the left lower lower lobe – squamous cell carcinoma.

Staging pT2aN1M0 RLL and pT1N0M0 LLL

Histology Biopsy from the right lung lesion – poorly differentiated adenocarcinoma.Biopsy from right hilar lymph node and mediastinal lymph nodes – no malignant cells identified.

PETCT 1. Perihilar right lung tumour with evidence of consolidation.2. Minor FDG activity on the right hilar and mediastinal lymph nodes

bilaterally. (SUV max 3.4).

20 | Page

Contouring Guidelines

1. General guidelines:

1.1. Contour the primary and nodal GTV under using the IGTV structure. If the GTVp is invading adjacent lymph nodes please draw this as one structure under the IGTVp

1.2. Make sure to select the correct structure when contouring as it is not possible to change this.

2. Case review

2.1. Review the case description and the diagnostic images provided; FDG PET-CT, diagnostic breath hold CT scan (BH), Mid-position (Mid-P) scan and 4D CT scans are provided.

2.2. The Maximum Intensity Projection (MIP) should be used for delineation.

3. General visualisation guidelines.

3.1. Planning CT;

The pre-set lung window setting should be used to delineate tumours surrounded by lung tissue while the mediastinum pre-set window setting should be used to delineate lymph nodes and tumours invading the mediastinum or chest wall. These settings are found under the contrast tab.

3.2. PET-CT;

Keep the default window settings. PET-CT threshold is set at 42% of SUVmax

3.3. The diagnostic images provided were not acquired with the patient in the treatment position. As a result mis-registration between the two scans may lead to uncertainty in delineating the final contour. The planning CT should be used when delineating the edge of the GTV. However, the diagnostic PET-CT and CT images can be used as a guide for tumour delineation side by side to the planning CT.

21 | Page

4. Using the PET/CT images.

4.1. The maximum extension of the tumour that can be visualised on CT must be included in the GTV even if there is no FDG uptake on the PET as shown in Fig.1a and 1b[1][2].

Fig.1a. Incorrect delineation: Here the FDG-PET scan was used as definitive localization tool for the GTV

Fig.1b. Correct delineation: Here the FDG-PET was used as a tool to facilitate tumour delineation while the planning CT was used as definitive localization tool for the GTV

4.2. Regions of atelectasis visible on the CT image beyond the edge of the increased FDG uptake should be excluded from the GTV. Post-obstructive inflammation should not be included in the GTV. In regions where the tumour is contiguous with a structure of similar density (e.g. atelectasis or liver), the PET can be used as a guide. However, it is important to try and visually account for possible misregistration of the tumour between the PET-CT and planning CT (Fig.2)[1].

Fig.2. Correct use of FDG PET-CT to exclude atelectasis [1].

22 | Page

5. Delineation of the lymph nodes: Which lymph nodes to include?

5.1. Lymph nodes which are proven malignant by biopsy or considered pathological on PET (focal accumulation above blood pool) are delineated as GTV as part of the GTVLn structure.

5.2. With respect to the inter-observer variation of reporting FDG-positive mediastinal nodes, in case of diagnostic uncertainty, a node should rather be included than excluded in the GTV.

5.3. Lymph nodes that are FDG-PET-positive and EBUS/EUS-negative should be included in the GTV as the false negative rates of EBUS/EUS are high (Fig. 3). PET positive nodes may only be omitted, if there is clear non-malignant biopsy explanation for the FDG positivity or if a mediastinoscopy has been performed showing no malignant cells in the lymph node.

Fig.3. Algorithm that can be used to determining when to include lymph nodes in GTV based on expected prevalence of cancer based on CT, PET and EBUS with the exception of symmetrical FDG-PET positive LN with a non-malignant diagnosis (anthracosis, silicosis, granulomatous disease after adequate full EBUS-mapping[3]. (Image reproduced with permission of the rights holder, Elsevier)

23 | Page

6. Delineation of lymph nodes: How to delineate?

6.1. For the purpose of the study delineate only the pathological lymph nodes[3]. The IASLC Mediastinum lymph node map 2009 can be used as a guide to identify lymph nodes[4]

Fig.4. CTV including 5-8mm margin[3]. (Image reproduced with permission of the rights holder, Elsevier)

7. Delineation of the I nternal Target Volume (IGTV)

GTV on the MIP incorporates the tumour motion and therefore it is referred to as the internal target volume (IGTV). After delineating the IGTV on the MIP images ensure that the IGTV lies within the respiratory boundaries on all respiratory phases of the 4D CT.

8. Additional useful references:

[1] Konert T, Vogel W, MacManus MP, Nestle U, Belderbos J, Grégoire V, et al. ESTRO acrop. Radiother Oncol 2015;116:27–34. doi:http://dx.doi.org.ejournals.um.edu.mt/10.1016/j.radonc.2015.03.014.

[2] Radiotherapy Oncology Group. Lung Atlas: RTOG 1106 Target Atlas 2019. https://www.rtog.org/CoreLab/ContouringAtlases/LungAtlas.aspx (accessed 22 March 2019).

[3] Nestle U, De Ruysscher D, Ricardi U, Geets X, Belderbos J, Pöttgen C, et al. ESTRO ACROP guidelines for target volume definition in the treatment of locally advanced non-small cell lung cancer. Radiother Oncol 2018;127:1–5. doi:10.1016/j.radonc.2018.02.023.

[4] IASLC. The Radiology Assistant: Mediastinum lymph node map 2009. http://www.radiologyassistant.nl/en/p4646f1278c26f/mediastinum-lymph-node-map.html (accessed 29 September 2018).

24 | Page

1 | Page

Big Brother Software set-up

Supplementary Data 2: Peer review instructions

1. Review the delineation of observer ______ by logging with the following password: _________________

2. If the delineation requires no changes proceed by submitting the delineation.

3. If the delineation requires adjustment, edit this accordingly. Use the annotation tool to annotate the editing as follows.

4. Submit the delineation. In the comments box that will appear after pressing submit label the contour as minor changes required, major changes required or uncertain if not sure about specific aspects.

5. After completion of the review, you can discuss the contours with the observer who performed the delineation.

1 | Page

Classification Annotation examplesMinor changes requires (MIC)Use this for minor changes that do not have a significant impact on OAR dose and PTV coverage)

Improve GTV coverage

Major changes required (MAC)Use this for major changes that do not have a significant impact on OAR dose and PTV coverage

OAR excludedExclusion of atelectasisMissing contourMissing parts of tumourMissing lymph node

Query (?):

Use this for areas you are not sure of

?lung collapse?LN4 involvement

Supplementary Data 3: Summary of editing performed following review based on expert evaluation

Course nOnly minor

changes performed

Major Correct

Major Incorrect

Major required

change not performed

Uncertain

Case 1 1 6* 2 2 0 0 12 5 1 4 0 1 0

Case 2 1 6 1 0 0 0 52 5 1 0 1 0 3

Total 21 5 6 1 1 924% 29% 5% 5% 43%

* One delineation was not provided

2 | Page

Supplementary Data 4: Comparison between the IND (blue), IPR (green) and expert consensus (red) delineations. Note the region close to the left pulmonary artery indicated with the red arrow whereby the tumour was included following IPR as per expert consensus. This change was affecting a small area and did not result in a major overall change in the CI and local SD when using the expert consensus delineation as a gold standard.

3 | Page

Supplementary Data 5: Summary of editing as classified by the participants

Course n Not provided Minor Major Uncertain

Case 1 1 6 1 2 2 02 5 2 0 0 3

Case 2 1 6 2 0 0 42 5 2 0 3

Total 21 7 2 2 1033% 10% 10% 48%

*One delineation was not provided

4 | Page

Supplementary Data 6: Level of confidence for delineating the GTV, OAR and peer review before and after the courses whereby a score of 1 indicates not confident and a score of 5 indicates very confident.

5 | Page

Supplementary Data 7: Content analysis of feedback questionnaire.

Course Positive Comments Suggested Improvements

1 Excellent case selection (n=1)

Excellent software (n=1)

Well organised (n=1)

Radiology input (n=2)

Radiology Pet-CT interpretation (n=5)

More aware about the importance of peer review (n=2)

Need for providing more feedback on own contours (n=2)More time for IPR (n=1)Extending to other sites (n=1)More practical IGRT training (n=1)

2 Excellent course (n=5)

Radiology PET-CT interpretation, (n=4)

Radiology input (n=2)

More aware about the importance of peer review (n=2)

Interobserver variation in GTV outlining (n=2)

Need for providing more feedback on own contours (n=1)More cases (n=2)

To include autosegmentation (n=1)

To include more about motion management (n=1)

Discuss margins (n=1)

6 | Page

7 | Page