Which contour edits actually matter?

Four panels: an axial CT slice with the reference target contour in white and an edited one in cyan, the edited region filled yellow; the re-optimised dose plan as a heat map; the change in dose as a red and blue diverging map; and a strip of coloured verdicts in which every judge says the edit was harmful while the ground truth says it was not. — One edit, and everything anyone said about it. The target volume was enlarged by 1.7 cc (yellow, left). The re-optimised plan (middle) redistributes dose, and the change it makes (right) stays inside every clinical constraint. All seven clinicians, and the model, called this edit harmful. It was not.

The problem

Automatic segmentation now draws most radiotherapy contours, which turns the clinician into a reviewer. A reviewer has to decide, for each difference they see, whether it is worth correcting: correcting everything throws away the time auto-contouring was meant to save, correcting nothing risks the patient. That decision is dosimetric, but it is made by looking at outlines, long before any dose is computed.

So we measured how well it is made. Fourteen glioblastoma patients, up to four plausible edits each to the clinical target volume, and every one of the 54 results re-planned in Eclipse, which establishes what each edit does to the 13 organs at risk. Then we asked clinicians to predict that from the contours alone, and asked a dose-prediction model the same question.

What we found

Clinicians do not agree with each other. Over the 21 pairs of seven evaluators, Cohen's Kappa runs 0.24 to 0.74, median 0.62. Only 11 pairs reach the 0.6 that counts as strong agreement.
Harm is systematically over-called. On average 18.7 of the 45 harmless edits (42%) were called "worse": time spent undoing changes that did nothing.
Nobody ever said an edit helped. The "better" label went unused across all 54 edits and all seven evaluators, though four edits genuinely lowered organ dose, by 0.67 Gy on average.
Real harm gets missed. Of the five edits that breached a constraint, two were caught by nobody at all.
The model beat the oncologists it was benchmarked against. On the conference definition of impact it reached precision and recall of 0.57, against 0.41–0.55 and 0.35–0.57 for three oncologists, in 30 s per edit against 44–71 s.

How "impact" came to be defined, twice

The conference study needed a rule it could compute for all 54 re-planned edits, so it used a relative one: an edit is impactful if the maximum dose to at least one organ moves by more than 10%. Presenting that work, the clinicians pushed back on the rule itself. A 10% change in maximum dose is not on its own clinically meaningful. Ten percent added to an organ far below its tolerance changes nothing for the patient, while a much smaller rise on an organ already near its limit can change the plan.

The journal study therefore rebuilt the ground truth around the constraints clinicians plan against: +1 for each organ an edit pushes past its guideline limit, −1 for each it brings back under, summed over all 13.

Ground truth	Test	Verdicts
MIDL 2024	the maximum dose to at least one organ at risk moves by more than 10%	33 worse, 17 no change, 4 better
RadOnc 2025	a clinical dose constraint is crossed, scored +1 or −1 per organ and summed over all 13	5 worse, 45 no change, 4 better

The two labellings agree on 18 of the 54 edits. Of the 33 the first rule called harmful, only 3 cross a clinical limit; 27 are neutral and 3 are improvements. It also explains the dose predictor's two scores: its own rule is a relative one, so it does well against the first definition and its recall falls to 0.33 against the second.

Who judged, and when. The conference paper reported three radiation oncologists; the journal reports seven, adding a fourth oncologist whose answers came from the same round and three medical physicists surveyed afterwards. None could have been influenced by the earlier result: all four oncologists were surveyed in January and February 2024, before the conference paper appeared, and their columns in the journal data are identical to the answers recorded then. The physicists judged under the same protocol, with no access to the dose distribution, the ground truth, the earlier results or each other's answers.

Watching it happen

Each clip sweeps through one patient. Left is what the evaluators were given: the reference target in white, the edited one in cyan, with tissue the edit added and tissue it removed filled in. Middle is the re-optimised plan. Right is what the edit did to the dose, which nobody was allowed to see. The strip carries both ground truths, all seven evaluators and the model, and the two lines beneath give the reason behind each verdict: a crossed constraint for the plan, a relative dose shift for the model. Both are generated from the data, and both rules reproduce all 54 published labels exactly.

All 54 edits, one after anotherThe whole study in one pass. Watch the verdict strip rather than the anatomy: the orange "worse" column runs down it almost unbroken while the ground truth beside it stays blue, and green appears only in the truth row.

Everyone agreed, and everyone was rightAll seven, and the model, called this edit harmless, and the re-optimised plan bears them out.

Everyone agreed, and everyone was wrongA 1.7 cc enlargement all seven called harmful. The dose moves, but never crosses a constraint. Consensus is not evidence.

Harm that nobody caughtThis edit takes the brainstem from 52.6 to 56.2 Gy against a 54 Gy limit. Not one of the seven flagged it; the model did.

Harm that was caughtSix of seven flagged this one, correctly. Where an edit pushes the target hard towards an organ, clinicians read it well; the failures are in the subtle cases.

An improvement nobody sawThis edit brought the left eye back under its limit. Five of seven called it harmful and the other two neutral.

The edit they disagreed about mostThree of seven called it harmful, four did not. Same image, same instructions, same profession.

The model right, most evaluators wrongFour of seven called this harmful; the plan disagrees, and so did the predictor.

The model wrong, evaluators rightIts characteristic failure: it predicts a smooth dose rather than the plan's beam structure, so it over-reads small changes near an organ.

Unanimous, and unanimously wrongEvery evaluator called this harmless, and so did the model. The re-optimised plan crosses a constraint.

Results

How much the evaluators agreed with each other

Cohen's Kappa for all 21 pairs. Above 0.6 is conventionally read as strong agreement; ten pairs fall below it, and each of the five weakest involves R‑1, the evaluator who used "worse" least often and who also scored closest to the ground truth.

two oncologists two physicists one of each

How each judge spent their verdicts

Every evaluator had 54 edits and three labels. The ground truth uses all three; no evaluator did. R‑1 called 44 edits harmless and R‑3 called 30 of them harmful, on identical images.

better no change worse

The model against three oncologists

The conference comparison, on the conference ground truth: weighted precision and recall over the three categories, and the mean time one edit took to judge. The model's advantage is not that it reads anatomy better, but that it answers a dosimetric question by predicting a dose rather than inferring one from the shape of an outline.

precision recall

What to do with this

Define impact against the constraints, not against a percentage. A 10% threshold flagged 33 of these 54 edits; only 3 cross a clinical limit.
Expect over-calling, and budget for it. Reviewers are cautious by training, and caution is cheap to express and expensive to act on. Two in five harmless edits were flagged for correction.
Show the dose, or nobody will see an improvement. An improvement of 0.67 Gy is not visible in an outline, and the "better" category went unused.
A fast dose predictor is a reasonable second opinion, not an oracle. It beat three oncologists on the conference standard and caught harm all seven missed, and it also over-flags.

Citation

@article{willmann2025predicting,
  title     = {Predicting the impact of target volume contouring variations on the organ at risk dose: results of a qualitative survey},
  author    = {Willmann, Jonas and Kamath, Amith and Poel, Robert and Riggenbach, Elena and Mose, Lucas and Bertholet, Jenny and M{\"u}ller, Silvan and Schmidhalter, Daniel and Andratschke, Nicolaus and Ermi{\c{s}}, Ekin and Reyes, Mauricio},
  journal   = {Radiotherapy and Oncology},
  pages     = {110999},
  year      = {2025},
  publisher = {Elsevier}
}

@inproceedings{kamath2024comparing,
  title        = {Comparing the Performance of Radiation Oncologists versus a Deep Learning Dose Predictor to Estimate Dosimetric Impact of Segmentation Variations for Radiotherapy},
  author       = {Kamath, Amith and Mercado, Zahira and Poel, Robert and Willmann, Jonas and Ermi{\c{s}}, Ekin and Riggenbach, Elena and Andratschke, Nicolaus and Reyes, Mauricio},
  booktitle    = {Medical Imaging with Deep Learning},
  volume       = {250},
  year         = {2024},
  organization = {PMLR}
}

Errata

Values in the published papers that do not follow from the released data. None affects a reported conclusion. Full audit in results/radonc_verification.csv and results/midl_verification.csv.

Radiotherapy and Oncology 2025

Contour variation volumes. Reported as a median of 13.2 cc (range 4.03–50.18). The correct values are 1.66 cc (0.50–6.27); the conversion to cubic centimetres applied the voxel volume twice.
Cohen's Kappa, lower bounds. Reported as a minimum of 0.33 and an inter-quartile range of 0.08. The released responses give 0.24 and 0.13. The median, maximum and the count of pairs above 0.6 are unaffected.
Precision, recall and F1. Computed with the evaluator labels as the reference and the ground truth as the prediction. The reported values should be read in that direction; relative performance between evaluators is unaffected.

MIDL 2024

Table 1, evaluator R1 time. Reported as 48 s, which is the per-patient standard deviation. The per-edit mean is 44 s, consistent with the other two rows.