MICCAI 2023  ·  Computers in Biology and Medicine 2025

Do we really need that skip-connection?

Understanding its interplay with task complexity

Amith Kamath1, Jonas Willmann1,2, Nicolaus Andratschke2, Mauricio Reyes1

1University of Bern    2University Hospital Zurich

Early accept at MICCAI 2023 (top 14%)  ·  Uni Bern BME Club Travel Award

MICCAI paper Journal extension Code Talk Try it

In short. Skip connections are treated as a default in U-Net style segmentation. They earn their place when the object and its background look different from each other. When the two textures converge, the same skips are what carry the failure, and a plain encoder–decoder with no skips at all holds up better.

Overview: synthetic images with a textured foreground on a textured background are segmented by three U-Net variants; task complexity is measured as the KL divergence between local binary pattern histograms of foreground and background.
The setup. We generate images whose foreground and background textures we can move together or apart, measure how similar they are, and watch what each architecture does as that similarity increases.

Abstract

Skip connections are considered essential in U-Net architectures for medical image segmentation, on the grounds that they preserve fine detail by bridging encoder and decoder layers. But are they always necessary? We introduce a measure of task complexity based on the texture disparity between foreground and background, and use it to ask when skip connections help and when they hurt.

Across controlled synthetic experiments and real datasets spanning breast ultrasound, colon histology, cardiac MRI and abdominal CT, a consistent picture emerges. Skip connections give little benefit at low and medium complexity, and become genuinely useful only when foreground and background are hard to tell apart. At the same time they carry a cost: architectures with denser skips degrade far more sharply under distribution shift than the same networks with the skips removed. Performance and robustness pull in opposite directions, and for safety-critical deployment the simpler architecture is often the better bet.

What we found

How task complexity is measured

We need a number that says how hard it is to tell foreground from background in a given image. We take local binary pattern histograms of each region and compute the KL divergence between them. A large divergence means the two textures are easy to separate and the task is easy; as the divergence shrinks the task gets harder in a way we can dial continuously with a blending parameter α.

The architectures

Six models in three groups, so that each network with skips has a near-identical partner without them. That pairing is what isolates the effect.

GroupModelsSkip connections
No-skipNoSkip U-Net, NoSkip V-NetNone; encoder and decoder communicate only through the bottleneck
StandardU-Net, V-NetIdentity skips, by concatenation and by addition respectively
EnhancedAGU-Net, U-Net++Attention-gated skips and dense nested skips

Implemented with MONAI and PyTorch. Three seeds per configuration on the medical datasets.

Watching it break

Each clip runs all six architectures over the same test images while the texture disparity is dialled away from the training distribution. Read a panel left to right: the clean image, the ground truth, the image the model actually sees, and its prediction laid over the truth with a Dice score (1.000 is perfect, 0.000 means it found nothing). Cyan is the expert annotation, magenta is the model. Each row is one architecture, arranged so that every skip-connected model sits beside its no-skip twin.

The panels are dense. Use the fullscreen control on any clip to read the scores.

The blob starts to look like its backgroundA textured shape on a textured background, with a dial (α) that mixes the background texture into the shape. At α = 0.10 the images match training. By α = 0.88 the two are nearly the same material, and the models with skips are the first to lose the object entirely.
The background starts to look like the blobThe same blend running the other way. The failure changes character completely: instead of losing the object, the skip-connected models start finding it everywhere. Two failures this different can average to the same Dice, which is the case against reporting only a mean.
Breast ultrasound (BUSI)Lesion segmentation in a modality that is speckled to begin with, so added speckle is a realistic stressor rather than an artificial one.
Colon histology (GLaS)Gland segmentation, where the boundary is defined by texture rather than intensity. This is the hardest setting for an architecture that leans on fine detail.
Left atrium (heart MRI)A structure with a strong, consistent shape prior. Degradation is gentler here and the architectures stay closer together.
Spleen (CT)A large organ in a modality with standardised intensities, included to test whether the pattern holds when texture is not the dominant cue.

Results

Mean Dice on the held-out test sets, averaged over three seeds, as the perturbation moves from background blur (easier than training) through the unperturbed images to speckle noise (harder). Lines that stay flat are robust; lines that fall off a cliff are not.

No-skip Standard Enhanced U-Net backbone V-Net backbone
Breast ultrasound (BUSI) 0.00 0.25 0.50 0.75 1.00 Easiest Easier Unpert. Harder Hardest NoSkip U-Net · Easiest · Dice 0.744 NoSkip U-Net · Easier · Dice 0.749 NoSkip U-Net · Unperturbed · Dice 0.761 NoSkip U-Net · Harder · Dice 0.735 NoSkip U-Net · Hardest · Dice 0.620 NoSkip V-Net · Easiest · Dice 0.743 NoSkip V-Net · Easier · Dice 0.762 NoSkip V-Net · Unperturbed · Dice 0.760 NoSkip V-Net · Harder · Dice 0.752 NoSkip V-Net · Hardest · Dice 0.730 U-Net · Easiest · Dice 0.745 U-Net · Easier · Dice 0.736 U-Net · Unperturbed · Dice 0.763 U-Net · Harder · Dice 0.723 U-Net · Hardest · Dice 0.607 V-Net · Easiest · Dice 0.713 V-Net · Easier · Dice 0.717 V-Net · Unperturbed · Dice 0.724 V-Net · Harder · Dice 0.712 V-Net · Hardest · Dice 0.690 AGU-Net · Easiest · Dice 0.804 AGU-Net · Easier · Dice 0.799 AGU-Net · Unperturbed · Dice 0.795 AGU-Net · Harder · Dice 0.646 AGU-Net · Hardest · Dice 0.484 U-Net++ · Easiest · Dice 0.746 U-Net++ · Easier · Dice 0.714 U-Net++ · Unperturbed · Dice 0.733 U-Net++ · Harder · Dice 0.432 U-Net++ · Hardest · Dice 0.165 Colon histology (GLaS) 0.00 0.25 0.50 0.75 1.00 Easiest Easier Unpert. Harder Hardest NoSkip U-Net · Easiest · Dice 0.690 NoSkip U-Net · Easier · Dice 0.746 NoSkip U-Net · Unperturbed · Dice 0.800 NoSkip U-Net · Harder · Dice 0.720 NoSkip U-Net · Hardest · Dice 0.713 NoSkip V-Net · Easiest · Dice 0.670 NoSkip V-Net · Easier · Dice 0.728 NoSkip V-Net · Unperturbed · Dice 0.818 NoSkip V-Net · Harder · Dice 0.762 NoSkip V-Net · Hardest · Dice 0.734 U-Net · Easiest · Dice 0.693 U-Net · Easier · Dice 0.742 U-Net · Unperturbed · Dice 0.799 U-Net · Harder · Dice 0.730 U-Net · Hardest · Dice 0.715 V-Net · Easiest · Dice 0.713 V-Net · Easier · Dice 0.742 V-Net · Unperturbed · Dice 0.786 V-Net · Harder · Dice 0.757 V-Net · Hardest · Dice 0.741 AGU-Net · Easiest · Dice 0.720 AGU-Net · Easier · Dice 0.783 AGU-Net · Unperturbed · Dice 0.817 AGU-Net · Harder · Dice 0.741 AGU-Net · Hardest · Dice 0.697 U-Net++ · Easiest · Dice 0.718 U-Net++ · Easier · Dice 0.771 U-Net++ · Unperturbed · Dice 0.820 U-Net++ · Harder · Dice 0.720 U-Net++ · Hardest · Dice 0.703 Left atrium (heart MRI) 0.00 0.25 0.50 0.75 1.00 Easiest Easier Unpert. Harder Hardest NoSkip U-Net · Easiest · Dice 0.805 NoSkip U-Net · Easier · Dice 0.823 NoSkip U-Net · Unperturbed · Dice 0.833 NoSkip U-Net · Harder · Dice 0.814 NoSkip U-Net · Hardest · Dice 0.761 NoSkip V-Net · Easiest · Dice 0.854 NoSkip V-Net · Easier · Dice 0.872 NoSkip V-Net · Unperturbed · Dice 0.882 NoSkip V-Net · Harder · Dice 0.879 NoSkip V-Net · Hardest · Dice 0.871 U-Net · Easiest · Dice 0.740 U-Net · Easier · Dice 0.805 U-Net · Unperturbed · Dice 0.901 U-Net · Harder · Dice 0.492 U-Net · Hardest · Dice 0.076 V-Net · Easiest · Dice 0.872 V-Net · Easier · Dice 0.896 V-Net · Unperturbed · Dice 0.925 V-Net · Harder · Dice 0.893 V-Net · Hardest · Dice 0.784 AGU-Net · Easiest · Dice 0.827 AGU-Net · Easier · Dice 0.890 AGU-Net · Unperturbed · Dice 0.929 AGU-Net · Harder · Dice 0.142 AGU-Net · Hardest · Dice 0.033 U-Net++ · Easiest · Dice 0.034 U-Net++ · Easier · Dice 0.017 U-Net++ · Unperturbed · Dice 0.925 U-Net++ · Harder · Dice 0.008 U-Net++ · Hardest · Dice 0.000 Spleen (CT) 0.00 0.25 0.50 0.75 1.00 Easiest Easier Unpert. Harder Hardest NoSkip U-Net · Easiest · Dice 0.105 NoSkip U-Net · Easier · Dice 0.271 NoSkip U-Net · Unperturbed · Dice 0.606 NoSkip U-Net · Harder · Dice 0.562 NoSkip U-Net · Hardest · Dice 0.434 NoSkip V-Net · Easiest · Dice 0.141 NoSkip V-Net · Easier · Dice 0.223 NoSkip V-Net · Unperturbed · Dice 0.732 NoSkip V-Net · Harder · Dice 0.415 NoSkip V-Net · Hardest · Dice 0.315 U-Net · Easiest · Dice 0.023 U-Net · Easier · Dice 0.107 U-Net · Unperturbed · Dice 0.746 U-Net · Harder · Dice 0.078 U-Net · Hardest · Dice 0.025 V-Net · Easiest · Dice 0.000 V-Net · Easier · Dice 0.082 V-Net · Unperturbed · Dice 0.925 V-Net · Harder · Dice 0.057 V-Net · Hardest · Dice 0.001 AGU-Net · Easiest · Dice 0.053 AGU-Net · Easier · Dice 0.299 AGU-Net · Unperturbed · Dice 0.928 AGU-Net · Harder · Dice 0.308 AGU-Net · Hardest · Dice 0.215 U-Net++ · Easiest · Dice 0.010 U-Net++ · Easier · Dice 0.072 U-Net++ · Unperturbed · Dice 0.908 U-Net++ · Harder · Dice 0.000 U-Net++ · Hardest · Dice 0.000

Every point is also in the table below, and in dice-summary.csv.

Mean Dice, all runs

ArchitectureEasiestEasierUnperturbedHarderHardestUnpert. → hardest
Breast ultrasound (BUSI)
NoSkip U-Net0.7440.7490.7610.7350.620−0.141
NoSkip V-Net0.7430.7620.7600.7520.730−0.030
U-Net0.7450.7360.7630.7230.607−0.155
V-Net0.7130.7170.7240.7120.690−0.034
AGU-Net0.8040.7990.7950.6460.484−0.311
U-Net++0.7460.7140.7330.4320.165−0.567
Colon histology (GLaS)
NoSkip U-Net0.6900.7460.8000.7200.713−0.087
NoSkip V-Net0.6700.7280.8180.7620.734−0.083
U-Net0.6930.7420.7990.7300.715−0.085
V-Net0.7130.7420.7860.7570.741−0.045
AGU-Net0.7200.7830.8170.7410.697−0.120
U-Net++0.7180.7710.8200.7200.703−0.117
Left atrium (heart MRI)
NoSkip U-Net0.8050.8230.8330.8140.761−0.072
NoSkip V-Net0.8540.8720.8820.8790.871−0.011
U-Net0.7400.8050.9010.4920.076−0.824
V-Net0.8720.8960.9250.8930.784−0.141
AGU-Net0.8270.8900.9290.1420.033−0.896
U-Net++0.0340.0170.9250.0080.000−0.925
Spleen (CT)
NoSkip U-Net0.1050.2710.6060.5620.434−0.172
NoSkip V-Net0.1410.2230.7320.4150.315−0.417
U-Net0.0230.1070.7460.0780.025−0.721
V-Net0.0000.0820.9250.0570.001−0.924
AGU-Net0.0530.2990.9280.3080.215−0.713
U-Net++0.0100.0720.9080.0000.000−0.908

The last column is the drop from the unperturbed test set to the hardest one. Read down it and the pattern is hard to miss: within every dataset, the no-skip models give up the least. On spleen CT every architecture suffers, but the no-skip pair retains roughly half its performance while the others go to near zero.

What to do with this

Citation

@inproceedings{kamath2023we,
  title     = {Do we really need that skip-connection? Understanding its interplay with task complexity},
  author    = {Kamath, Amith and Willmann, Jonas and Andratschke, Nicolaus and Reyes, Mauricio},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  pages     = {302--311},
  year      = {2023},
  organization = {Springer}
}
@article{kamath2025impact,
  title     = {The impact of U-Net architecture choices and skip connections on the robustness of segmentation across texture variations},
  author    = {Kamath, Amith and Willmann, Jonas and Andratschke, Nicolaus and Reyes, Mauricio},
  journal   = {Computers in Biology and Medicine},
  volume    = {184},
  pages     = {109364},
  year      = {2025},
  publisher = {Elsevier}
}