CVPR 2026

Revisiting Model Stitching
in the Foundation Model Era

Can heterogeneous Vision Foundation Models (VFMs) be stitched, fused, and served more efficiently?

Zheda Mai^1,3* Ke Zhang³ Fu-En Wang³ Zixiao Ken Wang³ Albert Y. C. Chen³ Lu Xia³ Min Sun³ Wei-Lun Chao² Cheng-Hao Kuo³

¹The Ohio State University

²Boston University

³Amazon

* Work done during internship at Amazon.

Paper Slides Poster Code

TL;DR We revisit model stitching for Vision Foundation Models (VFMs) and show that different VFMs are not only compatible, but complementary. Our stitching recipe fuses their strengths and enables efficient multi-VFM systems through VFM Stitch Tree.

1-Minute Quick Read

Card 1 — Motivation

Different VFMs, such as DINOv2 and SigLIP2, are trained with different data and objectives. They share some common visual abilities, but each also has unique strengths and weaknesses.

Card 2 — Model Stitching

Model Stitching connects the shallow layers of one VFM to the deep layers of another with a small stitch layer. If the stitched model still works well, the two VFMs have compatible internal representations.

Card 3 — Our Recipe

We find that naive stitching is not enough. Our two-stage recipe first matches the target model’s final features, then fine-tunes the stitch layer with task loss.

Card 4 — Takeaway

Stitching heterogeneous VFMs is not only possible—it can fuse complementary strengths. With VFM Stitch Tree, multi-VFM systems can recover much of the gain of multiple full VFMs with far lower overhead.

Motivation

VFMs are trained with different datasets, objectives, and modalities. They perform differently on downstream tasks, raising a key question: are they fundamentally different end-to-end, or are their internal representations still similar in some layers?

Question 1

Can heterogeneous VFMs be stitched with minimal performance loss?

Question 2

Can stitching do more than compatibility and fuse strengths?

Question 3

Can this become a practical systems tool for multi-VFM efficiency?

Vision foundation model capabilities screenshot — VFMs share common visual abilities while preserving unique strengths.

Internal representation compatibility screenshot — Are heterogeneous VFMs different end-to-end, or similar in some layers?

Model Stitching

Model Stitching: connect the shallow layers of one model to the deep layers of another via a stitch layer (e.g., linear or MLP) to form a stitched model. If the stitched model has minimal performance drop, it means their internal representations are similar.

Naive Stitching Fails

1. Layer Feature Matching

Layer Feature Matching trains the stitch layer to match the source and target features at the stitch position.

Why it fails. A small mismatch at the stitch position can be amplified by the frozen target layers, creating a large final-feature difference.

The remedy is Final Feature Matching: train the stitch layer so the stitched model matches the target model's final representation directly.

Layer feature matching result — Layer Feature Matching can get small layer feature differences locally but can drift at the final representation. Final Feature Matching reduces final feature mismatch.

Final feature matching result — Final Feature Matching consistently improves stitched model accuracy.

2. Task-Loss Training

Task-Loss Training optimizes the stitch layer directly with the downstream objective, such as classification loss.

Problem. It sounds natural, but shallow stitching makes optimization difficult: as all post-stitch target layers are frozen, gradients must backprop through a long, fixed transformation to update only the stitch layer, especially challenging for randomly initialized stitch layer.

On DINOv2 -> SigLIP2 stitched at layer 2, Task-Loss Training drops to 25.1%, far below both original models and both feature-matching strategies.

DINOv2	SigLIP2	Layer Feature Matching	Final Feature Matching	Task Loss Training
46.7%	53.4%	49.4%	51.4%	25.1%

DINOv2→SigLIP2 stitched at layer 2.

Our Stitching Recipe

The key is to separate representation alignment from task adaptation. We first make the stitched model imitate the target VFM's final representation, then use task loss only after the stitch layer has a strong initialization.

Stage 1: Align Representations

Final Feature Matching is label-free: it trains only the stitch layer so the stitched model mimics the target model's final features.

Stage 2: Adapt to the Task

Task-Loss Fine-Tuning then specializes the aligned stitched model to the downstream objective, avoiding the optimization challenges of direct task-loss training.

fMoW accuracy of stitched models compared with original VFM linear probing baselines — On fMoW, the stitched models outperform both original VFM linear-probing baselines. The gains become stronger at deeper stitch positions. LP: linear probing on a frozen original VFM.

Knowledge Fusion, Not Just Capacity

A stitched model has an extra trainable layer, so we need a control: does the improvement come from added capacity, or from combining two different VFMs?

Self-Stitch Baseline

Insert the same stitch layer at the same stitch position into source and target models with the same training data and objective.
Isolate whether improvement is only from the added capacity of the stitch layer.

	Classification			Segmentation
	fMoW	iNaturalist	Aircraft	ADE20K
Stitch Position	6 / 14 / 22	6 / 14 / 22	6 / 14 / 22	14 / 22
DINOv2→DINOv2	41.5 / 59.7 / 69.9	56.9 / 81.5 / 91.2	37.8 / 79.3 / 91.2	35.4 / 50.9
SigLIP2→SigLIP2	50.5 / 62.0 / 68.9	71.2 / 88.5 / 87.3	67.9 / 88.1 / 89.3	44.5 / 50.5
DINOv2→SigLIP2	59.8 / 68.0 / 71.8	75.9 / 89.1 / 92.8	77.8 / 87.6 / 92.4	44.9 / 51.2
SigLIP2→DINOv2	53.8 / 69.6 / 72.2	86.3 / 88.9 / 91.9	80.7 / 89.0 / 91.0	49.0 / 51.4

Stitching heterogeneous VFMs is not only feasible; it can fuse complementarity between models.

Generality: Not a One-off Result

The same stitching recipe generalizes across model pairs, stitch positions, datasets, and both classification and segmentation tasks.

Stitchability generalizes across tasks, datasets, models, pretraining, and stitch layer families

Key Finding

Heterogeneous VFMs trained with different data, objectives, and modalities remain stitchable across diverse vision tasks and datasets.
Stitched VFMs consistently outperform Self-Stitch baselines, indicating that the gains arise not merely from the added capacity of the stitch layer, but also from the complementary strengths of different VFMs.

VST: Share Shallow, Specialize Deep

Multi-VFM systems improve performance by combining complementary visual representations, but they also introduce k× memory and latency when running k VFMs independently.

Share Shallow

Reuse common early visual processing instead of running every VFM from scratch.

Specialize Deep

Keep model-specific deep branches, connected by lightweight stitch layers.

Efficiency Knob

Choose the stitch position to trade off accuracy recovery against memory and latency.

VFM Stitch Tree compared with naive multi-VFM systems — VST shares shallow layers, then branches into specialized VFMs through stitch layers.

VFM Stitch Tree inside MoF-LLaVA — In MoF-LLaVA, VST replaces independent VFM backbones with a shared shallow trunk and task-useful deep branches.

VST gain versus overhead compared with naive multi-VFMs

MoF-LLaVA Result

VFM Stitch Tree (VST) shares common shallow layers and keeps model-specific deep branches connected by lightweight stitch layers, enabling a practical accuracy-efficiency knob.

In MoF-LLaVA, VST recovers most of the two-VFM gain with much lower overhead: VST-22 uses only 4.3% extra resources, while VST-14 reaches 39.0% overhead and recovers substantially more gain than the low-overhead setting.

Takeaways

Stitch layer training matters.

Stitching fuses strengths.

Stitching enables efficient systems.

Model Stitching, from an analytical tool to a practical framework for combining the complementary strengths of VFMs.