SNU Computer Vision Team Develops an Efficient Method for Scaling Pretrained AI Models

- Introduces “Cluster-aware Upcycling,” which leverages the semantic structure of pretrained models to promote specialization among expert modules

- Joint research by the SNU College of Engineering and LG AI Research presents a way to efficiently scale large AI models

- Paper accepted to CVPR 2026, one of the world’s leading conferences in computer vision

▲ Research image (AI-generated)

A new technology has been developed that enables pretrained AI models to be expanded into larger models with specialized expert modules, without training the models from scratch.

A research team led by Professor Bohyung Han of the Department of Electrical and Computer Engineering at Seoul National University (SNU), in collaboration with LG AI Research, has developed a technique called “Cluster-aware Upcycling,” which efficiently converts pretrained AI models into Mixture-of-Experts (MoE) models.

A Mixture-of-Experts architecture works much like a group of specialists dividing tasks according to the nature of a question. When a generative AI model handles diverse requests—such as coding, translation, and mathematics—it activates only the expert modules best suited to each input token, rather than using every module simultaneously. This allows the model to increase its overall knowledge and processing capacity without a proportional increase in the computational cost required for each input.

The research team proposed a novel method that leverages the semantic structure already present in pretrained models, enabling expert modules to specialize in different types of inputs from the outset of training. The work demonstrates the potential for efficient scaling of Mixture-of-Experts models and has been accepted to CVPR 2026, one of the world’s leading conferences in computer vision.

Large-scale AI models generally improve in performance as they grow in size. However, in conventional dense architectures, where all parameters are used for every input, the computational cost of training and inference also grows in proportion to model size. Mixture-of-Experts architectures can help alleviate this limitation. Nevertheless, training a large Mixture-of-Experts model from scratch on large-scale datasets still requires substantial time and computational resources.

▲ Figure 1. Comparison of Sparse Upcycling and Cluster-aware Upcycling.

The proposed method leverages semantic structure to initialize both the experts and the router, promoting early expert specialization.

Previous studies have sought to reduce these costs through Sparse Upcycling, which copies the weights of a pretrained model into multiple expert modules. While this approach preserves the knowledge of the original model, all experts begin with identical weights, making it difficult for them to develop distinct roles early in training.

In addition, the router—which determines which expert processes each input—is randomly initialized without considering the characteristics of the data. This is analogous to assembling a group of specialists without assigning them specific domains and then distributing questions at random. As a result, multiple experts may redundantly learn similar functions, making meaningful specialization difficult during the early stages of training.

To address this issue, the research team focused on the fact that pretrained models already organize inputs with similar meanings into groups within their internal representations.

The proposed method, Cluster-aware Upcycling, first collects the activations of input tokens at each layer of the pretrained model and groups tokens with similar features into clusters. For each cluster, the method computes a whitening matrix, transforms the original model weights accordingly, and applies truncated singular value decomposition. This preserves weight directions that are most important for processing the corresponding cluster while reducing less relevant directions, giving each expert a distinct initialization tailored to its assigned cluster.

Unlike conventional approaches where all experts start from identical copies, each expert is initialized with weights better suited to processing a particular group of tokens.

▲ Figure 2. Cluster-aware initialization process.

Experts are initialized to preserve weight directions important for each cluster, while the router uses cluster centroids to assign inputs to appropriate experts.

Rather than being initialized randomly, the router is initialized using the centroid of each cluster. As a result, the semantically closest expert is given priority for each new token. Different experts can therefore be selected for different tokens even within the same sentence or image.

▲ Figure 3. Expert-ensemble self-distillation.

During training, a prediction that combines information from all experts provides an additional learning signal to preserve and strengthen expert specialization.

The research team further proposed an Expert-Ensemble Self-Distillation method. Some tokens are similarly relevant to multiple experts, making it difficult for the router to assign them clearly to a particular expert. Such ambiguous tokens can hinder individual experts from learning consistent, specialized roles.

To address this, predictions from an ensemble teacher are used as an additional training signal. This signal has a stronger effect on tokens whose expert assignments are ambiguous and a weaker effect on tokens that already have clear assignments, thereby preserving and reinforcing expert specialization.

The team applied the proposed method to CLIP, a widely used vision-language model, and evaluated it in zero-shot settings without additional dataset-specific training. The method consistently outperformed conventional Sparse Upcycling on image-text retrieval and a range of image classification benchmarks.

Notably, average accuracy on VTAB-Natural, which comprises a range of natural-image tasks, increased from 62.0% to 63.3%. Accuracy increased from 42.3% to 43.5% on ObjectNet, whose distribution differs from that of the training data, and from 71.2% to 72.1% on ImageNet-R, which contains images with different visual styles and forms of representation. These consistent improvements in new environments without additional dataset-specific training indicate that the proposed method effectively improves the generalization of pretrained models.

Further analysis of expert behavior showed reduced similarities in both expert weights and internal representations, indicating that the experts developed more distinct representation spaces. Router uncertainty also decreased, while tokens were more evenly distributed across experts rather than being concentrated in only a few. These findings show that the proposed method strengthens expert specialization, a core objective of Mixture-of-Experts architectures.

This research presents a new way to reuse AI models that have already required substantial time and resources to train. Rather than building a Mixture-of-Experts model from scratch, the proposed method efficiently expands an existing pretrained model.

The method could be applied to scale enterprise generative AI models into larger models capable of handling diverse tasks such as coding, document writing, translation, and mathematical reasoning. It may also be used in medical AI systems to process different types of data, including CT scans, X-rays, and pathology images. More broadly, the approach could support the scaling of a range of foundation models, including large language models and multimodal models that jointly process information such as images, videos, and speech.

In industrial settings, the technology could enable organizations to reuse existing models and training assets, broaden the range of their services and enhance their specialization, and reduce the time and computing resources needed to train new models from scratch.

▲ (From left) Sanghyeok Chu (Ph.D. candidate, Department of Electrical and Computer Engineering), Bohyung Han (Professor, Department of Electrical and Computer Engineering and Interdisciplinary Program in Artificial Intelligence)

Professor Bohyung Han, who supervised the research, said:

“This study is meaningful in that it goes beyond simply copying weights when converting pretrained models into Mixture-of-Experts architectures. We instead use the semantic structure already present within the model to encourage distinct roles among the expert modules.”

He added:

“By jointly initializing the experts and the router based on the semantic structure of the data, we were able to guide expert specialization more reliably from the early stages of training. Going forward, we plan to evaluate the scalability and efficiency of the method on larger vision-language models and large language models, as well as in a broader range of domains such as healthcare.”

The first author of the study, Sanghyeok Chu, a Ph.D. candidate at SNU, plans to continue his research on the efficient scaling and architectural design of large vision-language models. His future work will explore computationally efficient architectures, including Mixture-of-Experts and linear-time models, and develop technologies that enable multimodal foundation models to process complex and long inputs efficiently under limited computational resources.

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2022-II220959 (No.2022-0-00959), (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making).

[Reference Materials]

-Title/Journal: Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling, CVPR 2026

[Contact Information]

Professor Bohyung Han, Department of Electrical and Computer Engineering, Seoul National University / bhhan@snu.ac.kr