UniW(A)M logo

UniW(A)M

A Unified Mixture-of-Experts Framework for World and World-Action Models

Jingbo Xu1*, Jianjie Fang1*, Zhuohang Li1, Zihao Song1, Zhaolu Wang2, Xvyuan Liu3, Xin Zhang4, Haisheng Su4, Yinzhou Tang1, Chen Gao1,2†, Wei Wu4†, Yong Li1†

1Tsinghua University2Zhongguancun Academy3East China Normal University4Manifold AI

*Joint first authors.†Corresponding authors.

World models (WMs) and world-action models (WAMs) focus on future world prediction and executable action generation, respectively. Although these models learn shared physical dynamics and interaction semantics from embodied trajectories, they are typically trained in isolation, which limits the reuse of knowledge between prediction and control. The central challenge is therefore to develop a unified learning framework that shares these dynamics while preserving the distinct modeling capabilities required by each paradigm. We propose UniW(A)M, a mixture-of-experts world-(action) model built upon diffusion Transformers that unifies action-conditioned world modeling and diverse WAM variants. Shared experts capture common dynamics, while atomic capability experts are reused across interfaces so that each interface keeps its own conditional pathway. A plug-in adaptive interface router (PAIR) learns chunk-level interface selection through lightweight supervised target-domain adaptation, keeping the backbone frozen. We evaluate UniW(A)M on LIBERO, RoboTwin 2.0, and three real-world manipulation tasks, where it achieves 88.3% average real-world success versus 76.7% for the strongest baseline. On WorldArena, it leads action-conditioned world modeling on all three tasks. AC-WM co-training contributes to action generation on every action interface, and with supervised target-domain adaptation, PAIR raises OOD success from 63.3% to 76.7%.

88.3%
Real-world average success
vs 76.7% strongest baseline
97.5%
LIBERO average success
99.2% on Spatial
66.6%
RoboTwin 2.0 clean → randomized
+20.6 pts over π0.5
76.7%
OOD success with PAIR
up from 63.3%, backbone frozen
3/3
WorldArena tasks led
action-conditioned prediction
Overview

One model that predicts the world and acts in it.

World models predict how a scene evolves; world-action models generate actions to change it. Both learn from the same embodied trajectories, yet they are usually trained apart. UniW(A)M trains them together in a single mixture-of-experts diffusion Transformer, so the two supervision signals shape one shared representation of the dynamics.

Overview of UniW(A)M
Overview. UniW(A)M unifies action-conditioned world modeling (AC-WM) and four world-action-model interfaces over a shared embodied context. AC-WM predicts future video, while execution adaptively switches among the four WAM interfaces chunk by chunk.
◆

Shared + atomic experts

One shared expert models common dynamics; four atomic experts — VGM, VLA, WM, IDM — each own one predictive role and are reused by every interface that needs it.

⇄

CATR routing

Capability-Aligned Token Routing sends video and action target tokens to their own eligible experts and learns mixture weights within each set.

◎

PAIR at deployment

A plug-in router picks the interface for every action chunk — fast action paths for reaching, video-aware paths for contact — with the backbone frozen.

Method

Five interfaces. Four atomic capabilities. One backbone.

Each interface is defined by a video–action visibility mask. Pick one to see which experts it activates and what its tokens can attend to during training and at deployment.

Atomic capabilities
Experts · training
Experts · inference
Deployment
Training mask
Inference mask
video queryaction queryblockedrow = query · column = key · f₀ observed frame
VGMaction-free video prediction
VLAdirect action generation
WMaction-conditioned video prediction
IDMvideo-conditioned action generation
UniW(A)M architecture
Architecture. (a) Stage I — unified learning: five interfaces share a diffusion Transformer whose FFN layers are replaced by a CATR mixture of one shared and four atomic experts. (b) Stage II — PAIR adaptation: with the backbone frozen, PAIR learns interface selection from offline MAE-based supervision.
Expert activation during training and inference
Expert activation. Shared experts serve every interface. Action-Centric-WAM drops WM at inference; Action-Only-WAM keeps VGM only for the conditioning frame.
Real-world manipulation

Every interface beats the strongest baseline.

Three dual-arm tasks on an AgileX platform: deformable cloth folding, contact-rich box folding and long-horizon tabletop cleaning. 20 trials per task.

Folding clothes
Deformable

Folding Clothes

605 episodes · 28.8 h

Folding box
Contact-rich

Folding Box

1,217 episodes · 16.4 h

Tabletop cleaning
Long-horizon

Tabletop Cleaning

2,426 episodes · 14.2 h

Success rate (%)

Baselines in grey; UniW(A)M interfaces in color.

Real-world tasks in motion

Head-camera episodes from our dual-arm dataset and OOD test scenes. One garment, box or table per clip; garments start laid flat.

Simulation benchmarks

Strong in simulation, robust to scene shift.

Trained only on clean RoboTwin 2.0 scenes, UniW(A)M reaches 66.6% on randomized scenes — 20.6 points above the strongest listed baseline.

RoboTwin 2.0 · 50 tasks

Clean-only training.

LIBERO

Success rate (%) · best in bold, second underlined.

RoboTwin 2.0 tasks

Twelve of the 50 dual-arm tasks, shown with clean-scene episodes from the benchmark data.

LIBERO tasks

Two tasks from each suite.

World modeling · WorldArena

The same model is also the best world model here.

In AC-WM mode, UniW(A)M predicts future video from given actions. It achieves the highest EWMScore-P on all three tasks against Cosmos-Predict 2.5 and Ctrl-World.

Action-driven and interaction metrics

0–100 scale, higher is better.

EWMScore-P

Aggregate score per task.

Action-conditioned future prediction

Given the same action sequence, UniW(A)M keeps garments, cardboard edges and bottle labels consistent; red circles mark baseline artifacts.

Qualitative world-model comparison
PAIR · Plug-in Adaptive Interface Router

Choose the right interface for every chunk.

The jointly trained interfaces trade speed for foresight. PAIR is a lightweight classifier on frozen UniW(A)M features that picks one interface per action chunk, trained from 2,000 target-domain chunks per task.

Out-of-distribution success (%)

Unseen garments, box types and cluttered backgrounds.

Which interface does PAIR pick?

Selection share by execution phase.

Fast action pathways (Action-Centric, Action-Only) take 60–75% of choices while reaching; video-aware pathways (Joint, IDM) take 60–70% during contact and 70–90% during core manipulation.

Success vs. completion time

OOD average; time over successful trials.

Inference latency per step

Full chunk time ÷ 16 denoising steps (ms).

Routing vs. heuristics and LoRA

Folding Clothes, OOD success (%). LoRA adapts one fixed interface on the same target-domain chunks.

Critical-operation case study
Why switch at chunk boundaries? Successful (top) and failed (bottom) critical operations in each task. Failures are local execution errors — a missed grasp edge, a flap pushed the wrong way — that a better-suited interface avoids.
Inside UniW(A)M

Training together makes every interface stronger.

The interfaces share one set of dynamics, and each expert takes on a clear role. All results below use the same data and total training budget.

More interfaces, stronger policies

Folding Clothes. Interfaces join training one at a time.

Joint-WAM rises from 50% to 90% as more interfaces train together, even with fewer updates per interface.

Capability-aligned routing

Average success over three tasks (%).

Sending video and action tokens to their own experts (CATR) takes Action-Centric from 10.0% to 81.7%, and AC-WM co-training adds another 15–43 points.

Every expert has a role

Folding Clothes success (%) with one expert left out, 20K matched steps.

Action experts carry 55–70 points of success and world-prediction experts 20–45. The WM expert also carries AC-WM: without it, EWMScore-P drops from 61.7 to 31.2.

How the gate splits the work

Gate-weighted utilization, shared vs. atomic (%).

Action tokens lean more on their atomic expert than video tokens do in every WAM.

Mixture of experts vs. a dense backbone

Folding Clothes success (%). A dense DiT of similar size (16.4B vs 16.2B), or one with matched initialization, stays behind UniW(A)M.

Qualitative rollouts by interface

Frames from the four WAM interfaces on each real-world task.

Real-world sequences
Citation

BibTeX

@article{xu2026uniwam,
  title   = {UniW(A)M: A Unified Mixture-of-Experts Framework for
             World and World-Action Models},
  author  = {Xu, Jingbo and Fang, Jianjie and Li, Zhuohang and Song, Zihao and
             Wang, Zhaolu and Liu, Xvyuan and Zhang, Xin and Su, Haisheng and
             Tang, Yinzhou and Gao, Chen and Wu, Wei and Li, Yong},
  journal = {arXiv preprint arXiv:2610.18697},
  year    = {2026}
}