PEWM
Primitive execution sequence for a robot picking up a tape measure

Primitive-centric robot learning

Learning Primitive Embodied World Models

Towards Scalable Robotic Learning

Qiao Sun, Liujia Yang, Wei Tang, Wei Huang, Kaixin Xu, Yongchao Chen, Mingyu Liu, Jiange Yang, Haoyi Zhu, Yating Wang, Tong He, Yilun Chen, Xili Dai, Nanyang Ye, Qinying Gu

Read paper Code Models
↓

The central observation

Behavior is combinatorial.
Motion primitives are not.

Embodied trajectories are sparse and expensive, while the space of reusable primitive motions is comparatively small. PEWM restricts generation to short, semantically atomic transitions, creating a denser interface between language, video, and action.

A high-level VLM acts as the planner. A compact video world model acts as the fast, reactive dynamics module. The two meet through language and spatial start-goal grounding.

Method

Plan locally. Generate precisely. Compose indefinitely.

PEWM framework with VLM planning, video generation, and 6-DoF pose extraction
The closed-loop system grounds each primitive in the current observation, predicts 32 future frames, recovers a 6-DoF trajectory, executes it, and observes again.
  1. 01

    Primitive data

    Five synchronized views, on-the-fly temporal boundaries, one shared language label, and full-arm visibility turn each session into multiple dense training units.

  2. 02

    Spatial generation

    DynamiCrafter predicts a 32-frame visual rollout from the current image, primitive instruction, and signed Gaussian Start-Goal Guidance.

  3. 03

    Visual action

    Gen6D estimates gripper pose from generated RGB, followed by scale alignment, pose filtering, and transformation into Cartesian execution space.

  4. 04

    Closed-loop composition

    Qwen2.5-VL decomposes long tasks and replans after each real execution. Symbolic gripper actions bypass video generation.

Primitive embodied data

Designed for density, not just scale

Every real episode is captured by five cameras and split at teleoperation button events. One instruction supervises every synchronized view. Simulation supplies clean kinematics; real data supplies appearance and true dynamics.

11,465
real primitives
7,326
simulated primitives
5.8
primitives per session
29×
collection efficiency
Real robot collection workspace with five labeled camera viewpoints
Five synchronized views capture the full arm, workspace, and wrist motion during teleoperation.
PEWM annotation interface showing five synchronized robot views and one shared instruction
One primitive instruction is attached to all five synchronized views in the original annotation interface.

Qualitative rollouts

Short motions become reusable visual actions

Selected simulation and real-world generations from the authors' original PrimitiveWorld project page. The labels below follow that source.

Generated simulation rollout for picking an object
Simulation Pick object
Generated simulation rollout for opening a drawer
Simulation Open drawer
Generated simulation rollout for opening a pot lid
Simulation Open pot lid
Generated real-world rollout for picking up a tape measure, close view
Real world Pick tape, close view
Generated simulation rollout for opening a box
Simulation Open box
Generated real-world rollout for picking up a tape measure, wide view
Real world Pick tape, wide view

Original media: PrimitiveWorld at commit ce98058. These are project artifacts, not outputs regenerated by the dependency-light smoke backend in this repository.

Results

Compact generation with physical utility

Paper-reported results on 32-frame sequences and real-robot tasks. The released code keeps these separate from integration-only smoke tests.

0.8126SSIM
21.0644PSNR
0.0002FVD
11.45 / 13EPiCS
TaskPlanningFrame realismExecution
Pick up cup18 / 2017 / 2016 / 20
Move cloth16 / 2015 / 2014 / 20
Fold cloth15 / 2014 / 2013 / 20
Model1.4BDynamiCrafter denoiser
Memory11 GBA100 VRAM reported
Rollout16 s32 frames, base model
Causal12 FPSdistilled student reported

Open resources

Reproduce the interfaces today. Fill in the learned artifacts transparently.

PaperarXiv:2508.20840 CodeData, planning, generation, pose, control ModelsDynamiCrafter, causal student, LoRA-P, LoRA-G Model cardHugging Face release template

The implementation labels missing paper artifacts explicitly. Compatibility backends such as Wan are provided for local integration tests, never presented as the trained PEWM model.