VTAMVideo-Tactile-Action Models for Complex Physical Interaction Beyond VLAs

Haoran Yuan♦*‡, Weigang Yi♦*, Zhenyu Zhang♠*, Wendi Chen♥*,
Yuchen Mo♦, Jiashi Yin♦, Xinzhuo Li♦, Xiangyu Zeng♦,
Chuan Wen♥, Cewu Lu♥, Katherine Driggs-Campbell♦, Ismini Lourentzou♦†
♦University of Illinois Urbana-Champaign♠Stanford University♥Shanghai Jiao Tong University
‡Project lead *Equal contribution †Corresponding author
PaperarXivCode (coming soon)

TL;DR

We introduce VTAM, a video–tactile–action model for contact-rich robotic manipulation. By combining visual observations with touch, VTAM predicts actions and object dynamics while capturing contact and deformation, providing physically grounded guidance for tasks where vision alone is insufficient.

Chip pick-and-place

VTAM uses touch to guide precise contact during chip manipulation.

Cucumber peeling

VTAM uses visual and tactile feedback to guide contact while peeling a cucumber.

Whiteboard wiping

Visual and tactile feedback guide sustained contact during wiping.

Abstract

Video-Action Models (VAMs) have emerged as a promising framework for embodied intelligence, learning implicit world dynamics from raw video streams to produce temporally consistent action predictions. Although such models demonstrate strong performance on long-horizon tasks through visual reasoning, they remain limited in contact-rich scenarios where critical interaction states are only partially observable from vision alone. In particular, fine-grained force modulation and contact transitions are not reliably encoded in visual tokens, leading to unstable or imprecise behaviors. To bridge this gap, we introduce the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a complementary grounding signal. VTAM augments a pretrained video transformer with tactile streams via a lightweight modality transfer finetuning, enabling efficient cross-modal representation learning without tactile-language paired data or independent tactile pretraining. To stabilize multimodal fusion, we introduce a tactile regularization loss that enforces balanced cross-modal attention, preventing visual latent dominance in the action model. VTAM demonstrates superior performance in contact-rich manipulation, maintaining a robust success rate of 90% on average. In challenging scenarios such as potato chip pick-and-place requiring high-fidelity force awareness, VTAM outperforms the pi0.5 baseline by 80%. Our findings demonstrate that integrating tactile feedback is essential for correcting visual estimation errors in world action models, providing a scalable approach to physically grounded embodied foundation models.

Contributions

  • VTAM grounds contact-rich manipulation in visual and tactile signals. We introduce a Video-Tactile Action Model that augments a pretrained video transformer with tactile perception. This complements visual observations with force and contact information that vision alone cannot reliably encode.
  • Modality transfer fine-tuning adds tactile perception efficiently. The model learns cross-modal representations without tactile–language paired data or independent tactile pretraining. Its first stage learns joint visuo-tactile predictive dynamics through multi-view diffusion over two RGB views and one GelSight stream.
  • Tactile regularization and joint action–state–force prediction preserve sensitivity. A regularization loss balances cross-modal attention to prevent visual latents from dominating. In the control stage, conditional diffusion jointly predicts actions, states, and a deformation-derived virtual force proxy, preserving tactile sensitivity during multimodal optimization.
  • Real-world experiments demonstrate improved contact-rich control. VTAM achieves 90% average success across chip pick-and-place, cucumber peeling, and whiteboard wiping, with task success rates of 90%, 85%, and 95%, respectively. On potato chip pick-and-place, it exceeds the vision-only pi 0.5 baseline by 80 percentage points.

VTAM Method

VTAM adds tactile perception to a pretrained video model and uses the resulting interaction representation to predict actions. Two RGB views and a GelSight stream capture the scene and contact state; the control model conditions on these features together with past robot actions and states.

VTAM architecture: first-person, third-person, and GelSight tactile inputs condition a joint action, state, and force diffusion model

  1. Stage I · Learn visuo-tactile dynamics

    Multi-view diffusion learns joint predictive dynamics from first-person video, third-person video, and tactile observations in latent space. Modality transfer fine-tuning adds touch without requiring tactile–language paired data or independent tactile pretraining.

  2. Stage II · Learn contact-aware control

    Conditional diffusion jointly predicts future actions, states, and a deformation-derived virtual force proxy. Tactile regularization balances cross-modal attention, preserving sensitivity to contact while preventing visual latents from dominating the optimization.

Quantitative Results

Contact-rich manipulation

Task success rates for chip pick-and-place, cucumber peeling, and whiteboard wiping. VTAM reaches 90%, 85%, and 95%, respectively. Each model is evaluated over 80 trials.

Chip pick-and-place

0%25%50%75%100%Genie Envisioner0%Chip pick-and-place · Genie Envisioner: 0%pi0.5 (Vision)10%Chip pick-and-place · pi0.5 (Vision): 10%pi0.5 + Tactile5%Chip pick-and-place · pi0.5 + Tactile: 5%VTAM (Ours)90%Chip pick-and-place · VTAM (Ours): 90%

Cucumber peeling

0%25%50%75%100%Genie Envisioner0%Cucumber peeling · Genie Envisioner: 0%pi0.5 (Vision)0%Cucumber peeling · pi0.5 (Vision): 0%pi0.5 + Tactile0%Cucumber peeling · pi0.5 + Tactile: 0%VTAM (Ours)85%Cucumber peeling · VTAM (Ours): 85%

Whiteboard wiping

0%25%50%75%100%Genie Envisioner2.5%Whiteboard wiping · Genie Envisioner: 2.5%pi0.5 (Vision)0%Whiteboard wiping · pi0.5 (Vision): 0%pi0.5 + Tactile0%Whiteboard wiping · pi0.5 + Tactile: 0%VTAM (Ours)95%Whiteboard wiping · VTAM (Ours): 95%
All success rates
Model Chip Peel Wipe
Genie Envisioner 0% 0% 2.5%
pi0.5 (Vision) 10% 0% 0%
pi0.5 + Tactile 5% 0% 0%
VTAM (Ours) 90% 85% 95%

Qualitative Results

Chip Pick-and-Place

VTAM (Ours)
Genie Envisioner
pi0.5 (Vision)
pi0.5 + Tactile

Cucumber Peeling

VTAM (Ours)
Genie Envisioner
pi0.5 (Vision)
pi0.5 + Tactile

Whiteboard Wiping

VTAM (Ours)
Genie Envisioner
pi0.5 (Vision) / pi0.5 + Tactile

Model Behavior Diversity

VTAM (Ours) - Failure Case
VTAM (Ours) - Retrial Recovery

BibTeX

@article{vtam2026,
  title={VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs},
  author={Yuan, Haoran and Yi, Weigang and Zhang, Zhenyu and Chen, Wendi and Mo, Yuchen and Yin, Jiashi and Li, Xinzhuo and Zeng, Xiangyu and Wen, Chuan and Lu, Cewu and Driggs-Campbell, Katherine and Lourentzou, Ismini},
  journal={arXiv},
  year={2026}
}