Evaluating Tactile Foundation Models for End-to-End Imitation Learning in Contact-Rich Manipulation

Tactile foundation models encode tactile observations for Diffusion Policy, evaluated on five real-world and two simulation tasks.

How can tactile foundation models improve robot manipulation?

Comparing pretrained representations and adaptation strategies for imitation learning.

5,136 real-world rollouts

5 + 2 real-world + simulation tasks

7 pretrained encoder variants

Video

Abstract

Tactile sensing plays an important role in imitation learning for contact-rich manipulation by providing direct observations of physical contact with objects. Recently, several tactile foundation models (TFMs) have been proposed to learn general-purpose tactile representations through pretraining on large-scale tactile datasets. However, how these pretrained tactile representations should be leveraged for end-to-end imitation learning remains insufficiently explored. In this work, we integrate multiple TFMs into Diffusion Policy and systematically evaluate their effectiveness for contact-rich manipulation under common experimental settings. We further investigate adaptation strategies, including encoder fine-tuning and task-specific mid-training, to understand how pretrained tactile representations can be effectively used for downstream imitation learning. Through extensive evaluation across diverse manipulation tasks in both real-world and simulation environments, including more than 5,000 real-world policy rollouts, we provide empirical insights into the use of TFMs for end-to-end imitation learning.

Method

We compare tactile encoders and adaptation strategies within a common Diffusion Policy architecture.

Tactile images are encoded by a pretrained tactile encoder, projected to a shared feature size, and combined with proprioception to condition a diffusion action policy. — A common policy architecture for comparing tactile representations.

Compare representations

Compare four Sparsh variants, AnyTouch, T3, and UniT with camera and tactile ResNet-18 baselines. All tactile encoders produce 64-dimensional features.

Adapt the encoder

Compare training from scratch, freezing, and end-to-end fine-tuning with Sparsh DINOv2.

Explore mid-training

Adapt the encoder by predicting future tactile features from task demonstrations, using actions, a contrastive objective, and an EMA teacher.

Experiments

32 demonstrations per task. Videos show the robot view and both tactile sensors.

Real-world manipulation

A UR5e arm with a Robotiq 2F-140 gripper and GelSight Mini sensors on both fingers.

Marker Drawing

Draw two parallel lines on a whiteboard while maintaining contact.

Heavy Box Sorting

Lift three visually identical boxes and place the weighted box into a basket.

Paper Box Handling

Grasp and move a deformable paper container without crushing it.

Spoon Sorting · Handle grasp

Sort a spoon into the corresponding cup based on the grasped part.

Spoon Sorting · Bowl grasp

Sort the spoon when grasped by its bowl.

Pen Insertion

Insert a ballpoint pen refill into a pen barrel.

Simulation manipulation

A Sawyer arm with optical tactile sensing simulated using TACTO and PyBullet.

Plate Insertion

Insert a plate into a narrow slit.

Spoon Sorting

Sort the spoon based on the grasped part.

Quantitative Results

Tactile representations

Tactile input outperforms the camera baseline on average. Sparsh DINOv2 leads in the real world; T3 leads in simulation. Gains from pretraining vary by model and task.

Mean task success rate (%) · Table I in the paper
Policy	Real world	Simulation
DP-ResNetCamera	39.6	32.9
DP-ResNetTactile	54.2	77.1
DP-SparshMAE	57.6	73.8
DP-SparshDINO	58.3	70.8
DP-SparshDINOv2	67.4	85.0
DP-SparshIJEPA	56.9	65.0
DP-AnyTouch	47.9	82.1
DP-T3	48.6	87.5
DP-UniT	50.7	81.7

Four real-world tasks and two simulation tasks; three seeds, with 12 and 20 rollouts per seed and task, respectively. Pretrained encoders are fine-tuned. Pen Insertion appears only in the adaptation experiments.

Fine-tuning improves Sparsh DINOv2

Across four real-world tasks, fine-tuning reaches 81.7% average success, compared with 65.3% for a frozen encoder and 38.8% for training from scratch.

Fine-tuning comparison: average success rates are 44.3% for ResNetTactile, 38.8% from scratch, 65.3% frozen, and 81.7% fine-tuned. — Mean and standard deviation over 15 training seeds, with 12 evaluation rollouts per seed.

Mid-training has task-dependent effects

Mid-training raises average success from 51.2% to 54.8%, improving three tasks but reducing performance on Pen Insertion.

Task-specific mid-training improves average success from 51.2% to 54.8%, with benefits varying by task. — Mean and standard deviation over 10 training seeds, with 12 evaluation rollouts per seed.

Task objects differ between experiments, so compare success rates within each evaluation.

BibTeX

@misc{murooka2026tactile,
  title = {Evaluating Tactile Foundation Models for End-to-End Imitation Learning in Contact-Rich Manipulation},
  author = {Masaki Murooka and Ryoichi Nakajo and Tomohiro Motoda and Koshi Makihara and Hanbit Oh and Keisuke Shirai and Yukiyasu Domae},
  year = {2026},
  eprint = {2611.00957},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url = {https://arxiv.org/abs/2611.00957}
}