Tactile · Coding Agents for Visuo-Tactile Training Environments

Expanded literature review: 56 papers with seed coverage, evidence depth, and research gaps. This page is the initial research-direction summary.

Research direction, literature map, and first experiments · September 7, 2026

literature reviewsynthetic environmentsFlexiTacworld modelsVLA

Research question

Can coding agents discover and verify contact-rich tasks whose training data improves generalization on unseen real-world manipulation, compared with fixed task templates and domain randomization?

The proposed system generates executable tasks, parameterized geometry, and teacher controllers. A calibrated simulator produces synchronized vision and tactile observations; an independent verifier accepts data for policy and world-model training. This is a research proposal, not a demonstrated result.

What prior work already establishes

Work	Relevant evidence	Boundary
VT-Refine Huang et al., 2025	Real demonstrations, TacSL/Isaac Gym tactile simulation, policy fine-tuning, and real evaluation.	Supports calibrated real-to-sim-to-real learning; not general VLA training from purely synthetic data.
RoboGen Wang et al., ICML 2024	Generates tasks, environments, supervision, and skills.	Agent-generated environments and training data alone are established prior art.
UniVTAC Chen et al., 2026	Automated visuo-tactile data generation, injected failures, corrective actions, and contact validity checks.	Optical tactile sensors; not a ready-made FlexiTac pressure-array implementation.
ViTacWorld Huang et al., 2026	Action-conditioned visual/tactile rollouts for policy improvement.	Successful generated-data selection still partly uses manual inspection.
Dream-Tac Lou et al., 2026	Joint future vision, tactile dynamics, and actions.	Adding tactile prediction to a world-action model is not sufficient novelty.

FlexiTac (Huang and Li, 2026) is a piezoresistive pressure-array sensor, distinct from optical tactile cameras. 3D-ViTac (Huang et al., 2024) offers a useful shared spatial representation. ContactWorld (Zhang et al., 2026) motivates evaluating spatial structure and cross-modal representation compatibility.

What the coding agent should generate

Task specification: goals, observation limits, geometric variations, physical parameter ranges, and contact regimes.
Teacher programs: visual approach followed by contact-sensitive grasping, probing, retreat, and correction.
Development experiments: replayable failures and curriculum proposals based on current policy weaknesses.
Data manifests: task version, asset family, seed, calibration, teacher, and filtering provenance.

Keep the accepted contact model, final success evaluator, and held-out test set outside the agent-editable search surface. Privileged simulator states may help teachers and verifiers, but must not silently enter deployed student inputs.

ASPIRE (Lu et al., 2026) provides a program execution/repair/reuse pattern. ENPIRE (Xiao et al., 2026) provides a reset/verification/improvement loop; its gripper torque is a tactile surrogate, not a spatial pressure array.

One state, two observation channels

Task and asset programs → shared physical state → synchronized RGB and pressure-array observations → independent validation → datasets → policies and world models.

Geometry, scale, poses, and timestamps must agree. Shared poses alone do not guarantee force consistency: a geometric penetration-based tactile readout can disagree with the dynamics solver. Validate static indentation, dynamic loading, and grasp replay; preserve raw readings and calibration units.

The FlexiTac IsaacSim example is an interface scaffold: its environment exposes RGB and tactile grids, but inspected code returns zero task reward and no episode termination. Its SDF spring approximation is not the calibrated spring-damper pipeline demonstrated in VT-Refine. This code was inspected, not executed.

SimFoundry can supply optional scene assets; the inspected release explicitly excludes the robotics data-generation and policy-training pipeline. Begin with controlled rigid geometry. Consider PhysTwin or IsaacIPC later for deformable interactions.

First task and modality checks

Candidate: choose a visually specified socket, approach it, and correct an occluded insertion. Vary target placement to require vision; vary hidden held-object alignment to test the usefulness of tactile feedback. First establish that the selected pressure array can distinguish the relevant contact states.

Train vision-only, tactile-only, and fused policies with comparable capacity and training budgets.
Include proprioception and available gripper feedback controls; tactile pressure does not directly measure shear.
Use appearance-matched contact interventions and contact-matched target changes. Avoid color or asset-ID leakage.
Use zeroed/shuffled tactile as diagnostics, not sole proof of modality necessity.
Recheck agent-selected tasks with independent seeds or a second policy family.

Policy Consensus (Chen et al., 2025) shows that naive fusion can hurt while better fusion helps. FELT (Li et al., 2026) provides a vision-predicted tactile baseline; its generator learns from paired data. Predicted tactile is not an independent measurement of hidden contact.

Data products and training

Consumer	Training data	Evaluation
Tactile policy / VLA	Validated successful and recovery segments, language, synchronized observations and actions.	Held-out task success, recovery, contact failures, and load-constraint violations.
Action-conditioned world model	Full interaction sequences including probing, failures, and interventions.	Contact onset, contact regions, calibrated magnitude error, rollout stability, uncertainty, and downstream planning.

Record timestamps, action units, coordinate conventions, sensor validity masks, task/asset lineage, calibration versions, and data origin. Distinguish measured real tactile, physics-simulated tactile, and vision-predicted tactile. Missing touch must not be encoded as measured no-contact.

Start with a lightweight ACT or diffusion policy to test data value. A latent visual/tactile world model can follow. Add pressure tokens and temporal/spatial metadata to a VLA only after validating the data pipeline. MolmoAct2 is one candidate backbone with published SO-100/101 workflows; native pressure-array integration was not established in this review.

Milestones and decision gates

Milestone	Deliverable	Gate
A · Sensor and environment	One rigid contact task with synchronized RGB/tactile/action replay.	Calibration, contact consistency, and timestep sensitivity checks.
B · Teacher and dataset	Approach/contact/recovery controllers and versioned successful/failed trajectories.	Learning baseline, no privileged-state leakage, grouped asset splits.
C · Agent contribution	Fixed randomization vs agent generation vs agent generation with modality screening.	Matched student compute and real-data budget; compare both matched valid-data volume and total generation cost.
D · Transfer	Independent real objects and contact conditions; data scaling curves.	Real improvement with uncertainty reported before expanding to complex deformation or large VLA training.

Agent debug trials, invalid environments, teacher learning, and discarded trajectories count toward generation cost. Final test cases must not guide agent revisions. If hardware evaluation is unavailable, conclusions remain simulation-only.

Evidence and release limitations

This board starts with a literature and source-code review dated September 7, 2026. Paper results are author-reported; proposed tasks and experiment gates are untested. No simulator installation, calibration, policy training, or real deployment was performed. No claim of being the first such system is made.

Before reuse, record component licenses: VT-Refine restricts code use to noncommercial research/evaluation; its data has separate terms. NVIDIA-owned ASPIRE and SimFoundry code is Apache-2.0 with third-party exceptions. A repository-wide license for the inspected FlexiTac IsaacSim scaffold was not established.

Tactile research board · Primary-source review and proposed experiments · No experimental results claimed.