TactoHand
Learning Geometry-Based Dense Hand-Surface
Interaction Fields from Video
Tsinghua University / Peking University / BeingBeyond
Interactive predictions
Choose a real sequence from six datasets. Compare contact, proximity and force in the image and on a freely rotating hand surface, then inspect the aggregate 6D wrench.
Run your video on Hugging Face to explore predictions on your own footage.
Probability at each hand vertex. Surface color and force arrows can be inspected together.
- —
- —
- —
- —
- —
- —
Cyan: force direction · violet: torque axis. Lengths are scaled for visibility; read magnitudes above. Directions below 0.01 N / 0.001 N·m are hidden.
Prediction targets
Each hand is represented by 778 MANO vertices. Vertices predicted to be in contact are shown in red. Proximity is defined as an exponential function of the distance from each vertex to the object surface. Force components are predicted in a local coordinate frame at each vertex.
Contact, proximity and vertex-force predictions from HOCap, TACO and DexYCB. View full-resolution figure
Dataset construction at scale
Contact and proximity labels are generated from all datasets. Force labels are generated from datasets that provide object information.
ARCTIC
TACO
H2O
HOCap
DexYCB
HOI4D
HOT3D
OakInk v2
GigaHands
HOGraspNet
OakInk v1
HO3Dv3
H2O-3D
HO3Dv2
ReInterHand
InterHand2.6M
HIC
DECAF
MOW
ObMan
Show3D
Hi4D
RICH
PROXContact and proximity label generation
Distance-only labels can include back-facing vertices and miss vertices displaced inside an object. Surface-normal filtering and inside-mesh inclusion address these cases.

Filter implausible contacts. Surface-normal filtering removes back-facing false positives.
Recover missed contacts. Inside-mesh inclusion retains contact vertices affected by penetration.



Back of hand / palm in each pair.
Contact on the grasping surface

HACO

TactoHand

Recovering missed contact

HACO

TactoHand

From object motion to force labels
Inverse dynamics converts object trajectories into a target wrench. A sequence-level optimization allocates vertex forces using surface geometry, local friction cones and temporal smoothness, with an environment-support term for supporting surfaces.
Rigid-object force allocation
qt,i weights vertices by signed distance, and Gt,i maps each vertex force to a force-and-torque pair. The two local tangent directions define the six friction facets. The support term penalizes the magnitude and frame-to-frame changes of et.
Network architecture
A DINOv3 image encoder and temporal transformer process the video clip. Three task-specific queries decode contact, proximity and local force components on both hands.

From dense forces to a 6D wrench
Net force is the sum of vertex forces. Net torque sums their moments about the object center of mass, with all quantities expressed in the same coordinate frame.
Differentiable motion supervision
Predicted vertex forces are aggregated into an object wrench. A differentiable integrator propagates the object state, and position and orientation errors supervise force prediction.

Contact and proximity prediction
TactoHand is evaluated on four benchmarks, including HOCap held out from training. On HOCap, the temporal model reaches 56.03% contact Macro-F1 and 42.43 mm proximity ADE.
Proximity accuracy
ADE / ADE-100 in millimeters ↓. ADE-100 clips predicted and reference distances at 100 mm.
| Dataset | TactoHand | InterField-SF | InterField-LSTM |
|---|---|---|---|
| HOCap | 42.43 / 6.37 | 132.51 / 38.15 | 134.69 / 40.10 |
| ARCTIC (s05, ego) | 16.25 / 9.38 | 29.65 / 9.81 | 29.68 / 9.77 |
| DexYCB | 20.11 / 2.81 | 128.99 / 32.83 | 127.43 / 31.05 |
| MOW | 49.96 / 12.70 | 108.13 / 33.15 | 110.08 / 35.00 |
Swipe horizontally for the full table.
Contact accuracy
Macro-F1 / Micro-F1 in percent ↑.
| Dataset | TactoHand | HACO | EgoPHI | ContactPrompt |
|---|---|---|---|---|
| HOCap | 56.03 / 68.54 | 18.0 / 19.5 | 11.4 / 16.2 | — |
| ARCTIC (s05, ego) | 51.91 / 63.82 | 42.2 / 55.8 | 31.8 / 38.5 | — |
| DexYCB | 63.64 / 71.38 | 34.3 / 41.5 | 2.7 / 4.3 | — |
| MOW | 35.91 / 45.21 | 34.5 / 41.5 | 12.4 / 28.7 | 29.7 / 36.5 |
Swipe horizontally for the full table.
TactoHand results use the temporal checkpoint trained with rollout supervision. DexYCB uses the official S0 test split, ARCTIC uses s05 egocentric frames, and HOCap is held out from training. MOW uses hand crops. See the paper for metric definitions.
Contact prediction comparisons

Palm / back of hand in each pair.
Box grasp

ContactPrompt

EgoPHI

HACO

TactoHand

Ground truth

Tabletop interaction

ContactPrompt

EgoPHI

HACO

TactoHand

Ground truth

Container grasp

ContactPrompt

EgoPHI

HACO

TactoHand

Ground truth

Scissors interaction

ContactPrompt

EgoPHI

HACO

TactoHand

Ground truth

Rectangular object grasp

ContactPrompt

EgoPHI

HACO

TactoHand

Ground truth

Hand–object reconstruction refinement
Image evidence alone can leave the hand and object misaligned. Contact and proximity add complementary geometric constraints to reconstruction refinement.
ArtHOI and EasyHOI results
Predicted contact and proximity refine ArtHOI and EasyHOI across DexYCB, HOCap and MOW. Adding TactoHand improves PA-VPE for both methods on all three datasets and yields the lowest hand–object Chamfer distance in each comparison.
Reconstruction accuracy
PA-VPE in mm ↓; CDₕₒ is bidirectional mean squared hand–object Chamfer distance in 10⁻³ m² ↓.
| Dataset | Method | PA-VPE ↓ | CDho ↓ | ||||
|---|---|---|---|---|---|---|---|
| + TactoHand | Base | + HACO | + TactoHand | Base | + HACO | ||
| DexYCB | EasyHOI | 9.077 | 9.937 | 9.516 | 2.004 | 24.140 | 12.552 |
| DexYCB | ArtHOI | 4.626 | 7.317 | 6.027 | 1.431 | 11.587 | 13.627 |
| HOCap | EasyHOI | 7.592 | 9.179 | 9.051 | 11.461 | 59.416 | 68.943 |
| HOCap | ArtHOI | 6.073 | 6.984 | 6.684 | 1.104 | 5.234 | 5.199 |
| MOW | EasyHOI | 13.939 | 14.832 | 14.276 | 15.266 | 69.080 | 72.638 |
| MOW | ArtHOI | 13.100 | 13.572 | 13.261 | 32.413 | 73.993 | 73.147 |
Swipe horizontally for the full table.

Mouse grasp
Input

Ground truth

ArtHOI
Base

+ HACO

+ TactoHand

EasyHOI
Base

+ HACO

+ TactoHand

Remote-control grasp
Input

Ground truth

ArtHOI
Base

+ HACO

+ TactoHand

EasyHOI
Base

+ HACO

+ TactoHand

Bowl grasp
Input

Ground truth

ArtHOI
Base

+ HACO

+ TactoHand

EasyHOI
Base

+ HACO

+ TactoHand

Feel free to use our data, model and code.
Paper, code, checkpoints and data
Citation
@misc{xi2026tactohand,
title={TactoHand: Learning Geometry-Based Dense Hand-Surface Interaction Fields from Video},
author={Xi, Ziheng and Li, Weibo and Yu, Jiayi and Zheng, Sipeng and Zhang, Wanpeng and Xu, Haiweng and Luo, Hao and Mao, Chuan and Lu, Zongqing},
year={2026},
note={Preprint}
}

