TactoHand

Learning Geometry-Based Dense Hand-Surface
Interaction Fields from Video

Ziheng XiWeibo LiJiayi YuSipeng ZhengWanpeng ZhangHaiweng XuHao LuoChuan MaoZongqing Lu

Tsinghua University   /   Peking University   /   BeingBeyond

Loading film…Download 1080p
0:00 / 3:59

Interactive predictions

Choose a real sequence from six datasets. Compare contact, proximity and force in the image and on a freely rotating hand surface, then inspect the aggregate 6D wrench.

Run your video on Hugging Face to explore predictions on your own footage.

Choose an interaction Loading sequences…
Surface field
Vector overlays
0.00 / 3.10 s
Camera viewOriginal orientation
L · anatomical left
R · anatomical right
3D hand surfaceDrag to rotate · scroll to zoom
L · anatomical left
R · anatomical right
Hover or tap the mesh to inspect a vertex.
xyz camera frame
Contact probability
00.51

Probability at each hand vertex. Surface color and force arrows can be inspected together.

Loading geometry and predictions…10 Hz predictions · interpolated display

Prediction targets

Each hand is represented by 778 MANO vertices. Vertices predicted to be in contact are shown in red. Proximity is defined as an exponential function of the distance from each vertex to the object surface. Force components are predicted in a local coordinate frame at each vertex.

Contact

Contact locations on the hand.

Binary contact predictions on two hands grasping a controller, with the input frame from HOCap.
No contactContact
Predicted contact probability, c hat t i, between zero and one
A dense contact probability at each vertex.

Proximity

Distance from each hand vertex to nearby surfaces.

Continuous distance predictions on a hand next to a board, with the input frame from TACO.
021030100+
Distance · mm
Predicted proximity, r hat t i, between zero and one
Proximity shown as distance in millimeters.

Vertex force

Force direction and magnitude at each vertex.

Predicted vertex-force vectors on a drill grasp, with the input frame from DexYCB.
Vertex forces · N
Predicted local force components, ell hat t i, in R cubed
Three components in a local surface basis.

Contact, proximity and vertex-force predictions from HOCap, TACO and DexYCB. View full-resolution figure

Dataset construction at scale

Contact and proximity labels are generated from all datasets. Force labels are generated from datasets that provide object information.

24curated sources
49,018,969captured frames across views
282,874curated sequence entries
Example from ARCTICARCTIC
Example from TACOTACO
Example from H2OH2O
Example from HOCapHOCap
Example from DexYCBDexYCB
Example from HOI4DHOI4D
Example from HOT3DHOT3D
Example from OakInk v2OakInk v2
Example from GigaHandsGigaHands
Example from HOGraspNetHOGraspNet
Example from OakInk v1OakInk v1
Example from HO3Dv3HO3Dv3
Example from H2O-3DH2O-3D
Example from HO3Dv2HO3Dv2
Example from ReInterHandReInterHand
Example from InterHand2.6MInterHand2.6M
Example from HICHIC
Example from DECAFDECAF
Example from MOWMOW
Example from ObManObMan
Example from Show3DShow3D
Example from Hi4DHi4D
Example from RICHRICH
Example from PROXPROX
24 sources of human interaction. Object manipulation, hand–hand interaction, interaction with people and contact with scenes, across camera views and capture settings.View full-resolution figure

Contact and proximity label generation

Distance-only labels can include back-facing vertices and miss vertices displaced inside an object. Surface-normal filtering and inside-mesh inclusion address these cases.

Contact-label construction: surface-normal filtering and inside-mesh inclusion.
Surface-normal filtering removes back-facing vertices; inside-mesh inclusion recovers contacts displaced into the object. View full resolution

Filter implausible contacts. Surface-normal filtering removes back-facing false positives.

Recover missed contacts. Inside-mesh inclusion retains contact vertices affected by penetration.

Comparison column headings
Mug grasp: comparison of geometry-derived contact labels
Tool grasp: comparison of geometry-derived contact labels

Back of hand / palm in each pair.

Contact on the grasping surface

Full scene and hand crop from the annotation comparison

HACO

HACO contact labels: back and palm

TactoHand

TactoHand contact labels: back and palm

Recovering missed contact

Full scene and hand crop from the annotation comparison

HACO

HACO contact labels: back and palm

TactoHand

TactoHand contact labels: back and palm
Contact-label comparisons. HACO’s distance-only labels and TactoHand’s geometric labels are compared on mug and tool grasps. View full resolution

From object motion to force labels

Inverse dynamics converts object trajectories into a target wrench. A sequence-level optimization allocates vertex forces using surface geometry, local friction cones and temporal smoothness, with an environment-support term for supporting surfaces.

Force allocation: friction cones, signed surface distance, local force directions and distance-dependent weighting.
Local polyhedral friction cone
Signed distance to the object
Per-vertex hand-local frame
Force allocation by signed distance

Rigid-object force allocation

Minimize distance-weighted squared vertex forces, squared changes between adjacent frames, and a support regularizer. At each time, the vertex and support wrenches must sum to the motion-derived target. At each active vertex, the normal component is nonnegative. For each of the six facet directions, the force projected onto that direction is at most the normal component times the friction coefficient times cosine of pi over six. The facet directions are spaced sixty degrees apart in the contact plane. Inactive vertex forces are zero. Support is nonnegative and zero without environmental contact.

qt,i weights vertices by signed distance, and Gt,i maps each vertex force to a force-and-torque pair. The two local tangent directions define the six friction facets. The support term penalizes the magnitude and frame-to-frame changes of et.

Friction cones constrain force directions. Signed distance selects candidate vertices and sets their allocation penalties. View full resolution

Network architecture

A DINOv3 image encoder and temporal transformer process the video clip. Three task-specific queries decode contact, proximity and local force components on both hands.

TactoHand paper overview: temporal video encoding, three interaction-field queries and motion-level force supervision
One temporal model, three aligned fields. The architecture connects video features to contact, proximity and force, with a shared hand surface and motion-level supervision. View full resolution

From dense forces to a 6D wrench

Net force is the sum of vertex forces. Net torque sums their moments about the object center of mass, with all quantities expressed in the same coordinate frame.

Net force equals the sum of predicted vertex forcesNet torque is the sum of moments of predicted vertex forces about the object center of mass

Differentiable motion supervision

Predicted vertex forces are aggregated into an object wrench. A differentiable integrator propagates the object state, and position and orientation errors supervise force prediction.

Differentiable rigid-body supervision across image timestamps and intermediate force intervals.
Each 10 Hz image transition contains three 30 Hz force intervals. Predicted vertex forces act at the image-aligned interval; intermediate intervals use detached motion-derived reference forces. Pose errors supervise the propagated state. View full resolution

Contact and proximity prediction

TactoHand is evaluated on four benchmarks, including HOCap held out from training. On HOCap, the temporal model reaches 56.03% contact Macro-F1 and 42.43 mm proximity ADE.

Proximity accuracy

ADE / ADE-100 in millimeters ↓. ADE-100 clips predicted and reference distances at 100 mm.

DatasetTactoHandInterField-SFInterField-LSTM
HOCap42.43 / 6.37132.51 / 38.15134.69 / 40.10
ARCTIC (s05, ego)16.25 / 9.3829.65 / 9.8129.68 / 9.77
DexYCB20.11 / 2.81128.99 / 32.83127.43 / 31.05
MOW49.96 / 12.70108.13 / 33.15110.08 / 35.00

Swipe horizontally for the full table.

Contact accuracy

Macro-F1 / Micro-F1 in percent ↑.

DatasetTactoHandHACOEgoPHIContactPrompt
HOCap56.03 / 68.5418.0 / 19.511.4 / 16.2—
ARCTIC (s05, ego)51.91 / 63.8242.2 / 55.831.8 / 38.5—
DexYCB63.64 / 71.3834.3 / 41.52.7 / 4.3—
MOW35.91 / 45.2134.5 / 41.512.4 / 28.729.7 / 36.5

Swipe horizontally for the full table.

TactoHand results use the temporal checkpoint trained with rollout supervision. DexYCB uses the official S0 test split, ARCTIC uses s05 egocentric frames, and HOCap is held out from training. MOW uses hand crops. See the paper for metric definitions.

Contact prediction comparisons

All five contact prediction examples: input, ContactPrompt, EgoPHI, HACO, TactoHand and ground truth; palmar and dorsal views

Palm / back of hand in each pair.

Box grasp

Box grasp: full frame and hand crop

ContactPrompt

Box grasp: ContactPrompt, palm and back of hand

EgoPHI

Box grasp: EgoPHI, palm and back of hand

HACO

Box grasp: HACO, palm and back of hand

TactoHand

Box grasp: TactoHand, palm and back of hand

Ground truth

Box grasp: Ground truth, palm and back of hand

Tabletop interaction

Tabletop interaction: full frame and hand crop

ContactPrompt

Tabletop interaction: ContactPrompt, palm and back of hand

EgoPHI

Tabletop interaction: EgoPHI, palm and back of hand

HACO

Tabletop interaction: HACO, palm and back of hand

TactoHand

Tabletop interaction: TactoHand, palm and back of hand

Ground truth

Tabletop interaction: Ground truth, palm and back of hand

Container grasp

Container grasp: full frame and hand crop

ContactPrompt

Container grasp: ContactPrompt, palm and back of hand

EgoPHI

Container grasp: EgoPHI, palm and back of hand

HACO

Container grasp: HACO, palm and back of hand

TactoHand

Container grasp: TactoHand, palm and back of hand

Ground truth

Container grasp: Ground truth, palm and back of hand

Scissors interaction

Scissors interaction: full frame and hand crop

ContactPrompt

Scissors interaction: ContactPrompt, palm and back of hand

EgoPHI

Scissors interaction: EgoPHI, palm and back of hand

HACO

Scissors interaction: HACO, palm and back of hand

TactoHand

Scissors interaction: TactoHand, palm and back of hand

Ground truth

Scissors interaction: Ground truth, palm and back of hand

Rectangular object grasp

Rectangular object grasp: full frame and hand crop

ContactPrompt

Rectangular object grasp: ContactPrompt, palm and back of hand

EgoPHI

Rectangular object grasp: EgoPHI, palm and back of hand

HACO

Rectangular object grasp: HACO, palm and back of hand

TactoHand

Rectangular object grasp: TactoHand, palm and back of hand

Ground truth

Rectangular object grasp: Ground truth, palm and back of hand
Contact prediction across five interactions. Every compared method and the geometry-derived ground truth are shown, with palm and back-of-hand views. Blue marks contact. Open the full-resolution figure

Hand–object reconstruction refinement

Image evidence alone can leave the hand and object misaligned. Contact and proximity add complementary geometric constraints to reconstruction refinement.

ArtHOI and EasyHOI results

Predicted contact and proximity refine ArtHOI and EasyHOI across DexYCB, HOCap and MOW. Adding TactoHand improves PA-VPE for both methods on all three datasets and yields the lowest hand–object Chamfer distance in each comparison.

Reconstruction accuracy

PA-VPE in mm ↓; CDₕₒ is bidirectional mean squared hand–object Chamfer distance in 10⁻³ m² ↓.

DatasetMethodPA-VPE ↓CDho ↓
+ TactoHandBase+ HACO+ TactoHandBase+ HACO
DexYCBEasyHOI9.0779.9379.5162.00424.14012.552
DexYCBArtHOI4.6267.3176.0271.43111.58713.627
HOCapEasyHOI7.5929.1799.05111.46159.41668.943
HOCapArtHOI6.0736.9846.6841.1045.2345.199
MOWEasyHOI13.93914.83214.27615.26669.08072.638
MOWArtHOI13.10013.57213.26132.41373.99373.147

Swipe horizontally for the full table.

All three HOCap reconstruction examples: mouse, remote control and bowl; ArtHOI, EasyHOI, their HACO and TactoHand variants, and ground truth

Mouse grasp

Input

Mouse grasp: input

Ground truth

Mouse grasp: ground truth
ArtHOI

Base

Mouse grasp: ArtHOI Base

+ HACO

Mouse grasp: ArtHOI + HACO

+ TactoHand

Mouse grasp: ArtHOI + TactoHand
EasyHOI

Base

Mouse grasp: EasyHOI Base

+ HACO

Mouse grasp: EasyHOI + HACO

+ TactoHand

Mouse grasp: EasyHOI + TactoHand

Remote-control grasp

Input

Remote-control grasp: input

Ground truth

Remote-control grasp: ground truth
ArtHOI

Base

Remote-control grasp: ArtHOI Base

+ HACO

Remote-control grasp: ArtHOI + HACO

+ TactoHand

Remote-control grasp: ArtHOI + TactoHand
EasyHOI

Base

Remote-control grasp: EasyHOI Base

+ HACO

Remote-control grasp: EasyHOI + HACO

+ TactoHand

Remote-control grasp: EasyHOI + TactoHand

Bowl grasp

Input

Bowl grasp: input

Ground truth

Bowl grasp: ground truth
ArtHOI

Base

Bowl grasp: ArtHOI Base

+ HACO

Bowl grasp: ArtHOI + HACO

+ TactoHand

Bowl grasp: ArtHOI + TactoHand
EasyHOI

Base

Bowl grasp: EasyHOI Base

+ HACO

Bowl grasp: EasyHOI + HACO

+ TactoHand

Bowl grasp: EasyHOI + TactoHand
Reconstruction across three interactions. Mouse, remote-control and bowl interactions compare ArtHOI and EasyHOI, their HACO and TactoHand variants, and ground truth. Red circles identify contact artifacts in the HACO variants. Open the full-resolution figure

Feel free to use our data, model and code.

We hope this work helps you build and study embodied AI systems.

Paper, code, checkpoints and data

Citation
@misc{xi2026tactohand,
  title={TactoHand: Learning Geometry-Based Dense Hand-Surface Interaction Fields from Video},
  author={Xi, Ziheng and Li, Weibo and Yu, Jiayi and Zheng, Sipeng and Zhang, Wanpeng and Xu, Haiweng and Luo, Hao and Mao, Chuan and Lu, Zongqing},
  year={2026},
  note={Preprint}
}