EgoEMG

A multimodal egocentric dataset with bilateral surface EMG and vision for hand pose estimation

EMG-to-pose·vision-to-pose·EMG+vision fusion

Code Checkpoints & Data · Baidu Netdisk Code 8059 Paper — under review

2× replay · episode_000040

0.0 / 0.0 s

wrist sEMG · 2 kHz · both wrists wrist IMU · 120 Hz joint angles · 20 DoF/hand drag the slider or the panels to scrub

01Abstract

Surface electromyography (sEMG) captures muscle activity during hand movement, while egocentric vision observes global hand configuration. These complementary signals are promising for robust hand-pose estimation, but no existing dataset jointly provides synchronized bilateral wrist EMG, egocentric vision, and bimanual pose labels. EgoEMG contains bilateral wristband EMG sampled at 2 kHz, 120 Hz IMU signals, egocentric video, external RGB-D video, and motion-capture-derived wrist and finger joint angles — over 18 hours from 53 participants performing 60 gesture classes. We establish benchmarks for EMG-to-pose, vision-to-pose, and EMG+vision fusion under shared joint-angle targets and Gesture / User / Both generalization splits. Across seven visual backbones, residual fusion reduces average error relative to the corresponding vision-only baseline.

53participants

18+ hsynchronized recordings

60gesture classes30 unimanual · 30 bimanual

2 kHzbilateral wristband EMG

120 HzIMU on each wrist

5sensing modalitiessEMG · IMU · egocentric RGB
RGB-D · motion capture

02Dataset

Each recording synchronizes five sensing modalities: bilateral wristband sEMG at 2 kHz, a 120 Hz IMU on each wrist, a head-mounted egocentric RGB camera, an external RGB-D camera, and a 26-camera optical motion-capture rig providing wrist and finger joint-angle labels for both hands.

Data collection setup: bilateral EMG wristbands, head-mounted egocentric RGB, external RGB-D, and optical motion capture with hand markers — **Data collection setup.** Bilateral EMG wristbands, head-mounted egocentric RGB, external ZED 2i RGB-D, and optical motion capture with hand markers for pose labels.

Representative synchronized samples showing ground-truth hand pose, bilateral EMG, and egocentric RGB frames — **Representative synchronized samples.** Each 6 s window shows ground-truth hand pose, bilateral EMG, egocentric RGB at window center, and external RGB-D.

Train

Gesture

User

Both

participants ↓gestures →

Shared generalization splits

Gesture: Unseen gestures from training participants — tests generalization to new articulations.
User: Unseen participants performing training gestures — tests cross-user generalization.
Both: Unseen participants and unseen gestures — the hardest setting.

All three benchmark tasks are evaluated on identical splits, so EMG, vision, and fusion results are directly comparable.

03Method & Benchmarks

All three tasks share the same 22-DoF joint-angle prediction target per hand and the Gesture / User / Both generalization splits.

1 / 3

EMGFormer architecture — **EMGFormer** regresses per-hand joint angles from wristband EMG windows. EMGFormer-M reaches **13.8°** avg MAE on EgoEMG; EMGFormer-S sets **12.3°** on the EMG2Pose benchmark's User+Stage split — ahead of all prior methods.

Vision-to-pose baseline — **ResNet / ViT backbones** predict joint angles from 256×256 egocentric center-frame hand crops. ResNet-18 reaches **5.85°** avg MAE; the hand-specialized WiLoR backbone is strongest overall.

EMG+vision fusion — **Residual fusion**: the vision branch predicts the pose and the EMG branch predicts a zero-initialized residual on identical center frames. **5.41°** avg MAE with ResNet-18 — 7.5% below vision-only at only +3M parameters.

04Headline Results

Task	Split	Ours	Best baseline
EMG-to-pose on EgoEMG	Avg (Gesture/User/Both)	13.8° (EMGFormer-M) *	15.8° (emg2pose)
EMG-to-pose on EMG2Pose	User+Stage	12.3° (EMGFormer-S)	14.6° (Position/Velocity MT)
Vision-to-pose on EgoEMG	Avg (Gesture/User/Both)	5.85° (ResNet-18) *	—
EMG+vision fusion on EgoEMG	Avg (Gesture/User/Both)	5.41° (ResNet-18) *	5.85° (vision-only)

MAE in degrees. * = released-checkpoint values; unstarred values are paper-reported. Avg is the average over the Gesture/User/Both test splits (per-sample-weighted for EMG-to-pose, unweighted split mean for vision and fusion); the EMG2Pose row follows the benchmark's User+Stage protocol. Full per-method and per-split tables are in the repository README.

05Qualitative Results

Pyrender-shaded MANO mesh overlays on the egocentric stream, across nine episodes spanning the train / user / gesture / both splits.

episode_000001 · train / gesture

episode_000003 · train / gesture

episode_000005 · train / user

episode_000008 · zbk · user / both

episode_000013 · train / gesture

episode_000017 · user / both

episode_000020 · wmh · train / user / gesture / both

episode_000028 · wsj · train / gesture

episode_000033 · train / gesture

06Get the Data & Code

📦 Download the preview shard — Baidu Netdisk is the official release path. Open the share pan.baidu.com/s/1aG2e-mHJkmP4KiYtYRcReA (extraction code 8059), download the dataset_egoemg_preview folder, and place it at $EGOEMG_ROOT/data/. Full assets (memmap, videos, checkpoints) are documented in docs/ASSET_SETUP.md.

Setup

git clone https://github.com/zhenqis123/EgoEMG.git
cd EgoEMG
conda env create -f environment.yml && conda activate egoemg
pip install -e '.[viz]'
export EGOEMG_ROOT=/absolute/path/to/egoemg_assets

Evaluate a released checkpoint

python -m egoemg.test_analysis \
  experiment=emgformer/egoemg_emgformer_middle \
  'checkpoint=checkpoints/egoemg_emgformer_middle.ckpt'

07Citation

The paper is under review. Until a public version is available, cite this repository and the exact commit.

@misc{egoemg2026,
  title        = {EgoEMG: A multimodal egocentric dataset with bilateral surface
                  EMG and vision for hand pose estimation},
  author       = {Zhenqi Shi and others},
  howpublished = {GitHub repository},
  note         = {https://github.com/zhenqis123/EgoEMG},
  year         = {2026},
}