Technical Field Note #24 · November 2026

3D hand skeletons from a head-mounted stereo pair during real work: hands 28–42 cm away, palm segments stable to 4–10% per frame

By Hoonhoe Kim, Co-Founder & CTO, Origin Data Lab

Field Note #24 · Metric 3D hand pose from a head-mounted stereo+IMU rig (ODL-EGO S1) on real work, checked by the consistency of rigid hand segments

A single camera can find hand landmarks but not their true size or distance; robot learning needs both. We ran MediaPipe Hands separately on the left and right eye of our head-mounted fisheye stereo rig, matched the hands, and triangulated all 21 landmarks with the factory calibration, on three 30-second clips of real produce-portioning work (2,252 hand-frames at 15 fps). Each hand was found in both eyes in 67–96% of frames. Wrists were 28–42 cm from the camera (median per clip), where one pixel of disparity is about 0.6 cm of depth. Because no ground truth was available, we checked the result with physics: bones do not change length. Rigid palm segments came out the same from frame to frame within a median 4–10% and the same across all three clips within 1 cm (wrist to index knuckle 7.7–8.7 cm). Finger segments were noisier (8–20%), and palm width, which lies almost along the viewing direction, was unreliable (20–35%). The model hand that MediaPipe itself outputs has different proportions (e.g. 2.1–2.9 cm for finger segments that stereo measures at 4.0–4.2 cm), so stereo adds the wearer's own hand size and distance.

Published November 2026⏱ 12 min readStatus Preliminary · automated analysisOpen data GitHub · Hugging Face · Kaggle
67–96%Frames with the hand found in both eyesper hand and clip; 2,252 hand-frames in 3 × 30 sVerified (automated)
28–42 cmWrist distance from the cameramedian per clip; 5–95% range 18–49 cmVerified (automated)
4–10% (MAD)Rigid palm segments, frame-to-frame spreadwrist–index knuckle 7.7–8.7 cm across clipsVerified (automated)
2.6 px medianLandmark disagreement between eyesvertical mismatch after rectification; calibration itself < 1 pxVerified (automated)

Why this matters

2D landmarks are not enough for robots

A robot that learns from human demonstrations needs to know where the hand is in space and how large it is, not just where it appears in the image. Monocular hand-pose models return a 3D shape at a canonical size, but not the wearer's real hand size or its distance from the camera.

A calibrated stereo pair can provide both. This note shows what that gives on real work video, and how to check it without a motion-capture system.

Setup

Three 30-second clips of real work, both eyes

Recording: 2026-11-08, produce portioning for school meals, R8 stereo+IMU rig (ODL-EGO S1) on the head, 1600×1200 per eye at 30 fps, 6.5 cm baseline. Clips: 11:48 session (bagging), 10:49 session (sorting papers and produce) and 10:00 session (packing), 30 s each, the windows with the most two-hand activity; every second frame analysed (450 frame pairs per clip).

Method: MediaPipe Hands 0.10.14 in video mode on the lower work region of each eye (confidence 0.3, the setting from Field Note #23). Left and right hands matched by handedness, the 21 landmark pixels undistorted with the factory fisheye model, and triangulated with the factory stereo extrinsics. The calibration file is for 800×600 and was scaled ×2 (Field Note #20).

Left and right camera views of two hands working above a bag, with hand skeletons and wrist distances of 45 and 42 cm
Figure 1. The same instant in both eyes. Each eye finds the landmarks independently; their difference in position gives distance.

Finding 1

Both hands, in both eyes, most of the time, at 18–49 cm

The right hand was found in both eyes in 94–96% of frames and the left in 67–75% (it was more often at the edge of the view or holding bags). 98.6% of matched hand-frames triangulated to a plausible distance (10–150 cm). Median wrist distance per clip was 28–42 cm, 5–95% range 18–49 cm.

At 42 cm the hands span about 70 px of disparity, so one pixel of landmark error is about 0.6 cm in depth. After rectification, the same landmark sat a median 2.6 px apart vertically between the eyes. The calibration's own epipolar error is below 1 px (Field Note #16), so this is the detector's landmark noise, and it sets the precision.

Three-dimensional plot of two hand skeletons over one second, in the camera frame
Figure 2. One second of both hands in 3D, camera frame; older frames fainter.
Left: wrist distance over 30 seconds for two hands. Right: histogram of left-right vertical landmark mismatch
Figure 3. Left: wrist distance over clip A. Right: left/right vertical mismatch of landmarks per hand-frame.

Finding 2

Bones keep their length: the check that needs no ground truth

Lengths of rigid segments must not change while a hand moves. Wrist to index knuckle measured a median 8.2 cm (six per-hand medians 7.7–8.7 cm), wrist to little-finger knuckle 7.5 cm (7.1–7.7), thumb CMC–MCP 3.4 cm (3.3–3.7), with a frame-to-frame median absolute deviation of 4–10%. Proximal finger segments (about 4 cm) were noisier at 8–20%.

Palm width, between the index and little-finger knuckles, varied 20–35%. For a hand seen from above it lies mostly along the viewing direction, where stereo is weakest and the two knuckles are hardest to locate. Treat it as unreliable from this viewpoint.

Table 1. Bones keep their length: the check that needs no ground truth
SegmentStereo length, median of all framesRange of six per-hand mediansPer-frame spread (MAD)Frames within ±10%MediaPipe model hand
Wrist – index knuckle (MCP)8.2 cm7.7–8.7 cm4.6–8.9%54–80%9.7 cm
Wrist – little-finger knuckle7.5 cm7.1–7.7 cm4.3–9.8%51–79%7.4 cm
Thumb CMC – MCP3.4 cm3.3–3.7 cm5.2–7.3%63–77%3.3 cm
Index MCP – PIP4.1 cm3.9–4.2 cm7.6–13.2%40–62%2.4 cm
Middle MCP – PIP4.2 cm3.8–4.3 cm7.5–17.8%29–61%2.9 cm
Ring MCP – PIP4.0 cm3.6–4.1 cm8.0–20.3%25–61%2.1 cm
Index knuckle – little-finger knuckle (palm width)5.0 cm4.1–5.7 cm19.7–35.2%16–31%5.0 cm
Box plots of seven hand segment lengths measured by stereo, with MediaPipe model values as diamonds
Figure 4. Per-frame stereo length of seven rigid segments, all clips. Diamonds: the model hand MediaPipe outputs for the same frames.

Finding 3

Stereo measures this hand; the model returns a template

MediaPipe also outputs its own metric-scale 3D hand. For the same frames its wrist-to-index segment was 9.7 cm and its proximal finger segments 2.1–2.9 cm, against 8.2 cm and 4.0–4.2 cm from stereo: different proportions, not just a scale factor. The model is a template; stereo measures the wearer.

How to check your own data

Bone length is a free QA signal

Any stereo or multi-view hand pipeline can run this check per frame.

  • Triangulate landmarks from both eyes and compute 5–7 rigid segment lengths per frame.
  • Flag frames where a segment deviates more than 20% from that hand's median; they are usually mismatched or mislocated landmarks.
  • Do not trust segments that lie along the viewing direction (here: palm width).
  • Compare across sessions: the same wearer should give the same segment lengths every day.

What this means

Metric 3D hands from the same data, with a built-in check

Verified: on this rig, real work video yields metric 3D hand skeletons for both hands most of the time, at 18–49 cm, with rigid palm segments consistent to within 4–10% per frame and 1 cm across clips.

Not yet verified: absolute accuracy against a measured hand or motion capture. Stereo depth scale on this rig was checked separately against targets (Field Note #9).

Process: deliveries can include per-frame 3D landmarks with a bone-length consistency flag, so buyers can filter frames before training.

  • Per-frame 3D landmarks, both hands, camera frame.
  • Bone-length flag per frame (>20% deviation).
  • Next: compare with a tape-measured hand of the wearer.

SCOPE AND LIMITS

Limitations

What this note does not show, stated up front.

  • Three 30-second clips from one wearer, one task and one day; one rig.
  • No ground truth: absolute segment lengths were not compared with a measured hand or motion capture.
  • Landmarks come from MediaPipe Hands; its errors (especially for hands seen from the back or partly covered) propagate into 3D.
  • Every second frame was processed; frames where a hand was found in only one eye were not triangulated.
  • Preliminary analysis: AI-assisted automated analysis; no third-party review.

FAQ

Questions data buyers ask

Q01Can a head-mounted stereo camera measure 3D hand pose?
Yes. On real work video, triangulating MediaPipe landmarks from both eyes gave both hands in 3D at 18–49 cm in most frames, with rigid palm segments consistent to 4–10% per frame.
Q02How accurate is the depth of hand landmarks?
At 42 cm one pixel of disparity is about 0.6 cm. Landmarks disagreed by a median 2.6 px between the eyes, so per-landmark depth noise is a few millimetres to about a centimetre.
Q03How can 3D hand pose be validated without motion capture?
Use bone lengths: rigid segments must keep the same length in every frame and every session. Their spread is a direct measure of noise and a per-frame QA flag.
Q04Is MediaPipe's 3D world output metric?
It is metric-scaled but comes from a template hand. Its proportions differed from the stereo measurement (e.g. finger segments 2.1–2.9 cm vs 4.0–4.2 cm).
Q05Which hand measurements are unreliable from a head-mounted view?
Those along the viewing direction, such as palm width between the index and little-finger knuckles (20–35% spread).

OPEN DATA AND CODE

Reproduce and inspect the evidence

Every number in this note can be checked: derived tables, analysis code, figures and a runnable notebook are public.

GitHubAnalysis code and reproduction stepsorigin-data-lab/stereo-3d-hand-skeleton-real-workView repository Hugging FaceDataset card, tables and figuresorigin-data-lab/stereo-3d-hand-skeleton-real-workOpen dataset KaggleData explorer and runnable notebookorigindatalab/odl-stereo-3d-hand-skeletonExplore on Kaggle
Run the notebook on Kaggle

Licence: CC BY 4.0 (data and figures) · MIT (code)

Cite this note

Origin Data Lab (2026). Technical Field Note #24: 3D hand skeletons from a head-mounted stereo pair during real work. https://origindatalab.io/technical-field-notes/stereo-3d-hand-skeleton-real-work.html

CONTINUE READING

Related field notes

FIELD NOTE #23Hands are in view 75% of the time; an off-the-shelf detector finds a quarter of themFIELD NOTE #21Head pose from stereo alone during real work, checked with the gyroscope and accelerometerFIELD NOTE #20Stable is not the same as correct: a calibration file for the wrong resolution

Want this measured on your data?

Origin Data Lab plans and runs egocentric, stereo+IMU and field video capture with documented QA, for physical AI and robotics teams.

Request sample dataContact usData services