Technical Field Note #23 · November 2026

Hands are in view 75% of the time; an off-the-shelf detector finds a quarter of them: MediaPipe on head-mounted fisheye work video

By Hoonhoe Kim, Co-Founder & CTO, Origin Data Lab

Field Note #23 · Hand detection recall on real head-mounted fisheye work video (ODL-EGO S1), and the simple change that doubled it

Hand presence is the first filter most teams apply to egocentric video for robot learning, usually with an off-the-shelf detector. We checked how well that works on our own data: 2.57 hours (9,250 seconds, 4 sessions) of real produce-portioning work filmed with a head-mounted fisheye stereo rig, one frame per second from the left eye. In a visual review of 60 random seconds, the wearer's hands were visible in 45 (75%), 73 hands in total. MediaPipe Hands on the full frame found 27% of those hands; at 800×600 or full 1600×1200 resolution it still found 26–27%, after undistorting the fisheye image to a pinhole view 22%, and rotated 180° 25%. What helped was where it looked: running it on the lower work region, where hands actually are, found 47%, and with the detection threshold lowered to 0.3, 58%, with two extra detections in 60 seconds. Over the whole 2.57 hours, the share of seconds with a detected hand rose from 34% to 57%, still below the 75% that were visible. A hand filter built on a default detector would have thrown away more than half of the usable footage.

Published November 2026⏱ 11 min readStatus Preliminary · automated analysisOpen data GitHub · Hugging Face · Kaggle
75%Seconds with the wearer's hand visible (review)45 of 60 random seconds, 73 handsAI visual review, not human-verified
27%Visible hands found, full frame (default)26–27% at 480, 800 and 1600 px wideVerified (automated)
58%Visible hands found, lower work region + threshold 0.32 extra detections in 60 sVerified (automated)
34% → 57%Seconds with a detected hand, all 2.57 hfull frame vs combined passes; undistorted: 27%Verified (automated)

Why this matters

A hand filter that misses hands throws away good data

Robot-learning buyers often keep only egocentric footage where hands are in view, and the usual way to decide that at scale is an off-the-shelf hand detector. If the detector misses hands, good footage is silently discarded and hand-presence statistics in a dataset card are understated.

Field Note #7 measured MediaPipe on short cooking tests. This note measures it on what we actually deliver: hours of real work, filmed from the head with a fisheye lens, the wearer in a long-sleeved work jacket.

Setup

2.57 hours of real work, one frame per second

Recording: 2026-11-08, produce portioning for school meals, four sessions of 24, 33, 53 and 44 minutes, R8 stereo+IMU rig (ODL-EGO S1) on the head, 1600×1200 at 30 fps. Analysis used the left eye's keyframes, one per second: 9,250 seconds.

Detector: MediaPipe Hands 0.10.14 (single-image mode, up to 2 hands). Reference: an AI visual review of 60 random seconds that counted the wearer's visible hands (whole or partly visible, including gloved; other people's hands not counted). The review is not human-verified.

Finding 1

Hands were visible in 75% of seconds; the default detector found 27% of them

In the 60 reviewed seconds the wearer's hands were visible in 45, 73 hands in total. The full-frame detector found at least one hand in 36% of those 45 seconds and 27% of the 73 hands, with no false detections.

Typical misses: hands seen from above with the back of the hand towards the camera, wrists covered by the jacket sleeves, hands small near the bottom of the wide image, and hands holding bags or paper.

Bar chart of the share of visible hands found by eight detector variants, from 22% to 58%
Figure 1. Share of the 73 visible hands found in the same 60 reviewed seconds by each variant.

Finding 2

More pixels and undistortion do not help; looking in the right place does

Running the detector on 800×600 or the full 1600×1200 image found 26–27%, the same as 480×360. Undistorting the fisheye frame to a pinhole view made it worse (22%); 92% of the hands the raw pass found and the undistorted pass missed lay fully inside the undistorted view, so the loss is not field of view. Rotating the image did not help either (25%).

Cropping to the lower work region (the central lower 60% of the frame, scaled to 800 px wide) found 47%; lowering the detection threshold from 0.5 to 0.3 on that crop found 58% and caught at least one hand in 60% of hand-seconds, with two extra detections in 60 seconds. On the stored 480×360 JPEG keyframes used for the whole-day run, the full-frame pass found 25% and the lower-region rule 49%.

Table 1. More pixels and undistortion do not help; looking in the right place does
Detector pass (same 60 seconds)Visible hands foundSeconds with a visible hand where ≥1 was foundDetections with no wearer hand
Full frame, 480×36027%36%0
Full frame, 800×60027%36%1
Full frame, 1600×120026%33%1
Undistorted to pinhole, 480×36022%29%0
Rotated 180°25%29%0
Full frame, confidence 0.334%40%1
Lower work region (800 px wide)47%51%2
Lower work region + confidence 0.358%60%2
Three egocentric work frames with hand skeletons drawn inside a box covering the lower part of the image
Figure 2. Seconds where the full-frame pass found nothing and the lower-region pass found the hands.

Finding 3

Over the whole day: 34% of seconds with a detected hand, 57% after the change

Applied to all 9,250 seconds, the full-frame pass found a hand in 34.2% of seconds, the undistorted pass in 26.8%, the lower-region pass in 54.5% and both passes combined in 56.7%. Per session the combined rate ranged from 46% to 67%. All of these remain below the 75% visible in the review, so detector output should be read as a lower bound on hand presence.

Grouped bars of the share of seconds with a detected hand per session for four detector passes, below a dashed line at 75 percent
Figure 3. Share of seconds with at least one detected hand, per session and overall. Dashed line: visible in the reviewed sample.

How to check your own data

Measure recall before you filter

A detector pass is cheap; a reviewed sample is what tells you what it misses.

  • Review 50–100 random seconds by eye and count visible hands, including partial and gloved ones.
  • Run your detector on the same seconds; compare recall before using it as a filter.
  • For head-mounted views, run on the lower work region and lower the threshold; check false detections on the same sample.
  • Report hand presence as detector rate plus reviewed recall, not detector rate alone.

What we changed

Hand presence is now reported with its recall

Verified: on this rig and task, region-focused detection roughly doubles hand recall compared with default full-frame MediaPipe, and undistortion does not help.

Process: our per-second QC index (Field Note #25) uses the combined passes, and delivery reports state hand presence together with the reviewed recall, so buyers can see how much footage a hand filter would remove.

  • Detector: lower work region + full frame, threshold 0.3.
  • Every delivery: 60-second reviewed sample with recall.
  • Never discard footage on detector output alone.

SCOPE AND LIMITS

Limitations

What this note does not show, stated up front.

  • One wearer, one task, one rig, one day; recall will differ for other tasks, clothing and lighting.
  • The reference is an AI visual review of 60 seconds (73 hands), not a human-verified annotation; its sampling uncertainty is roughly ±10 percentage points.
  • One frame per second (keyframes) was analysed, not every frame; temporal tracking across frames was not tested here.
  • Only MediaPipe Hands was tested; other detectors and hand-pose models may behave differently.
  • Preliminary analysis: AI-assisted automated analysis; no third-party review.

FAQ

Questions data buyers ask

Q01How well does MediaPipe Hands detect hands in head-mounted egocentric video?
On 2.57 h of real fisheye work video it found 27% of visible hands with default settings. Cropping to the lower work region and lowering the threshold to 0.3 raised that to 58%.
Q02Does undistorting fisheye images improve hand detection?
Not here: undistorting to a pinhole view lowered recall from 27% to 22%, and almost all of the lost hands were still inside the undistorted view.
Q03Does higher resolution help?
No. 480×360, 800×600 and 1600×1200 gave 26–27%. The detector struggles with small, partly covered hands in a wide view, not with pixel count.
Q04How often are the wearer's hands visible during real work?
In 45 of 60 random seconds (75%) of produce-portioning work, by visual review.
Q05Should I filter egocentric footage by hand detection?
Only after measuring the detector's recall on a reviewed sample of your own footage. A default detector here would have discarded more than half of the seconds that had visible hands.

OPEN DATA AND CODE

Reproduce and inspect the evidence

Every number in this note can be checked: derived tables, analysis code, figures and a runnable notebook are public.

GitHubAnalysis code and reproduction stepsorigin-data-lab/egocentric-fisheye-hand-detection-recall-real-workView repository Hugging FaceDataset card, tables and figuresorigin-data-lab/egocentric-fisheye-hand-detection-recall-real-workOpen dataset KaggleData explorer and runnable notebookorigindatalab/odl-egocentric-hand-detection-recallExplore on Kaggle
Run the notebook on Kaggle

Licence: CC BY 4.0 (data and figures) · MIT (code)

Cite this note

Origin Data Lab (2026). Technical Field Note #23: Hands are in view 75% of the time; an off-the-shelf detector finds a quarter of them. https://origindatalab.io/technical-field-notes/egocentric-fisheye-hand-detection-recall-real-work.html

CONTINUE READING

Related field notes

FIELD NOTE #17Where a head-mounted camera points during 2 h 25 min of real workFIELD NOTE #15Head motion and motion-blur budget in 2 h 25 min of real egocentric workFIELD NOTE #07Are both hands in view? Measuring hand visibility in egocentric capture

Want this measured on your data?

Origin Data Lab plans and runs egocentric, stereo+IMU and field video capture with documented QA, for physical AI and robotics teams.

Request sample dataContact usData services