Technical Field Note #07 · October 2026
Hand Visibility in Egocentric Video: Detector vs Reality
A single detector pass is not a visibility measurement. Three passes, a temporal consensus and an independent visual review — with the human check still to come.
We ran MediaPipe Hands three ways over every frame of three egocentric recordings (7,557 frames: a head-mounted wide-angle stereo rig and two phone captures of cooking tasks) and compared the result with an independent AI visual review of 120 random frames. A single detector pass found both hands in 42%–74% of phone frames and 58% of rig frames, while the visual review saw both hands in 38/40, 39/40 and 40/40 sampled frames. Combining three passes with a ±2-frame window raised per-hand recall from 73%–81% to 92%–99%. All figures are AI-only preliminary results; a human-verification sample of 416 frames is designed but not yet performed.
Problem
A Detector Rate Is Not a Visibility Rate
Robot-learning buyers increasingly specify hand visibility: both hands in view for most of the task, hands not cut by the frame edge. The fastest way to report it is to run a hand detector and count frames with two detections. That number measures the detector as much as the footage. Hands holding tools, partly occluded by objects, motion-blurred or seen at an unusual angle are missed, so the detector rate understates what is actually in view.
This note quantifies that gap on our own captures, shows how much of it an automated consensus recovers, and sets out the human check that would turn a preliminary AI result into a verified one.
Status: AI-only preliminary results. No human verification has been performed yet; no figure in this note is a certified hand-visibility rate.
Origin Data Lab
Data
Three Recordings, Different Conditions
| Recording | Device and mount | Image | Task | Frames |
|---|---|---|---|---|
| Stereo rig (left camera) | Head-mounted stereo + IMU rig, wide-angle lens | 1600×1200, 30 fps | Bench work with a drill, screwdriver and a heater | 3,600 (120 s) |
| Phone · knife prep | Body-worn Android phone, rear camera | 1920×1080, 30 fps (rotated for display) | Peeling and cutting onions and a sweet potato | 1,979 (66 s) |
| Phone · seasoning | Same phone and operator setup | 1920×1080, 30 fps | Pouring two liquid seasonings and stirring | 1,978 (66 s) |
- Same operator for both phone recordings; the stereo recording is a different session and task.
- Source videos, raw sensor data and customer material are not published. The public package contains per-frame detection results and review labels only.
Definitions
What Counts as Visible
| State | Definition |
|---|---|
| Fully in view (F) | Wrist to fingertips inside the image; may hold an object or be partly hidden by it |
| Cut by border (C) | Part of the hand is outside the image boundary |
| Not visible (N) | No part of the hand is in the image |
- Both hands visible: neither hand is N.
- Both hands fully in view: both hands are F.
- Detector rates count MediaPipe detections; review rates count what the reviewer saw. The two are always reported separately.
- Wearer side comes from MediaPipe handedness, which assumes a mirrored image; labels are flipped for egocentric video. When both hands are detected, the wearer-left hand lies left of the wearer-right hand in 99–100% of frames.
Method
Three Passes, a Temporal Window and an Independent Review
- Pass 1 — single: MediaPipe Hands 0.10.14, full model, static-image mode, up to two hands, detection confidence 0.5, input scaled to 960 px width.
- Pass 2 — flipped: the same model on the horizontally mirrored frame, mapped back.
- Pass 3 — lite: the lite model on the original frame.
- Consensus: a hand counts as present if any pass found it within ±2 frames (±67 ms).
- Independent check: an AI visual review (multimodal model, blind to detections) of 40 random frames per recording, judging each hand as F, C or N.
- Runtime: 7,557 frames × 3 passes in about 12 minutes on a 2-core CPU, no GPU.
# per hand side k in {L, R}, per frame t
pass3[k][t] = single[k][t] or flipped[k][t] or lite[k][t]
consensus[k][t] = max(pass3[k][t-2 : t+2]) # ±2 frames = ±67 ms at 30 fps
consensus_both[t] = consensus[L][t] and consensus[R][t]\frac{\hat p+\frac{z^2}{2n}\pm z\sqrt{\frac{\hat p(1-\hat p)}{n}+\frac{z^2}{4n^2}}}{1+\frac{z^2}{n}},\quad z=1.96The three passes disagree on which hands are present in 61% of rig frames and 31%–44% of phone frames. That disagreement is itself a measurement: on egocentric footage a single detector call is unstable from frame to frame and from model to model.
The ±2-frame window was selected after inspecting the reviewed sample, so its recall is optimistic until confirmed on fresh frames; the human sample below is drawn independently for that reason.
Results
Single Pass vs Consensus vs Review
| Recording | Frames | Single pass | 3-pass union | Consensus (±2 frames) | AI visual review (95% CI) |
|---|---|---|---|---|---|
| Stereo rig (head-mounted, wide-angle) | 3,600 | 58.1% | 82.7% | 95.3% | 40/40 (91.2–100.0%) |
| Phone · knife prep | 1,979 | 41.8% | 66.6% | 84.3% | 38/40 (83.5–98.6%) |
| Phone · seasoning | 1,978 | 74.1% | 84.9% | 89.9% | 39/40 (87.1–99.6%) |
On the stereo rig every reviewed frame showed both hands, yet a single pass found both in only 58% of frames; the consensus reached 95%. On the knife-prep phone recording the single pass reported 42% against 38/40 reviewed frames with both hands visible.
Detector Error
How Many Visible Hands the Detector Misses
| Recording | Visible hands | Single pass | Majority of 3 | Any of 3 | Consensus (±2 frames) |
|---|---|---|---|---|---|
| Stereo rig | 80 | 80.0% (70.0–87.3%) | 81.2% | 92.5% | 98.8% (93.3–99.8%) |
| Phone · knife prep | 78 | 73.1% (62.3–81.7%) | 79.5% | 89.7% | 94.9% (87.5–98.0%) |
| Phone · seasoning | 79 | 81.0% (71.0–88.1%) | 82.3% | 87.3% | 92.4% (84.4–96.5%) |
- Typical misses: hands gripping a tool or bottle, hands partly behind an object, and motion-blurred hands.
- False positives could not be estimated: only 3 reviewed hands were absent (consensus marked none of them present). Precision needs the targeted sample below.
Frame Edges
Border Cuts and Where Hands Sit
- AI visual review: no reviewed rig frame had a hand cut by the border (0/40); phone frames had 5/40 (knife prep) and 4/40 (seasoning).
- Both hands fully in view: 40/40 (rig), 34/40 and 35/40 (phone).
- Detected hand centres within 10% of the frame edge: 0.0% (rig), 4.6% and 0.1% (phone).
- The detector rarely reports a cut hand at all: when the palm leaves the frame it usually stops detecting, so border cuts must be measured by review, not inferred from landmarks.
Motion
Detection Falls With Motion Blur (Field Note #06)
- Rig: consensus both-hands rate 97% below 0.5 px of predicted blur, 86% at 2–6 px.
- Phone, seasoning: 98% below 10 °/s, 43% above 80 °/s; single pass 88% → 7%.
- Part of the drop at high rotation is real (hands leave the view during head turns) and part is detector failure on blurred hands; separating the two is one purpose of the human sample.
Task Structure
Visibility by Keystep (Field Note #05)
- In the seasoning task, pouring, stirring and closing keysteps reach 95%–100% under consensus.
- The lowest values are non-task spans and placing actions: ks_A_10 non-task 28%, ks_A_08 place 33%, ks_B_11 place 20%.
- Whether a hand truly left the view while placing an object, or was missed, is exactly what the targeted human sample is designed to resolve.
Verification Plan
The Human Check, Sized in Advance
| Stratum | Stereo rig | Phone · knife prep | Phone · seasoning | Purpose |
|---|---|---|---|---|
| Random frames | 100 | 100 | 100 | Unbiased visibility rate and detector recall; Wilson half-width ≈ 4–6 points at 90–95% |
| Consensus says ‘not both’ | 39 | 37 | 40 | Separate real out-of-view moments from detector misses; estimate precision of absence calls |
- 416 frames in total; we estimate two to three reviewer-hours.
- Planned outputs: human-verified both-hands and fully-in-view rates with intervals, detector recall and precision, and a re-check of the ±2-frame window on fresh frames.
- Until that review is complete, Origin Data Lab reports these results as AI-only preliminary and does not claim any certified visibility threshold.
For Data Buyers
What to Ask a Capture Partner
- Is the reported hand-visibility rate a detector rate, a consensus rate or a reviewed rate?
- What is the detector’s measured recall on your footage, with an interval?
- How are hands cut by the frame edge counted?
- Which frames were reviewed by people, how were they sampled, and how many?
- Are per-frame results delivered, so you can apply your own threshold?
Reproducibility
Re-running the Analysis
pip install -r requirements.txt # mediapipe==0.10.14 (Python 3.11), numpy, pandas, statsmodels
python code/analyze_hand_visibility.py data/hand_visibility_frames.csv data/ai_review_labels.csv
# re-running detection needs the source videos (not public):
python code/detect_hands.py <recording> <video.mp4> <width> <height> <start> <stop>The public package contains per-frame results for all frames, the review labels, the analysis code and a notebook that reproduces every number in this note from the CSV files.
Limitations
- All results are AI-only: MediaPipe detections and an AI visual review. No human verification has been performed.
- The visual review covers 40 frames per recording; intervals are wide and false-positive rates could not be estimated.
- The ±2-frame consensus window was chosen after inspecting the reviewed sample.
- Three recordings, one operator per session, two task families; device, mounting and task differ between the rig and the phone.
- Occlusion by objects is folded into ‘fully in view’; hand pose quality and landmark accuracy are not evaluated.
Frequently asked questions
What share of egocentric frames show both hands?
In our AI visual review of 40 random frames per recording, both hands were visible in 40/40 frames on a head-mounted wide-angle rig and 38/40 and 39/40 frames on two phone recordings. These are AI-only preliminary figures from small samples, not human-verified rates.
Is MediaPipe hand detection rate a reliable measure of hand visibility?
Not on its own. A single MediaPipe pass found both hands in 58% of stereo-rig frames although the review saw both hands in every sampled frame. Per-hand recall of a single pass was 73%–81%.
How can automated hand-visibility estimates be improved?
Combining a full-model pass, a mirrored pass and a lite-model pass, and accepting a hand found within ±2 frames, raised per-hand recall to 92%–99% on the reviewed frames. The window was chosen on the same sample, so it still needs confirmation on fresh frames.
Do head-mounted wide-angle cameras keep hands in view better than phones?
In these recordings hands stayed in the centre of the wide-angle frame and no reviewed frame had a border cut, while phone frames had cuts in 5/40 and 4/40 reviewed frames. Device, task and session differed, so this is an observation, not a controlled comparison.
Does motion blur affect hand detection?
Yes. On the phone seasoning recording the consensus both-hands rate fell from 98% below 10 °/s of camera rotation to 43% above 80 °/s, partly because hands leave the view and partly because blurred hands are missed.
How many frames need human review to verify hand visibility?
We sized 100 random frames per recording, giving a Wilson 95% half-width of about 4–6 percentage points at 90–95% visibility, plus 116 frames where the consensus reports fewer than two hands. That review has not yet been carried out.
RESEARCH RESOURCES
Explore the Technical Resources
Access the code, datasets and evaluation materials.
Robotics Training Data
Explore custom real-world data production for robotics training.