Start and end are set by approach, contact, tool engagement, release or placement that can be seen in the video.
Technical Field Note #05 · October 2026
Keystep Annotation for Egocentric Robot-Learning Data
A schema, the automated QA rules behind it, and what a body-worn IMU can — and cannot — confirm about human-labelled action boundaries
Two real cooking tasks (knife prep; pouring seasoning and stirring), 66 s each, captured with a body-worn Android phone: 1920×1080 at 30 fps and a 410.6 Hz native IMU. A person annotated 25 keysteps from the video; we then checked every label against structural rules and against the wearer’s measured rotation. Single operator, method demonstration — not a benchmark.
25 / 0Keysteps / structural errors (gaps, overlaps, missing fields, out-of-spec verbs)
7.5×Median wearer rotation in non-task spans vs task spans (61 vs 8 °/s)
p = 0.005 / 0.002Labelled boundaries vs 2,000 random boundary sets, samples A / B
p = 0.67Task→task boundaries in knife prep: the IMU cannot see them
Problem
Why Keystep Labels Fail in Practice
Language-conditioned policies, vision-language-action (VLA) models and task-segmentation models all consume the same thing from human demonstration video: time spans bound to an action, an object and a description. Most annotation problems we see are not exotic. Spans leave gaps or overlap. Hands are labelled from the image instead of the wearer. Annotators guess intent or object identity they cannot see. Descriptions drift into robot commands the video never showed. And nobody states how precise the boundaries are, so a buyer cannot tell a 0.5 s semantic boundary from a frame-accurate one.
- Sample A
- Knife prep — two onions and a sweet potato · 66.0 s · 1,979 frames · 13 keysteps in 3 phases
- Sample B
- Pouring two liquid seasonings and stirring · 65.97 s · 1,978 frames · 12 keysteps in 3 phases
- Operator
- One person, left-handed (confirmed by the operator)
- Annotation
- Manual from video, second-pass visual review; semantic boundaries at about 0.5–1.0 s
- IMU
- Phone accelerometer + gyroscope, 410.6 Hz, 0 dropouts; measures camera / wearer motion, not hand joints
Schema
Task → Phase → Keystep
Occluded, blurred or out-of-frame details get a generic name, unknown or null — never an inferred identity or intent.
Keysteps tile the whole video. Anything unrelated to the task is labelled non_task rather than left out.
Boundary precision is written into the file, so downstream users know what the timestamps can support.
One keystep (sample A, unchanged from the delivered file)
{ "id": "ks_A_01", "t_start": 0.0, "t_end": 7.5,
"phase": "onion_A_preparation", "verb": "peel", "object": "onion_A",
"tool": "knife", "active_hand": "left", "support_hand": "right",
"instruction": "Hold and rotate onion_A with the right hand while removing
outer material with a knife in the left hand.",
"state_before": "onion_A with outer material",
"state_after": "onion_A partially peeled", "outcome": "success" }
HANDSactive_hand / support_hand use the wearer’s anatomical left and right, never the image side. Handedness is recorded once per file.
TEXTinstruction describes what was observed. It is not a robot command and implies nothing about grasp pose or force.
OUTCOMEsuccess only when completion is visible; otherwise unknown. non_task spans carry outcome null.
NOTESreview_note records why a reviewer merged or split spans (4 notes in sample B, e.g. a concurrent stir during a pour).
Structural QA
Automated Checks on Every Keystep
| Rule | Sample A | Sample B |
|---|---|---|
| Keysteps / phases | 13 / 3 | 12 / 3 |
| First start / last end = video span | 0.0–66.0 s | 0.0–65.967 s |
| Gaps / overlaps between keysteps | 0 / 0 | 0 / 0 |
| Every keystep inside its phase | Yes | Yes |
| Missing values among 13 required fields | 0 | 0 |
| Verbs outside the vocabulary | 0 | 0 |
| Keystep length, min–max (mean) | 1.2–16.0 s (5.1) | 1.5–19.0 s (5.5) |
| Verbs used | peel 3 · place 3 · trim 2 · non_task 2 · cut · grasp · discard | pour 3 · open 2 · close 2 · place 2 · non_task 2 · stir |
| Acting hand (task keysteps) | left 6 · right 5 | left 3 · right 7 |
What structural QA does not tell you
These rules prove the file is well formed. They say nothing about whether a boundary is in the right place. For that you need a second annotator, or an independent signal. The phone IMU is the independent signal every egocentric capture already has.
Method
Checking Labels Against Wearer Motion
The first frame came 188 ms (A) and 200 ms (B) after the app reference point, measured by fitting video optical flow to the gyroscope. IMU time was shifted by that offset.
Gyroscope magnitude, independent of how the phone was oriented on the body.
For each boundary: |median |ω| in the second after − median in the second before|, averaged over boundaries.
The same score for the same number of boundaries placed uniformly at random; p = share of random sets scoring at least as high.
Results
The IMU Confirms Coarse Structure, Not Fine Hand Actions
| Boundary set | n | Labelled (°/s) | Random mean (°/s) | Random p95 (°/s) | p |
|---|---|---|---|---|---|
| A · all boundaries | 12 | 21.9 | 7.2 | 14.7 | 0.005 |
| B · all boundaries | 11 | 29.9 | 10.0 | 20.1 | 0.002 |
| A · task→task only | 8 | 3.3 | 4.1 | 6.8 | 0.67 |
| B · task→task only | 7 | 10.2 | 3.9 | 7.5 | 0.003 |
Reading the Numbers
Knife work happens in the hands while the head and torso stay still, so peel → place → trim leaves no trace in a body-worn IMU. Seasoning is different: reaching for a bottle, opening it and setting it down move the upper body, and those boundaries show up. With seven or eight boundaries per set, these are small-sample tests on one operator; treat them as a demonstration of the method, not as general rates.
In Production
Use the IMU to Queue Reviews, Not to Grade Labels
The useful output is a review queue. Task keysteps with unusually high rotation, and boundaries where a large rotation event sits just outside the labelled span, go to a reviewer. In sample A the queue contains one item: trim (ks_A_05) has a 95th percentile of 142 °/s. The cause is the first 0.32 s of the span (27.01–27.32 s), the tail of the preceding head turn — inside the stated 0.5–1.0 s boundary precision, so the label stands.
USEFlag task spans with large head/body rotation, and non-task spans with almost none, for a second look.
USEConfirm boundaries that involve body motion — reaching, turning, setting objects aside.
DON’TUse body IMU agreement to certify fine manipulation boundaries. That needs hand tracking or a second annotator.
DON’TUse label–motion agreement to measure video–IMU offset. Sliding the labels against the IMU peaked at +0.08 s (A) and −0.04 s (B) from the measured alignment — well inside label precision, so labels cannot resolve milliseconds.
Technical Boundaries
What This Note Does Not Claim
Inter-Annotator Agreement
One annotator plus one reviewer. Dual annotation with boundary-difference and label-agreement reporting is a production step, not shown here.
Generality
Two tasks, one left-handed operator, 132 s in total. The statistics illustrate the method; they are not rates for other tasks or people.
Hand Pose and Robot Actions
No hand keypoints, grasp poses, force/torque or robot action labels. Keysteps do not imply human-to-robot retargeting success.
Data
Rotation Inside Every Keystep
| Keystep | Verb | Acting hand | Length | Median |ω| (°/s) | p95 |ω| (°/s) |
|---|---|---|---|---|---|
| ks_A_01 | peel | left | 7.5 s | 5 | 15 |
| ks_A_02 | place | left | 2.0 s | 10 | 21 |
| ks_A_03 | peel | right | 16.0 s | 5 | 14 |
| ks_A_04 | non-task | — | 1.5 s | 53 | 119 |
| ks_A_05 | trim | left | 4.0 s | 22 | 142 |
| ks_A_06 | place | right | 4.5 s | 11 | 40 |
| ks_A_07 | peel | left | 10.5 s | 8 | 26 |
| ks_A_08 | place | right | 2.0 s | 8 | 19 |
| ks_A_09 | trim | left | 5.3 s | 7 | 18 |
| ks_A_10 | non-task | — | 1.2 s | 75 | 188 |
| ks_A_11 | discard | right | 2.8 s | 14 | 31 |
| ks_A_12 | grasp | right | 1.5 s | 15 | 30 |
| ks_A_13 | cut | left | 7.2 s | 5 | 35 |
| ks_B_01 | open | right | 2.0 s | 17 | 59 |
| ks_B_02 | pour | left | 3.5 s | 21 | 46 |
| ks_B_03 | non-task | — | 4.5 s | 57 | 149 |
| ks_B_04 | pour | left | 14.3 s | 4 | 27 |
| ks_B_05 | non-task | — | 2.4 s | 66 | 183 |
| ks_B_06 | close | right | 2.8 s | 12 | 38 |
| ks_B_07 | place | right | 1.8 s | 22 | 43 |
| ks_B_08 | open | right | 1.5 s | 44 | 56 |
| ks_B_09 | pour | left | 9.7 s | 3 | 16 |
| ks_B_10 | close | right | 2.0 s | 6 | 28 |
| ks_B_11 | place | right | 2.5 s | 14 | 47 |
| ks_B_12 | stir | right | 19.0 s | 6 | 13 |
FAQ
Common Questions
What is a keystep annotation?
A keystep is a time span in a video labelled with one observable action unit: a verb, the object acted on, the tool, which hand acts and which supports, a short natural-language description, and the visible state before and after. Keysteps sit inside phases (sub-goals), which sit inside a task goal.
How precise should keystep boundaries be for robot learning?
It depends on the use. Semantic boundaries at about 0.5–1.0 s are enough for task segmentation, language-conditioned policy data and world-model training. Frame-accurate segmentation needs a separate, more expensive protocol and should be specified and priced as such.
Can an IMU validate human action labels?
Partly. In our measurement a body-worn IMU clearly separated non-task spans (head or body turns) from task work and confirmed boundaries that involve body motion. It could not confirm boundaries between fine hand actions such as peel → place → trim, because the hands move while the head and torso stay still.
How should left and right hands be labelled in egocentric video?
Use the wearer’s anatomical left and right, never the image left and right, and record the operator’s handedness as confirmed by the operator. Keep separate fields for the acting hand and the supporting hand.
Do video and IMU timestamps need to be aligned before checking annotations against motion?
Yes. Annotations are drawn on video time. In this capture the first video frame came 188–200 ms after the app reference point, so IMU time was shifted by the measured offset before any comparison.
Machine-Readable Summary
Key Technical Facts
Evidence Summary
Origin Data Lab annotated two 66-second egocentric cooking demonstrations (1920×1080, 30 fps, 410.6 Hz phone IMU, one left-handed operator) with 25 keysteps in a task → phase → keystep schema: verb, object, tool, acting and supporting hand (wearer’s anatomical side), a natural-language description, visible state before and after, and outcome. Automated QA found 0 gaps, 0 overlaps, 0 missing required fields and 0 out-of-vocabulary verbs; boundary precision is stated as 0.5–1.0 s.
After aligning IMU to video time (+188 ms and +200 ms, measured from video–gyro fitting), median wearer rotation was 61 °/s in non-task spans versus 8 °/s in task spans. Labelled boundaries carried 21.9 and 29.9 °/s of motion change versus 7.2 and 10.0 °/s for random positions (p = 0.005 and 0.002). Excluding non-task boundaries, knife-prep boundaries were indistinguishable from random (p = 0.67) while seasoning boundaries remained distinct (p = 0.003): a body-worn IMU confirms body-level structure but not fine hand-action boundaries.
How to cite this note
Origin Data Lab (2026). Keystep Annotation for Egocentric Robot-Learning Data: Schema, QA and an IMU Cross-Check. Technical Field Note #05. https://origindatalab.io/technical-field-notes/keystep-annotation-egocentric-robot-learning.html
@misc{origindatalab2026fieldnote05,
author = {{Origin Data Lab}},
title = {Keystep Annotation for Egocentric Robot-Learning Data: Schema, QA and an IMU Cross-Check},
howpublished = {Technical Field Note \#05},
year = {2026},
month = oct,
url = {https://origindatalab.io/technical-field-notes/keystep-annotation-egocentric-robot-learning.html}
}
Related field notes
Annotated Demonstration Data
Your Ontology, Our Field Capture and QA
In production programs Origin Data Lab maps keysteps to your verb vocabulary and file format, runs dual annotation with agreement reporting, and ships the overlay videos and QA logs with every batch. Request the annotated evaluation sample used in this note.