Annotators record which hands are involved, their shape and orientation, where they are and the path they follow. Each description ends with a short phrase explaining what the gesture communicates in context.
Start from previously annotated gesture regions in 10 TED Expressive talks and their transcripts.
A primary annotator describes the first 231 of the 778 TED clips. A second expert independently reviews each description and accepts or revises it.
Using the finalized guidelines, the primary annotator completes the remaining TED clips. The same schema is used for 100 BEAT2 clips.
The automatic pipeline generates anchors for training; the human annotations above provide a reference for evaluation.
Discretize 3D gesture motion into separate body and hand primitives.
Reconstruct eight-frame primitives and describe arm position, hand shape, palm orientation and motion using rules.
Combine the ordered primitive descriptions into a physical narrative for the gesture.
An LLM reads the motion narrative and transcript. It identifies the active hand or hands, describes the movement, infers its intent and checks that all three are consistent.
BEAT2 · Deictic
else that person that was wild because some people have an ax to grind and some people you or that person they want to grind
Both hands are held forward at chest level with arms slightly bent, in a fist, with index finger pointed forward, to underline “you” and the person being talked to.
BEAT2 · Iconic
in 2015 i was in a rut i was working a dead-end job and unsure what to do next i found myself at a crossroads
Both hands move outwards to both sides at about chest level with both palms upwards, creating a big shrug gesture to demonstrate uncertainty and underscore being stuck at a crossroad.
Annotations: SEMANTIX · CC BY-NC 4.0