Paper: arXiv, CVPR'25 | Software: GitHub | Keywords: co-speech gestures, diffusion models, retrieval-augmented generation, neuro-explicit

Some example results from RAG-Gesture. Please unmute to follow the speech.

---

Why gestures matter (and why they are hard)

Human communication is not just words. When people speak, they continuously move their hands, arms, and bodies, and these co-speech gestures are tightly synchronised with speech in both time and meaning. For embodied agents — think telepresence avatars, game characters, digital assistants or humanoid robots — getting these gestures right is a big part of looking believable rather than “robotic”.

Researchers usually divide co-speech gestures into two families:¹

• Beat gestures — rhythmic movements driven by the prosody (the “rhythm”) of speech. They mark emphasis but do not carry specific meaning.
• Semantic gestures — movements that carry communicative intent. These include iconic gestures that mimic a concrete shape or action, metaphoric gestures that give abstract ideas a physical form, and deictic gestures that point at things.

Here is the catch. Modern neural networks learn from large gesture datasets, and in those datasets beat gestures are everywhere while semantic gestures appear only sporadically. So data-driven models become very good at producing plausible rhythmic motion, but they rarely produce the meaningful gestures that make communication rich. Older rule-based systems had the opposite problem: they could slot in the “right” semantic gesture, but stitching pre-recorded clips together tended to look stiff and unnatural.

The key idea: treat gesture synthesis as retrieval-augmented generation

Large language models faced a related problem — they “know” a lot but can miss specific facts — and a popular fix is Retrieval Augmented Generation: look the fact up in a database, then let the model generate an answer around it. We asked a simple question: what if meaningful gestures can be treated as specific facts? When a word in the speech is likely to call for a meaningful gesture, retrieve a good example of that gesture from a database, and let the generator build a natural motion around it.

Following an insight from Neff², we split the problem into two cleaner sub-problems:

1. Specification — which word should be gestured, and what kind of gesture should it get? This is where explicit linguistic knowledge comes in.
2. Animation — how do we actually produce that gesture and blend it seamlessly into the rest of the motion? This is where the diffusion model comes in.

As a result, this division makes our method neuro-explicit: the specification step uses interpretable, human-understandable rules grounded in linguistics and gesture theory, while the animation step uses a powerful neural generator. Each part does what it is best at.

A short overview of the method. Please unmute to follow the narration.

Step 1 — Specification: deciding what to retrieve

The retrieval step searches a database of co-speech gestures (we use the BEAT2 dataset³ for a good example to match a given word. We propose two complementary ways of doing this. Both end with the same two refinement steps — they prefer examples from the same speaker (since each person gestures in their own idiosyncratic way) and then re-rank by prosodic prominence, because gestures tend to land on the parts of speech we stress with pitch, loudness, or duration.

Option A — LLM-based gesture-type retrieval. Large language models can reason about a sentence. We simply prompt one to read a sentence and point out which words are likely to trigger a meaningful gesture, and what type each gesture should be (iconic, metaphoric, or deictic). For the sentence “He told me it was because he lives in Chigasaki,” the model might flag because as metaphoric and lives in as iconic. We then pull matching examples from the database using those type labels.

Option B — Discourse-based retrieval. Language has structure: clauses are linked by relations such as cause–consequence and contrast, and these are often signalled by little connective words like because, while, or on the other hand. Linguists have shown that these connectives systematically influence how people gesture⁴ — for example, contrast is often expressed by placing two arguments on two separate hands. We run a discourse parser over the speech, identify the connective and its relational sense, and retrieve gestures that co-occur with the same sense. A nice bonus: because we match on meaning rather than the exact word, a gesture recorded for because can inform a gesture for a causal since, which helps us find examples even for rare connectives.

A walkthrough of how the retrieval algorithms work. Please unmute to follow the narration.

Step 2 — Animation: injecting the retrieved gesture into a diffusion model

Now we have a retrieved gesture clip and we know which words in our new sentence it should land on. The challenge is to weave it into the generated motion so that it is meaningful and natural — no visible jumps in motion, no stiff copy-pasted sequence.

Our base generator is a latent diffusion model. In plain terms, a diffusion model learns to turn random noise into realistic data by removing the noise step by step; here it generates motion conditioned on the input speech (encoded with wav2vec2), the transcript (encoded with BERT), and the speaker’s identity. Rather than working on raw joint angles, we compress the motion of different body regions (upper body, hands, face, lower body) into compact latents with separate VAEs⁵, which keeps generation quality high.

The naive approaches both do not complete the job. Pure neural generation ignores our carefully retrieved gesture, while bluntly “pasting” the retrieved clip in (a kind of inpainting) forces the model to copy it exactly and hurts naturalness. We do something nuanced, in two parts.

Latent Initialization via DDIM inversion. A diffusion model can be run backwards to convert a finished motion clip into the noise that would have produced it. We use this trick (DDIM inversion) to push the retrieved gesture back into the model’s noise space, slice out just the window that corresponds to the gestured word, and place that slice into the starting noise of our new sequence at the matching word location. The generation now begins from a state that already knows about the retrieved gesture.

Retrieval Guidance. Initialization alone does not control how strongly the final motion follows the retrieved example. So at each denoising step we add a gentle nudge — a gradient step on a simple objective that measures the distance between the generated motion and the retrieved one inside the insertion window:

\[ G_{\text{retrieval}} = \left\lVert \hat{z}^{(t)}_{\text{retr}}[s_{\text{query}} : e_{\text{query}}] - \hat{r}^{(t)}[s_{\text{retr}} : e_{\text{retr}}] \right\rVert_2^2 \]

By choosing how many of these nudges to apply, the user gets a control knob for the influence of the retrieval: apply none and the retrieval only shapes the very start of generation; apply many and the motion follows the example closely. Crucially, all of this happens at inference time — we never retrain or fine-tune the diffusion model to use retrieval.

How Latent Initialization and Retrieval Guidance inject the exemplar gesture. Please unmute to follow the narration.

Does it work?

We visualise the retrieved gestures (shown in red) being merged into the generated motion. The augmented gestures appear precisely on the semantically important parts of speech, while the rest of the motion stays untouched.

Retrieval-augmented generation, example 1. Retrieved exemplars are visualised in red. Please unmute to follow the speech.

Retrieval-augmented generation, example 2.

Side-by-side qualitative comparison with baselines. Please unmute to follow the narration.

Perceptual Evaluation. Gesture quality is ultimately about human perception, so we ran user studies in which participants compared our gestures with those of the baselines, judging both naturalness and appropriateness to the speech. RAG-Gesture was preferred across the board; for the LLM-based variant, preferences came close to those for real, ground-truth motion. We also compared our training-free approach against a training-based RAG baseline built in the spirit of prior work, and participants still preferred our inference-time method by a wide margin — a nice illustration that you do not always need to retrain a big model to make it do something new.

Quantitative comparison. We evaluate on the BEAT2 dataset against recent data-driven baselines — CaMN³, EMAGE⁵, Audio2Photoreal⁶ — and the retrieval-augmented motion model ReMoDiffuse⁷. When trained on all 25 speakers, RAG-Gesture reaches state-of-the-art numbers on metrics that capture motion quality, beat alignment, and diversity, which shows that the approach scales and generalises across many different speaking styles.

How much retrieval is too much? Because our method has that control knob, we could study the sweet spot directly. Interestingly, evaluators liked neither extreme: not the purely data-driven output with no retrieval, nor the rigid copy-paste of full inpainting. The balanced setting — a strong nudge toward the retrieval that still lets the diffusion model keep things natural — was preferred.

Takeaways

RAG-Gesture shows that a long-standing weakness of neural gesture generators — their difficulty with rare but meaningful gestures — can be addressed by bringing back explicit knowledge in a principled way:

• Retrieval, not memorisation. Instead of hoping a network learns sparse semantic gestures from data, we look them up explicitly using linguistic structure (discourse relations) or LLM reasoning (based on gesture types).
• Injection without retraining. A combination of DDIM-based latent initialization and retrieval guidance lets us insert exemplar gestures into a frozen diffusion model purely at inference time, with a tunable control over their influence.
• Best of both worlds. This neuro-explicit recipe combines the naturalness of data-driven generation with the semantic precision of retrieval, and human evaluators consistently prefer the result.

The framework is also flexible: any retrieval algorithm can be plugged in, which opens the door to gestures driven by other goals — referential, emotional, or otherwise task-specific patterns. We hope it is a useful step toward embodied agents that do not just move while they talk, but gesture with meaning.

For the full set of results, we encourage you to watch the supplementary video and explore the project page.

References