VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents

A benchmark that tests whether AI agents can listen, speak and show nonverbal signals at the same time, as people do on a video call. It finds today's systems fall well short of humans, and that avatar pipelines built from separate speech and video steps cannot react while the user is still talking.

Publisher

arXiv (preprint; NVIDIA and David AI)

Authors

Amrita Mazumdar, Seonwook Park, Rajarshi Roy, Nikhil Srihari, Shengze Wang, Yuhao Zhou, Julia Wang, Koki Nagano, Shalini De Mello

Published

2026-05-28

Versions

Preprint, version 1. Not yet peer-reviewed.

Research area

Embodied AI & Decision Support

Type

paper

Original

arxiv.org

DOI

10.48550/arXiv.2605.30256

Tags

PerformanceAI embodiment

Keywords

benchmark, turn-taking, nonverbal cues, latency, full-duplex, avatars

Key points

In our own words. These are the only statements our AI presenters cite from this source.

1. kp-1It is presented as the first benchmark for conversational agents that both see and speak in real time, rather than testing speech alone.
2. kp-2It uses 237 short clips from real two-person video calls, covering 11 kinds of nonverbal conversational behavior.
3. kp-3Current vision-and-speech models use video to answer explicit questions about what they see, but largely ignore it during natural conversation; audio-only versions sometimes did better.
4. kp-4Avatar systems that chain speech recognition, a language model and a talking-head renderer take turns politely but cannot give cues such as nods while the user speaks.
5. kp-5In that chained setup, responses lagged human timing by roughly 3 seconds, and the appropriateness of nonverbal cues fell sharply compared with people.

Summary prepared by Blue Ocean Labs and pending source review. Always rely on the original text.

Our notes