Introducing AgentHands
Google Research has created AgentHands, a research prototype designed to augment conversational agents within extended reality (XR) environments. The system utilizes a large language model (LLM) to generate synchronized, expressive hand gestures, aiming to bridge the ‘mental mapping gap’ and improve user engagement in physical tasks. This work builds on prior research in Human-Input/Output (I/O) and Sensible Agent technologies.
A Multi-Dimensional Taxonomy
The development of AgentHands began with a formative study involving Google’s XR and Human-Computer Interaction (HCI) experts. This research resulted in a multi-dimensional taxonomy defining the characteristics of effective virtual hand agents in XR. The taxonomy considers six key dimensions: Handedness & Gesture, Spatiality, Temporal Dynamics & Visual Effects, Interactivity, and Visual Effects. These dimensions guide the creation of a library of hand gesture behaviors, categorized as deictic, iconic, or expressive, to facilitate LLM selection and execution.
Workflow and Implementation
The AgentHands system operates through a defined workflow. Initially, a lightweight object registration module uses eye gaze and scene reconstruction to allow users to ‘tag’ objects within the XR environment, creating a 3D registry with bounding boxes. The LLM then generates responses containing ‘GestureEvents,’ which are attached to specific trigger words and encode the primitives for a hand behavior based on the taxonomy. A local parser on the XR headset coordinates the text-to-speech (TTS) playback with the animation engine, ensuring synchronized hand gestures.
Demonstrations and Applications
Demonstrations showcased the system’s capabilities across various scenarios. In an orchid-care task, the agent demonstrates the process of checking the roots by moving its hands to the plant and outlining the air roots. For 3D printer operations, the agent provides precise knob interactions for menu navigation. The system’s ability to translate linguistic intent into physical action enhances understanding of the user’s surroundings and facilitates complex instructions.
Technical Details
The AgentHands system integrates a lightweight object registration module, a library of hand gesture behaviors, and a local parser for synchronized rendering. The system transforms standard LLM output into a rich, multimodal performance where the agent’s generated responses are manifested through both speech and spatially accurate movement.
