Leveraging the multi-modal nature of communication in immersive XR environments
File(s)
Author(s)
Bovo, Riccardo
Type
Thesis
Abstract
In the evolving landscape of immersive spatial computing, such as extended reality (XR), effective collaboration remains a cornerstone of productivity and innovation. In traditional co-located work, collaborators naturally align both their visual and cognitive attention through a blend of gaze, gesture, and referring expressions, enabling rapid establishment of common ground and precise coordination. This thesis investigates how such attentional cues can be reconstructed and visualised in immersive collaboration settings, particularly on low-cost VR/AR headsets that lack eye-tracking. Rather than treating raw gaze as the sole proxy for intent, the work adopts a complementary reconstruction approach that combines naturally occurring collaborative behaviours (head movement and verbal communication) with the shared spatial context of the task to approximate a partner’s attention and expose it through adaptive visual cues. Across a series of empirical studies, the thesis introduces and evaluates three components. First, a probabilistic Cone-of-Vision derived from head orientation supports broad mutual awareness without requiring eye-tracking. Second, a deep learning model refines this coarse signal, reducing spatial ambiguity while avoiding the computational cost of full computer vision pipelines. Third, a Speech-Augmented Cone-of-Vision integrates referring expressions in real time, dynamically scaling from scene-level awareness to object-level focus as collaborators speak. These findings challenge the assumption that raw sensor data is the gold standard for visual attention cues. By demonstrating the importance of broad attention cues for collaborative alignment and the role of speech in reducing ambiguity, this thesis proposes a shift from merely replicating gaze mechanics to visualizing the higher-order of cognitive attention. Ultimately, this work underscores that leveraging naturally occurring behaviours is not just a hardware workaround, but a fundamental requirement for the hyper-productive, augmented collaboration promised by spatial computing.
Version
Open Access
Date Issued
2025-03-09
Date Awarded
2026-03-01
Copyright Statement
Attribution-NonCommercial 4.0 International Licence (CC BY-NC)
License URL
Advisor
Heinis, Thomas
Sponsor
UK Research and Innovation
Engineering and Physical Sciences Research Council
Schlumberger Limited
Grant Number
EPSRC Industrial CASE student award No. EP/T517690/1
Publisher Department
Department of Computing
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)