Affordance inference and visualisation for assistive robotics

Abstract

The control of assistive robots can be challenging for some users, especially those lacking experience or suffering from disabilities. AR UIs have the potential to facilitate this task. In this thesis, we propose the use of affordance-aware AR HMD UIs as a novel design paradigm for the control of assistive robotic platforms. Rather than expecting users to know what capabilities the assistive robots have and what input commands are required to leverage these capabilities, we use the robots' sensor data to infer action possibilities in the environment and inform the users about what the robots can accomplish. We demonstrate that the proposed affordance-aware UIs can be effectively used for the control of complex robots, from smart wheelchairs, legged and mobile manipulators to multi-robot assistive platforms. The UI resulted in a significant improvement in the ease of control of a smart wheelchair. The NASA-Task Load Index (TLX) score reported by the experienced users was lower for the task of controlling the smart wheelchair with our AR UI (19.5, standard deviation (SD) = 11.0) than when using the joystick (39.1, SD = 17.3). The same result occurred with the non-experienced users that reported a NASA-TLX score of 19.9 (SD = 10.1) when using the AR UI and 44.79 (SD = 22.9) when using the joystick. Furthermore, our proposed UI positively impacted subjective and objective metrics of fluency in HRI when applied to the control of a multi-robot assistive platform; especially when virtual elements that automatically combine multiple robot actions are encapsulated in the UI. Subjective metrics were rated higher by the participants group that triggered multiple robot behaviours simultaneously, giving a mean score of 6.7, higher than the 5.98 reported by the group of participants who triggered those behaviours separately instead. Furthermore, the mean F-DEL value we measured as an objective metric due to its strong inverse correlation with subjective fluency was lower for the former group of participants. Moreover, our algorithm for the refinement of object positions obtained via vision-based colocalisation drastically improved the performance of the grasping behaviour of legged manipulators. Using this algorithm, we improved the average success rate for such behaviour from 31% to 75%. Finally, our algorithm for inferring longer horizon tasks combined atomic actions effectively. Thus, providing higher-level options to the user of an assistive mobile manipulator. 60% of all possible interactions are output after considering the first 100 relevant plans in our simulated bedroom environments. Thus, reducing the time to present options to the user when the algorithm considers all relevant plans from 24.7 s to 5.96 s. In addition, in contrast to other state-of-art proactive planning methods, our algorithm provides means for dichotomising the search space, which results in a reduction of at least 47% of the time associated with planner usage. The presented research illustrates the potential that affordance-aware AR UIs have on improving the control of assistive robots and bringing advanced robotics facilities to their users.