Appearance-based methods for object recognition and visual localisation from hand-held and wearable cameras

Abstract

Visual localisation and object recognition are key goals of artificial intelligence research that have been traditionally investigated separately. Appearance-based methods can be used to treat both problems from a common perspective. Therefore, the main purpose of this thesis is to explore appearance-based methods in the specific contexts of object recognition and visual localisation from wearable and hand-held devices. Specifically, the contributions of this thesis are as follows:

The first topic of study was the object recognition of grocery products acquired with hand-held and wearable cameras, a use case of particular relevance for the blind and partially sighted people. The main contributions around this topic are a) the SHORT dataset, comprising 100 categories and more than 135,000 images between its training and query sets; and b) an open-source pipeline and complete evaluation of popular bag-of-visual-words (BoVW) techniques when tested against SHORT. The SHORT dataset is novel as it introduces a clear distinction between high quality training images and query images taken in the wild. This is an anticipated scenario in which retailers would acquire images for their online shopping brochures and users would submit images of unpredictable quality for recognition. The performance results of the methods tested demonstrate the challenging characteristics of SHORT.

The second subject of study was indoor localisation from hand-held and wearable cameras. For this topic, the RSM dataset was constructed, containing more than 90,000 video frames along more than 3 km of indoor journeys. An open-source pipeline and evaluation is also contributed in this area. The methods include a selection of custom-created single-frame and spatio-temporal image description methods. These are tested against baseline appearance-based methods such as SIFT and HOG3D and state-of-the-art SLAM. Results show that appearance-based methods, even in the absence of tracking, can provide enough information to infer location with errors as small as 1.5 m over a 50 m journey. From the methods studied, results suggest that single-frame approaches perform slightly better than spatio-temporal ones.

In third place, I have developed a novel biologically inspired model of artificial place cells based on kernel distance metrics of appearance-based methods between query and database images. Localisation performance was also tested against the RSM dataset, achieving errors as low as 1.4 m over a 50 m trajectory and comparing favourably with the state of the art SLAM.

Finally, I have prototyped an assistive localisation system using wearable or hand-held visual input and tactile feedback to track the localisation of the user over haptic maps. An evaluation of the quality of the tactile feedback using this approach is also provided.