With recent development in virtual reality, computer simulation is increasingly being used in surgery for training, diagnosis, and pre-/intra-operative planning. Motivated by the current demand on clinical governance, surgical simulation is now a well-established modality for basic skills training and assessment. The practical deployment of the technique is a multi-disciplinary venture, encompassing many topics in engineering, medicine, and psychology. One of the key issues to be addressed in surgical simulation is the creation of photorealistic rendering for enhancing the visual fidelity and hence the overall quality of the simulation experience. Simulation of the human body, however, is a challenging task as it is complicated by the diverse material properties and deformable nature of in vivo structures. Due to the significant resources required to model the surgical scene, conventional computer graphics techniques generally fall short of meeting these objectives, particularly in terms of visual realism and real-time responses.
The purpose of this thesis is to describe a new framework of photorealistic rendering for surgical simulation with enhanced visual fidelity and interactivity. The thesis begins with a review of the different stages involved in surgical simulation and its associated technical challenges. The current state-of-the-art of the technique is discussed, which is followed by a detailed description of Image-Based Rendering (IBR), a technique that has attracted significant interests in recent years due to its convergence from computer vision, graphics, and image processing. IBR uses images as modelling and rendering primitives and has established itself as a unique alternative to geometry-based computer graphics. In this thesis, a classification of the current IBR techniques is introduced and particular emphasis has been placed on methods that allow enhanced depth perception for interactive environments.
To improve visual realism for endoscopic simulation and panorama, we have introduced a method for mapping depth-enhanced images onto cylindrical manifolds. An image-based approach for simulating soft tissue deformation is subsequently developed. The method is based on associating a depth map with the texture image and incorporating micro-surface details to achieve photorealistic rendering. To cater for general viewing geometry encountered in minimal invasive surgery, the technique is further extended to incorporate a number of virtual cameras for sampling from multiple viewpoints. To achieve enhanced visual realism, we have also investigated the effect of lighting and its associated specular reflection on tissue appearance. Human tissue is characterized by its inhomogeneous composite multi-layer construction, resulting in a largely random lighting behaviour. To simulate this effect, a noise-based empirical lighting model for producing a visual effect similar to that in real surgery is introduced. The proposed technique minimises the computation resources required for high fidelity surgical visualisation such that it can be efficiently integrated with biomechanical modelling.
To assess the quality of the perceived visual realism by using the proposed rendering techniques, visual scoring based on eye tracking is performed. Both fixation and saccadic eye movements are used to determine the underlying visual features that affect the overall quality of synthetic images. The use of eye tracking permits objective assessment of how visual realism is affected by individual image features, thus allowing a systematic examination of the strength, as well as potential drawbacks of the methods developed.