The recent momentum in aerial manipulation has led to an interest in developing virtual reality interfaces for aerial physical interaction tasks with simple, intuitive, and reliable control and perception. However, this requires the use of expensive subsystems and there is still a research gap between interface design, user evaluations and the effect on aerial manipulation tasks. Here, we present a methodology for low-cost available drone systems with a Unity-based interface for immersive FPV teleoperation. We applied our approach in a flight track where a cluttered environment is used to simulate a demanding aerial manipulation task inspired by forestry drones and canopy sampling. Through objective measures of teleoperation performance and subjective questionnaires, we found that operators performed worse using the FPV interface and had higher perceived levels of cognitive load when compared to traditional interface design. Additional analysis of physiological measures highlighted that objective stress levels and cognitive load were also influenced by task duration and perceived performance, providing an insight into what interfaces could target to support teleoperator requirements during aerial manipulation tasks.