We propose a method for computing on-line the controller of an Autonomous Underwater Vehicle under thruster failures. The method is general and can be applied to both redundant and under-actuated AUVs, as it does not rely on the modification of the thruster control matrix. We define an optimization problem on a specific class of functions, in order to compute the optimal control law that achieves the target without using the faulty thruster. The method is framed within model-based policy search for reinforcement learning, and we study its applicability on the model of the AUV Girona500. We performed experiments with policies of increasing complexity, testing the on-line feasibility of the approach as the optimization problem becomes more complex.