The established Machine Learning algorithm Random Forest (RF) has previously been shown to be effective at performing automated defect detection for test pieces which have been processed using fluorescent penetrant inspection (FPI). The work presented here investigates three methods (two previously proposed in other fields, one novel method) of modifying the FPI RF based on the individual performance of decision trees within the RF. Evaluating based on the 2 Score, which is the harmonic mean of precision and recall which places a larger weighting on recall, it is possible to reduce the RF in size by up to 50%, improving speed and memory requirements, whilst still gain equivalent results to a full RF. Introducing a performance based weighting or retraining decision trees which fall below a certain performance level however, offers no improvement on results for the increased computation time required to implement.