A method for objectively evaluating the defect detection performance of in-situ monitoring systems

Abstract

In-situ monitoring systems have the potential to assess material quality in additive manufacturing processes on-the-fly, paving the way for accelerated component qualification using defect digital twins. However, current systems vary widely in sensor technology and data analysis methods, leading to a lack of consensus in how the performance of these systems should be measured and compared. This work proposes a methodology and set of metrics, specifically Receiver Operating Characteristic (ROC) and Probability of Detection (POD) curves, to allow objective comparison of performance between any system, regardless of its underlying technology. We demonstrate this approach by comparing the ability to detect increases in part-wide porosity using two of the most common co-axial monitoring techniques in laser powder bed fusion; photodiodes and high-speed cameras. Using ROC curves, we show that melt pool metrics extracted from the camera offer a better trade off between detection and false alarms compared to the photodiodebased system when discriminating between samples at a 0.5% porosity threshold. POD curves were used to characterise detection capability across all porosity levels. It was found that the camera-based system can detect 43% of compromised parts (0.5% porosity), while the photodiode system detects 20%. However, for significantly compromised parts (5% porosity), the camera based method reaches 100%, while the photodiode only achieves 85%. The developed methodology shows that while the camera-based system is measurably superior, further improvement is needed before commercial implementation can be realised. Ultimately, the ROC-POD methodology allows objective assessments of detection performance, enabling quantifiable progress in the development of defect detection systems based on in-situ monitoring.