A method for objectively evaluating the defect detection performance of in-situ monitoring systems
File(s)
Accepted version
Author(s)
de Winton, Harry
Cegla, Fred
Hooper, Paul
Type
Journal Article
Abstract
In-situ monitoring systems have the potential to assess material quality in additive manufacturing processes on-the-fly, paving the way for accelerated component qualification using defect digital twins. However, current systems vary widely in sensor technology and data analysis methods, leading to a lack of consensus in how the performance of these systems should be measured and compared. This work proposes a methodology and set of metrics, specifically Receiver Operating Characteristic (ROC) and Probability of Detection (POD) curves, to allow objective comparison of performance between any system, regardless of its underlying technology. We demonstrate this approach by comparing the ability to detect increases in part-wide porosity using two of the most common co-axial monitoring techniques in laser powder bed fusion; photodiodes and high-speed cameras. Using ROC curves, we show that melt pool metrics extracted from the camera offer a better trade off between detection and false alarms compared to the photodiodebased system when discriminating between samples at a 0.5% porosity threshold. POD curves were used to characterise detection capability across all porosity levels. It was found that the camera-based system can detect 43% of compromised parts (0.5% porosity), while the photodiode system detects 20%. However, for significantly compromised parts (5% porosity), the camera based method reaches 100%, while the photodiode only achieves 85%. The developed methodology shows that while
the camera-based system is measurably superior, further improvement is needed before commercial implementation can be realised. Ultimately, the ROC-POD methodology allows objective assessments of detection performance, enabling quantifiable progress in the development of defect detection systems based on in-situ monitoring.
the camera-based system is measurably superior, further improvement is needed before commercial implementation can be realised. Ultimately, the ROC-POD methodology allows objective assessments of detection performance, enabling quantifiable progress in the development of defect detection systems based on in-situ monitoring.
Date Issued
2021-12
Date Acceptance
2021-10-23
Citation
Additive Manufacturing, 2021, 48 (Part B), pp.1-13
ISSN
2214-8604
Publisher
Elsevier
Start Page
1
End Page
13
Journal / Book Title
Additive Manufacturing
Volume
48
Issue
Part B
Copyright Statement
© 2021 Elsevier Ltd. All rights reserved. This manuscript is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Licence http://creativecommons.org/licenses/by-nc-nd/4.0/
Sponsor
Engineering & Physical Science Research Council (E
AWE Plc
AWE Plc
Identifier
https://www.sciencedirect.com/science/article/pii/S2214860421005832?via%3Dihub
Grant Number
EP/K503733/1
See further info
PO 30458871
Subjects
0910 Manufacturing Engineering
Publication Status
Published
Date Publish Online
2021-11-02