Free record-level privacy risk evaluation through artifact-based methods
File(s)usenixsecurity25-pollock.pdf (2.51 MB)
Published version
Author(s)
Pollock, Joseph
Shilov, Igor
Dodd, Euodia
de Montjoye, Yves Alexandre
Type
Conference Paper
Abstract
Membership inference attacks (MIAs) are widely used to empirically assess privacy risks in machine learning models, both providing model-level vulnerability metrics and identifying the most vulnerable training samples. State-of-the-art methods, however, require training hundreds of shadow models with the same architecture as the target model. This makes the computational cost of assessing the privacy of models prohibitive for many practical applications, particularly when used iteratively as part of the model development process and for large models. We propose a novel approach for identifying the training samples most vulnerable to membership inference attacks by analyzing artifacts naturally available during the training process. Our method, Loss Trace Interquartile Range (LT-IQR), analyzes per-sample loss trajectories collected during model training to identify high-risk samples without requiring any additional model training. Through experiments on standard benchmarks, we demonstrate that LT-IQR achieves 92% precision@k=1% in identifying the samples most vulnerable to state-of-the-art MIAs. This result holds across datasets and model architectures with LT-IQR outperforming both traditional vulnerability metrics, such as loss, and lightweight MIAs using few shadow models. We also show LT-IQR to accurately identify points vulnerable to multiple MIA methods and perform ablation studies. We believe LT-IQR enables model developers to identify vulnerable training samples, for free, as part of the model development process. Our results emphasize the potential of artifact-based methods to efficiently evaluate privacy risks.
Date Issued
2025-09-08
Date Acceptance
2025-06-06
Citation
Proceedings of the 34th USENIX Security Symposium, 2025, pp.5525-5544
ISBN
978-1-939133-52-6
Publisher
USENIX Association
Start Page
5525
End Page
5544
Journal / Book Title
Proceedings of the 34th USENIX Security Symposium
Copyright Statement
Copyright © 2025 The USENIX Association. USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone.
Source
34th USENIX Security Symposium
Publication Status
Published
Start Date
2025-08-13
Finish Date
2025-08-15
Coverage Spatial
Seattle, WA, USA
Date Publish Online
2025-08-13