Evaluating privacy and robustness in modern data processing systems

Abstract

Modern technologies collect, process, and share our data on an unprecedented scale. Understanding how these technologies affect our privacy before they are deployed is a key question. This involves taking an adversarial approach to evaluate privacy. Privacy attacks evaluate the robustness of technologies to adversaries aiming to learn sensitive information about individuals. In this thesis, we target technologies whose risks are not well understood in practice, such as anonymisation techniques for dynamic datasets, query-based systems (QBS), machine learning (ML) models, and on-device client-side scanning (CSS) for illegal content detection. We argue that there is a need to develop new threat models and methodologies to comprehensively evaluate their risks in practice. We make five contributions, proposing new threat models and attacks.

Our first contribution is a study of the robustness of anonymisation techniques relying on pseudonymisation and frequent re-pseudonymisation for interaction data, showing that people's interaction data can be accurately identified in large-scale populations even after multiple weeks. We then turn to aggregation systems, which are believed to be safer than releasing individual-level data. Our second contribution is to develop a method to automatically discover privacy vulnerabilities in QBSs that matches or outperforms previous expert-designed attacks. Third, we study a new type of leakage in ML models, the leakage of correlations between the input attributes of a tabular training dataset, showing that correlations can be used as building blocks for stronger attribute inference attacks. Fourth, we study white-box membership inference attacks, analysing the impact of misalignment, a known characteristic of deep neural networks stemming from their weight symmetries, on their performance. Finally, we develop detection avoidance attacks against very recently proposed perceptual hashing-based CSS systems, in order to evaluate whether their privacy risks can be balanced by effective detection.