Efficient and Robust Federated Learning with Diverse and Dynamic Data

Abstract

Traditional machine learning algorithms require data to be centralized. In practice, data is often generated at multiple locations. Transmitting all the data to a central location is often impractical due to resource constraints but also privacy concerns. Federated learning addresses this problem by allowing clients to collaboratively learn a global model, while keeping their raw data stored locally. Existing federated learning approaches face some limitations when applied to real-world systems. This thesis aims to address some of these limitations, specifically to improve resource efficiency and robustness of federated learning with heterogeneous and dynamic data.

In the first part of this thesis, we optimize resource efficiency and robustness of federated learning under spatial data diversity. We propose an algorithm able to learn systems characteristics in real-time and dynamically adapt the frequency of aggregation to maximize the learning accuracy for a given resource budget. Furthermore, most existing federated learning approaches focus on training a model using pre-defined datasets at the client nodes. However, the spatial diversity of data collected at each client can often affect model accuracy. To consider this data diversity but also noisy data, we propose an approach to select only relevant data for a given machine learning task.

In the second part of this thesis, we address temporal data heterogeneity, which is caused by clients dropping out of the systems before completion of the learning task due to energy or connectivity constraints at the clients. To this end, we propose a continual learning approach that allows the global model to continuously learn on data changing over time, without forgetting previously learned information. This new continual learning approach is readily applicable to federated learning.

Finally, aiming to address problems related to system heterogeneity, we propose an approach to efficiently monitor and forecast resource utilisation in large-scale and heterogeneous distributed systems.