Optimal importance sampling for federated learning
File(s)
Accepted version
Author(s)
Rizk, Elsa
Vlaski, Stefan
Sayed, Ali H
Type
Conference Paper
Abstract
Federated learning involves a mixture of centralized and decentralized processing tasks, where a server regularly selects a sample of the agents and these in turn sample their local data to compute stochastic gradients for their learning updates. The sampling of both agents and data is generally uniform; however, in this work we consider non-uniform sampling. We derive optimal importance sampling strategies for both agent and data selection and show that under convexity and Lipschitz assumptions, non-uniform sampling without replacement improves the performance of the original FedAvg algorithm. We run experiments on a regression and classification problem to illustrate the theoretical results.
Date Issued
2021-06-06
Date Acceptance
2021-01-30
Citation
ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp.3095-3099
Publisher
IEEE
Start Page
3095
End Page
3099
Journal / Book Title
ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Copyright Statement
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Identifier
https://ieeexplore.ieee.org/document/9413655
Source
ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Subjects
Science & Technology
Technology
Acoustics
Computer Science, Artificial Intelligence
Computer Science, Software Engineering
Engineering, Electrical & Electronic
Imaging Science & Photographic Technology
Computer Science
Engineering
federated learning
importance sampling
asynchronous SGD
non-IID data
heterogeneous agents
Publication Status
Published
Start Date
2021-06-06
Finish Date
2021-06-11
Coverage Spatial
Toronto, ON, Canada
Date Publish Online
2021-05-13