Exact subspace diffusion for decentralized multitask learning
File(s)
Accepted version
Author(s)
Wadehra, Shreya
Nassif, Roula
Vlaski, Stefan
Type
Conference Paper
Abstract
Classical paradigms for distributed learning, such as federated or decentralized gradient descent, employ consensus mechanisms to enforce homogeneity among agents. While these strategies have proven effective in i.i.d. scenarios, they can result in significant performance degradation when agents follow heterogeneous objectives or data. Distributed strategies for multitask learning, on the other hand, induce relationships between agents in a more nuanced manner, and en-courage collaboration without enforcing consensus. We develop a generalization of the exact diffusion algorithm for subspace constrained multitask learning over networks, and derive an accurate expression for its mean-squared deviation when utilizing noisy gradient approximations. We verify numerically the accuracy of the predicted performance expressions, as well as the improved performance of the proposed approach over alternatives based on approximate projections.
Date Issued
2024-01-19
Date Acceptance
2023-12-01
Citation
2023 62nd IEEE Conference on Decision and Control (CDC), 2024, pp.6172-6179
Publisher
IEEE
Start Page
6172
End Page
6179
Journal / Book Title
2023 62nd IEEE Conference on Decision and Control (CDC)
Copyright Statement
Copyright © 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Identifier
http://dx.doi.org/10.1109/cdc49753.2023.10384238
Source
2023 62nd IEEE Conference on Decision and Control (CDC)
Publication Status
Published
Start Date
2023-12-13
Finish Date
2023-12-15
Coverage Spatial
Singapore, Singapore