Dear Spiral user, We’re pleased to introduce Helix, https://helix.imperial.ac.uk/ Imperial’s new repository for research datasets. Please use Helix for all future dataset deposits. There is no change to Spiral - continue depositing as usual. For queries, contact the Research Data Management team at rdm-enquiries@imperial.ac.uk. Kind regards, the Spiral Team. The Spiral team
Second-order guarantees in centralized, federated and decentralized nonconvex optimization
OA Location
Author(s)
Vlaski, Stefan
Sayed, Ali H
Type
Journal Article
Abstract
Rapid advances in data collection and processing capabilities have allowed for the use of increasingly complex models that give rise to nonconvex optimization problems. These formulations, however, can be arbitrarily difficult to solve in general, in the sense that even simply verifying that a given point is a local minimum can be NPhard [1]. Still, some relatively simple algorithms have been shown to lead to surprisingly good empirical results in many contexts of interest. Perhaps the most prominent example is the success of the backpropagation algorithm for training neural networks. Several recent works have pursued rigorous analytical justification for this phenomenon by studying the structure of the nonconvex optimization problems and establishing that simple algorithms, such as gradient descent and its variations, perform well in converging towards local minima and avoiding saddle-points. A key insight in these analyses is that gradient perturbations play a critical role in allowing local descent algorithms to efficiently distinguish desirable from undesirable stationary points and escape from the latter. In this article, we cover recent results on second-order guarantees for stochastic first-order optimization algorithms in centralized, federated, and decentralized architectures.
Date Issued
2020
Date Acceptance
2020-12-01
Citation
Communications in Information and Systems, 2020, 20 (3), pp.353-388
URI
URL
DOI
ISSN
1526-7555
Publisher
International Press
Start Page
353
End Page
388
Journal / Book Title
Communications in Information and Systems
Volume
20
Issue
3
Copyright Statement
Copyright © 2020 The Author(s).
Identifier
https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000595949500005&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=a2bf6146997ec60c407a63945d4e92bb
Subjects
Computer Science
Computer Science, Information Systems
CONSENSUS
CONVERGENCE
LEARNING-BEHAVIOR
Science & Technology
Technology
Publication Status
Published
Date Publish Online
2020-12-02