Linear speedup in saddle-point escape for decentralized non-convex optimization
File(s)1910.13852v1.pdf (363.64 KB)
Accepted version
Author(s)
Vlaski, Stefan
Sayed, Ali H
Type
Conference Paper
Abstract
Under appropriate cooperation protocols and parameter choices, fully decentralized solutions for stochastic optimization have been shown to match the performance of centralized solutions and result in linear speedup (in the number of agents) relative to non-cooperative approaches in the strongly-convex setting. More recently, these results have been extended to the pursuit of first-order stationary points in non-convex environments. In this work, we examine in detail the dependence of second-order convergence guarantees on the spectral properties of the combination policy for non-convex multi agent optimization. We establish linear speedup in saddle-point escape time in the number of agents for symmetric combination policies and study the potential for further improvement by employing asymmetric combination weights. The results imply that a linear speedup can be expected in the pursuit of second-order stationary points, which exclude local maxima as well as strict saddle-points and correspond to local or even global minima in many important learning settings.
Date Issued
2020-04-09
Date Acceptance
2020-05-01
Citation
ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp.8589-8593
ISSN
1520-6149
Publisher
IEEE
Start Page
8589
End Page
8593
Journal / Book Title
ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Identifier
https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000615970408172&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=a2bf6146997ec60c407a63945d4e92bb
Source
IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)
Subjects
Acoustics
centralized algorithm
decentralized algorithm
diffusion strategy
Engineering
Engineering, Electrical & Electronic
LEARNING-BEHAVIOR
minima
Non-convex optimization
saddle-point
Science & Technology
second-order stationarity
Technology
Publication Status
Published
Start Date
2020-05-04
Finish Date
2020-05-08
Coverage Spatial
SPAIN, Barcelona