KungFu: making training in distributed machine learning adaptive
File(s)
Published version
Author(s)
Type
Conference Paper
Abstract
When using distributed machine learning (ML) systems to train models on a cluster of worker machines, users must con-figure a large number of parameters: hyper-parameters (e.g. the batch size and the learning rate) affect model convergence; system parameters (e.g. the number of workers and their communication topology) impact training performance. In current systems, adapting such parameters during training is ill-supported. Users must set system parameters at deployment time, and provide fixed adaptation schedules for hyper-parameters in the training program. We describe Kung Fu, a distributed ML library for Tensor-Flow that is designed to enable adaptive training. Kung Fu allows users to express high-level Adaptation Policies(APs)that describe how to change hyper- and system parameters during training. APs take real-time monitored metrics (e.g. signal-to-noise ratios and noise scale) as input and trigger control actions (e.g. cluster rescaling or synchronisation strategy updates). For execution, APs are translated into monitoring and control operators, which are embedded in the data flowgraph. APs exploit an efficient asynchronous collective communication layer, which ensures concurrency and consistency of monitoring and adaptation operations
Date Issued
2020-11
Date Acceptance
2020-08-31
Citation
Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation, 2020, pp.937-954
ISBN
9781939133199
Publisher
Usenix
Start Page
937
End Page
954
Journal / Book Title
Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation
Copyright Statement
© 2020 The Author(s).
Sponsor
Huawei Technologies Co. Ltd
Identifier
https://www.usenix.org/conference/osdi20/presentation/mai
Grant Number
YBN2017100016
Source
USENIX Symposium on Operating Systems Design and Implementation (OSDI)
Publication Status
Published
Start Date
2020-11-04
Finish Date
2020-11-06
Coverage Spatial
Virtual
Date Publish Online
2020-11-04