Tenplex: dynamic parallelism for deep learning using parallelizable tensor collections
File(s)
Published version
Author(s)
Wagenländer, Marcel
Li, Guo
Zhao, Bo
Mai, Luo
Pietzuch, Peter
Type
Conference Paper
Abstract
Deep learning (DL) jobs use multi-dimensional parallelism, i.e., combining data, model, and pipeline parallelism, to use large GPU clusters efficiently. Long-running jobs may experience changes to their GPU allocation: (i) resource elasticity during training adds or removes GPUs; (ii) hardware maintenance may require redeployment on different GPUs; and (iii) GPU failures force jobs to run with fewer devices. Current DL frameworks tie jobs to a set of GPUs and thus lack support for these scenarios. In particular, they cannot change the multi-dimensional parallelism of an already-running job in an efficient and model-independent way.
We describe Tenplex, a state management library for DL systems that enables jobs to change their parallelism dynamically after the GPU allocation is updated at runtime. Tenplex achieves this through a new abstraction, a parallelizable tensor collection (PTC), that externalizes the job state during training. After a GPU change, Tenplex uses the PTC to transform the job state: the PTC repartitions the dataset state under data parallelism and exposes it to GPU workers through a virtual file system; and the PTC obtains the model state as partitioned checkpoints and transforms them to reflect the new parallelization configuration. For efficiency, Tenplex executes PTC transformations in parallel with minimum data movement between GPU workers. Our experiments show that Tenplex enables DL jobs to support dynamic parallelization with low overhead.
We describe Tenplex, a state management library for DL systems that enables jobs to change their parallelism dynamically after the GPU allocation is updated at runtime. Tenplex achieves this through a new abstraction, a parallelizable tensor collection (PTC), that externalizes the job state during training. After a GPU change, Tenplex uses the PTC to transform the job state: the PTC repartitions the dataset state under data parallelism and exposes it to GPU workers through a virtual file system; and the PTC obtains the model state as partitioned checkpoints and transforms them to reflect the new parallelization configuration. For efficiency, Tenplex executes PTC transformations in parallel with minimum data movement between GPU workers. Our experiments show that Tenplex enables DL jobs to support dynamic parallelization with low overhead.
Date Issued
2024-11-15
Date Acceptance
2024-11-04
Citation
SOSP '24: Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, 2024, pp.195-210
ISBN
9798400712517
Publisher
ACM
Start Page
195
End Page
210
Journal / Book Title
SOSP '24: Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles
Copyright Statement
© 2024 Copyright held by the owner/author(s).
This work is licensed under a Creative Commons Attribution International 4.0 License.
This work is licensed under a Creative Commons Attribution International 4.0 License.
License URL
Identifier
http://dx.doi.org/10.1145/3694715.3695975
Source
SOSP '24: ACM SIGOPS 30th Symposium on Operating Systems Principles
Publication Status
Published
Finish Date
2024-11-06
Coverage Spatial
TX, Austin, USA
Date Publish Online
2024-11-15