Tempo: compiled dynamic deep learning with symbolic dependence graphs
File(s)
Published version
Author(s)
Silvestre, Pedro F
Pietzuch, Peter
Type
Conference Paper
Abstract
Deep learning (DL) algorithms are often defined in terms of temporal relationships: a tensor at one timestep may depend on tensors from earlier or later timesteps. Such dynamic dependencies (and corresponding dynamic tensor shapes) are difficult to express and optimize: while eager DL systems support such dynamism, they cannot apply compiler-based optimizations; graph-based systems require static tensor shapes, which forces users to pad tensors or break-up programs into multiple static graphs.
We describe Tempo, a new DL system that combines the dynamism of eager execution with the whole-program optimizations of graph-based compilation. Tempo achieves this through a declarative programming model with recurrent tensors, which include explicit temporal dimensions. Temporal dimensions can be indexed using symbolic expressions to express dynamic dependencies on past and future tensors. Based on this, Tempo constructs a symbolic dependence graph, which concisely encodes dynamic dependencies between operators, and applies whole-program optimizations, such as algebraic simplifications, vectorization, tiling, and fusion. By tiling dynamic dependencies into static-size blocks, Tempo can also reuse existing static code-generators. It then uses a polyhedral model to find a feasible execution schedule, which includes memory management operations. We show that Tempo achieves a 7× speedup over JAX for Llama-3.2-3B decoding; for reinforcement learning algorithms, Tempo achieves a 54× speedup, with 16× lower peak memory usage.
We describe Tempo, a new DL system that combines the dynamism of eager execution with the whole-program optimizations of graph-based compilation. Tempo achieves this through a declarative programming model with recurrent tensors, which include explicit temporal dimensions. Temporal dimensions can be indexed using symbolic expressions to express dynamic dependencies on past and future tensors. Based on this, Tempo constructs a symbolic dependence graph, which concisely encodes dynamic dependencies between operators, and applies whole-program optimizations, such as algebraic simplifications, vectorization, tiling, and fusion. By tiling dynamic dependencies into static-size blocks, Tempo can also reuse existing static code-generators. It then uses a polyhedral model to find a feasible execution schedule, which includes memory management operations. We show that Tempo achieves a 7× speedup over JAX for Llama-3.2-3B decoding; for reinforcement learning algorithms, Tempo achieves a 54× speedup, with 16× lower peak memory usage.
Date Issued
2025-10-12
Date Acceptance
2025-10-01
Citation
Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, 2025, pp.572-588
ISBN
9798400718700
Publisher
ACM
Start Page
572
End Page
588
Journal / Book Title
Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles
Copyright Statement
Copyright © 2025 Copyright held by the owner/author(s). This work is licensed under Creative Commons Attribution International 4.0.
License URL
Identifier
10.1145/3731569.3764840
Source
SOSP '25: ACM SIGOPS 31st Symposium on Operating Systems Principles
Subjects
CCS Concepts: • Computing methodologies → Machine learning; Symbolic and algebraic manipulation Deep Learning
Compilation
Dynamic Tensors
Publication Status
Published
Start Date
2025-10-13
Finish Date
2025-10-16
Coverage Spatial
Seoul, Republic of Korea
Date Publish Online
2025-10-12