Energy-based modelling for discrete and mixed data via heat equations on structured spaces
Author(s)
Schröder, Tobias
Ou, Zijing
Li, Yingzhen
Duncan, Andrew
Type
Conference Paper
Abstract
Energy-based models (EBMs) offer a flexible framework for probabilistic modelling across various data domains. However, training EBMs on data in discrete or mixed state spaces poses significant challenges due to the lack of robust and fast sampling methods. In this work, we propose to train discrete EBMs with Energy Discrepancy, a loss function which only requires the evaluation of the
energy function at data points and their perturbed counterparts, thus eliminating the need for Markov chain Monte Carlo. We introduce perturbations of the data
distribution by simulating a diffusion process on the discrete state space endowed with a graph structure. This allows us to inform the choice of perturbation from the
structure of the modelled discrete variable, while the continuous time parameter enables fine-grained control of the perturbation. Empirically, we demonstrate the
efficacy of the proposed approaches in a wide range of applications, including the estimation of discrete densities with non-binary vocabulary and binary image modelling. Finally, we train EBMs on tabular data sets with applications in synthetic data generation and calibrated classification.
energy function at data points and their perturbed counterparts, thus eliminating the need for Markov chain Monte Carlo. We introduce perturbations of the data
distribution by simulating a diffusion process on the discrete state space endowed with a graph structure. This allows us to inform the choice of perturbation from the
structure of the modelled discrete variable, while the continuous time parameter enables fine-grained control of the perturbation. Empirically, we demonstrate the
efficacy of the proposed approaches in a wide range of applications, including the estimation of discrete densities with non-binary vocabulary and binary image modelling. Finally, we train EBMs on tabular data sets with applications in synthetic data generation and calibrated classification.
Date Issued
2024-12-10
Date Acceptance
2024-12-01
Citation
Advances in Neural Information Processing Systems 37, 2024, pp.79246-79281
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Start Page
79246
End Page
79281
Journal / Book Title
Advances in Neural Information Processing Systems 37
Copyright Statement
© 2024 Neural Information Processing Systems Foundation, Inc. (NeurIPS).
Source
Advances in Neural Information Processing Systems 37
Publication Status
Published
Start Date
2024-12-10
Finish Date
2024-12-15
Coverage Spatial
Vancouver, Canada