Energy discrepancies: a score-independent loss for energy-based models
File(s)
Accepted version
OA Location
Author(s)
Type
Conference Paper
Abstract
Energy-based models are a simple yet powerful class of probabilistic models, but their widespread adoption has been limited by the computational burden of training them. We propose a novel loss function called Energy Discrepancy (ED) which does not rely on the computation of scores or expensive Markov chain Monte Carlo. We show that energy discrepancy approaches the explicit score matching and negative log-likelihood loss under different limits, effectively interpolating between both. Consequently, minimum energy discrepancy estimation overcomes the problem of nearsightedness encountered in score-based estimation methods, while also enjoying theoretical guarantees. Through numerical experiments, we demonstrate that ED learns low-dimensional data distributions faster and more accurately than explicit score matching or contrastive divergence. For high-dimensional image data, we describe how the manifold hypothesis puts limitations on our approach and demonstrate the effectiveness of energy discrepancy by training the energy-based model as a prior of a variational decoder model.
Editor(s)
Oh, A
Neumann, T
Globerson, A
Saenko, K
Hardt, M
Levine, S
Date Issued
2023-12-10
Date Acceptance
2023-12-01
Citation
Advances in neural information processing systems, 2023, https://proceedings.neurips.cc/paper_files/paper/2023/file/8e176ef071f00f1b233461c5ad5e1b24-Paper-Conference.pdf, pp.45300-45338
ISSN
1049-5258
Publisher
Curran Associates, Inc.
Start Page
45300
End Page
45338
Journal / Book Title
Advances in neural information processing systems
Volume
https://proceedings.neurips.cc/paper_files/paper/2023/file/8e176ef071f00f1b233461c5ad5e1b24-Paper-Conference.pdf
Copyright Statement
© 2023 Neural Information Processing Systems Foundation, Inc. (NeurIPS).
Source
37th Conference on Neural Information Processing Systems (NeurIPS)
Subjects
Computer Science
Computer Science, Artificial Intelligence
Computer Science, Information Systems
Science & Technology
Technology
Publication Status
Published
Start Date
2023-12-10
Finish Date
2023-12-16
Coverage Spatial
New Orleans, LA, USA