Data-space validation of high-dimensional models by comparing sample quantiles
File(s)
Published version
Author(s)
Type
Journal Article
Abstract
We present a simple method for assessing the predictive performance of high-dimensional models directly in data space when only samples are available. Our approach is to compare the quantiles of observables predicted by a model to those of the observables themselves. In cases where the dimensionality of the observables is large (e.g., multiband galaxy photometry), we advocate that the comparison is made after projection onto a set of principal axes to reduce the dimensionality. We demonstrate our method on a series of two-dimensional examples. We then apply it to results from a state-of-the-art generative model for galaxy photometry (pop-cosmos) that generates predictions of colors and magnitudes by forward simulating from a 16-dimensional distribution of physical parameters represented by a score-based diffusion model. We validate the predictive performance of this model directly in a space of nine broadband colors. Although motivated by this specific example, we expect that the techniques we present will be broadly useful for evaluating the performance of flexible, nonparametric population models of this kind, and other settings where two sets of samples are to be compared.
Date Issued
2025-01-01
Date Acceptance
2024-10-28
Citation
Astrophysical Journal Supplement Series, 2025, 276 (1)
ISSN
0067-0049
Publisher
IOP Publishing
Journal / Book Title
Astrophysical Journal Supplement Series
Volume
276
Issue
1
Copyright Statement
© 2024. The Author(s). Published by the American Astronomical Society. Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
License URL
Subjects
ASYMPTOTIC THEORY
DATA RELEASE
GEOMETRIC REPRESENTATION
MAXIMUM-LIKELIHOOD
OUT CROSS-VALIDATION
PCA CONSISTENCY
PRINCIPAL COMPONENT ANALYSIS
P-VALUES
SIZE DATA
STELLAR POPULATION SYNTHESIS
Publication Status
Published
Article Number
5
Date Publish Online
2024-12-12