Bayesian optimization (BO) is a powerful approach for seeking the global
optimum of expensive black-box functions and has proven successful for fine
tuning hyper-parameters of machine learning models. However, in practice, BO is
typically limited to optimizing 10--20 parameters. To scale BO to high
dimensions, we normally make structural assumptions on the decomposition of the
objective and/or exploit the intrinsic lower dimensionality of the problem,
e.g. by using linear projections. The limitation of aforementioned approaches
is the assumption of a linear subspace. We could achieve a higher compression
rate with nonlinear projections, but learning these nonlinear embeddings
typically requires much data. This contradicts the BO objective of a relatively
small evaluation budget. To address this challenge, we propose to learn a
low-dimensional feature space jointly with (a) the response surface and (b) a
reconstruction mapping. Our approach allows for optimization of the acquisition
function in the lower-dimensional subspace. We reconstruct the original
parameter space from the lower-dimensional subspace for evaluating the
black-box function. For meaningful exploration, we solve a constrained
optimization problem.