This thesis explores the integration of tree ensembles with global optimization methods for addressing complex black-box optimization problems across various real-world applications, including chemical engineering, energy systems, and hyperparameter optimization.

ENTMOOT, a novel software framework, integrates tree models effectively into decision-making and black-box optimization tasks.
The framework introduces a reliable uncertainty measure compatible with tree models, and proposes and evaluates strategies for tackling large-scale optimization problems, thereby addressing some common limitations of tree-based models in Bayesian optimization.
Further, it ensures globally optimal solutions of acquisition functions to facilitate ideal trade-offs when proposing new black-box experiments.

Moreover, we extend ENTMOOT for constrained multi-objective optimization of black-box problems and apply it to relevant problems related to energy systems.
This strategy proves preferable compared to evolutionary-based algorithms deployed in this field, especially when dealing with heterogeneous variable spaces and complex or unknown system dynamics that involve input constraints.

Continuing on this trajectory, we further develop the research to confront challenges inherent to the application of tree ensembles in black-box optimization tasks, such as algorithm tuning and neural architecture search.
By leveraging the kernel interpretation of tree ensembles as a Gaussian Process prior, we obtain model variance estimates and develop a compatible optimization formulation for the piece-wise constant acquisition function compatible with deterministic global solvers.
Importantly, this approach allows for the seamless integration of known constraints, thereby improving sampling efficiency across scenarios involving mixed-variable feature spaces and known input constraints.

The presented research offers comprehensive insights and innovative strategies to tap into the strengths of tree ensembles while mitigating some of their limitations, ultimately advocating their increased applicability across a spectrum of real-world optimization scenarios.