Robust machine learning methods for high-dimensional datasets with applications in genomics and finance
File(s)
Author(s)
Wong, Ming-Hei
Type
Thesis
Abstract
This thesis studies different robust machine-learning methods towards high-dimensional datasets under
concept drifts, using examples from single-cell RNA sequencing and stock trading in quantitative finance.
The diversity of next-generation sequencing methods leads to challenges in integrating datasets
prepared by different laboratories. A new batch correction and data integration method based on
probabilistic matrix factorisations, Integrative Hierarchical Poisson Factorisation (IHPF), is introduced
in this thesis. IHPF performs robustly under different noise levels in datasets and provides
interpretable latent factors that capture biological signals. The cell and gene
factorisation scores obtained are better than those of its predecessor, Hierarchical Poisson Factorisation (HPF).
Financial datasets are highly non-stationary with a low signal-to-noise ratio, often with regime-
dependent behaviours. We developed a new stagewise incremental learning pipeline to accommodate
distribution shifts in data by iteratively combining models trained with different parameter settings.
Various approaches of model stacking, including both rule-based methods such as dynamic best model
selection and dynamic hedging are used to adapt to different concept drift types.
I created a diverse set of XGBoost models that can model the level of disagreement between investors
in the stock market using Jackknife sampling. Hedging portfolios can be created based on the variance of
predictions from different models, which offers hedging benefits under unfavourable market conditions. A
dynamically hedged model can be formulated, which achieves a similar level of return as the baseline
model provided by Numerai, with significantly lower drawdowns.
I also created a multi-stage XGBoost model with skip connections which outperforms the baseline
models provided by Numerai. The predictions created are also orthogonal to the baseline predictions.
concept drifts, using examples from single-cell RNA sequencing and stock trading in quantitative finance.
The diversity of next-generation sequencing methods leads to challenges in integrating datasets
prepared by different laboratories. A new batch correction and data integration method based on
probabilistic matrix factorisations, Integrative Hierarchical Poisson Factorisation (IHPF), is introduced
in this thesis. IHPF performs robustly under different noise levels in datasets and provides
interpretable latent factors that capture biological signals. The cell and gene
factorisation scores obtained are better than those of its predecessor, Hierarchical Poisson Factorisation (HPF).
Financial datasets are highly non-stationary with a low signal-to-noise ratio, often with regime-
dependent behaviours. We developed a new stagewise incremental learning pipeline to accommodate
distribution shifts in data by iteratively combining models trained with different parameter settings.
Various approaches of model stacking, including both rule-based methods such as dynamic best model
selection and dynamic hedging are used to adapt to different concept drift types.
I created a diverse set of XGBoost models that can model the level of disagreement between investors
in the stock market using Jackknife sampling. Hedging portfolios can be created based on the variance of
predictions from different models, which offers hedging benefits under unfavourable market conditions. A
dynamically hedged model can be formulated, which achieves a similar level of return as the baseline
model provided by Numerai, with significantly lower drawdowns.
I also created a multi-stage XGBoost model with skip connections which outperforms the baseline
models provided by Numerai. The predictions created are also orthogonal to the baseline predictions.
Version
Open Access
Date Issued
2023-09-28
Date Awarded
01/10/2024
License URL
Advisor
Barahona, Mauricio
Publisher Department
Mathematics
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)