Impact Statement:
Large datasets can be costly to obtain and annotate [LeCun et al. 2015]. This is limiting in many realistic situations, for example, when some objects are rarely encountered or when it is necessary to perform real-time operations [Ochal et al. 2020], [Guan et al. 2020], [Zhang et al. 2020], [Massiceti et al. 2021]. Few-shot learning (FSL) alleviates this burden by training a model to rapidly adapt with a limited amount of data. However, recent progress in the field has focused on the idealized scenario with balanced classes, which is easily compromised in the real world. To this end, we evaluate various few-shot and meta-learning methods across multiple class imbalance distributions and offer practical advice and best practices for dealing with these more realistic settings. We hope our work will continue and help narrow the gap between theoretical and real-world performance in FSL.
Abstract:
Few-shot learning (FSL) algorithms are commonly trained through meta-learning (ML), which exposes models to batches of tasks sampled from a meta-dataset to mimic tasks seen during evaluation. However, the standard training procedures overlook the real-world dynamics where classes commonly occur at different frequencies. While it is generally understood that class imbalance harms the performance of supervised methods, limited research examines the impact of imbalance on the FSL evaluation task. Our analysis compares ten state-of-the-art ML and FSL methods on different imbalance distributions and rebalancing techniques. Our results reveal that: 1) some FSL methods display a natural disposition against imbalance while most other approaches produce a performance drop by up to 17% compared to the balanced task without the appropriate mitigation; 2) many ML algorithms will not automatically learn to balance from exposure to imbalanced training tasks; 3) classical rebalancing strategies, such as random oversampling, can still be very effective, leading to state-of-the-art performances and should not be overlooked.