In today's world, a vast amount of data is being generated by edge devices
that can be used as valuable training data to improve the performance of
machine learning algorithms in terms of the achieved accuracy or to reduce the
compute requirements of the model. However, due to user data privacy concerns
as well as storage and communication bandwidth limitations, this data cannot be
moved from the device to the data centre for further improvement of the model
and subsequent deployment. As such there is a need for increased edge
intelligence, where the deployed models can be fine-tuned on the edge, leading
to improved accuracy and/or reducing the model's workload as well as its memory
and power footprint. In the case of Convolutional Neural Networks (CNNs), both
the weights of the network as well as its topology can be tuned to adapt to the
data that it processes. This paper provides a first step towards enabling CNN
finetuning on an edge device based on structured pruning. It explores the
performance gains and costs of doing so and presents an extensible open-source
framework that allows the deployment of such approaches on a wide range of
network architectures and devices. The results show that on average, data-aware
pruning with retraining can provide 10.2pp increased accuracy over a wide range
of subsets, networks and pruning levels with a maximum improvement of 42.0pp
over pruning and retraining in a manner agnostic to the data being processed by
the network.