Change point detection is a common technique to analyse time-indexed data and today's literature offers a wide variety of methods for change point detection. Assessing the performance of a change point detection method usually involves evaluating a method's ability to detect the true change point locations (if known). Additional potentially interesting properties, well-studied for other predictive algorithms, are left unstudied. Beyond conventional performance measures, we explore the consistency of estimated change point locations when a method is challenged with slightly perturbed input signals, a concept known as algorithmic stability in the machine learning literature. In particular, we conceptualise and formalise ideas relating to stability for change point detection, and present a simulation study that compares stabilities of popular change point detection methods including Binary Segmentation, Wild Binary Segmentation, and PELT (Pruned Exact Linear Time). We verify prior beliefs that Binary Segmentation's greedy search procedure is vulnerable to data perturbations, and introduce an extension to Binary Segmentation that stabilises its search procedure to some extent. We also find that PELT, as an exact method, is often more stable than non-optimal competitors, given the optimal parameter setting is known. Finally, we explore the potential for using stability as a means for setting control parameters in the absence of ground truth.