The recently introduced correntropy function is an interesting and useful similarity measure between two random variables which has found myriad applications in signal processing. A series expansion for correntropy in terms of higher-order moments of the difference between the two random variables has been used to try to explain its statistical properties for uses such as deconvolution. We examine the existence and form of this expansion, showing that it may be divergent, e.g., when the difference has the Laplace distribution, and give sufficient conditions for its existence for differently characterized sub-Gaussian distributions. The contribution of the higher-order moments can be quite surprising, depending on the size of the Gaussian kernel in the definition of the correntropy. In the blind deconvolution setting we demonstrate that statistical exchangeability explains the existence of sub-optimal minima in the correntropy cost surface and show how the positions of these minima are controlled by the size of the Gaussian kernel.