Use of testosterone replacement therapy (TRT) and anabolic-androgenic steroids (AAS) has increased over the last 20 years, coinciding with an increase in men presenting with infertility and hypogonadism. Both agents have a detrimental effect on spermatogenesis and pose a clinical challenge in the setting of hypogonadism and infertility. Adding to this challenge is the paucity of data describing recovery of spermatogenesis on stopping such agents. The unwanted systemic side effects of these agents have driven the development of novel agents such as selective androgen receptor modulators (SARMs). Data showing natural recovery of spermatogenesis following cessation of TRT are limited to observational studies. Largely, these have shown spontaneous recovery of spermatogenesis after cessation. Contemporary literature suggests the time frame for this recovery is highly variable and dependent on several factors including baseline testicular function, duration of drug use and age at cessation. In some men, drug cessation alone may not achieve spontaneous recovery, necessitating hormonal stimulation with selective oestrogen receptor modulators (SERMs)/gonadotropin therapy or even the need for assisted reproductive techniques. However, there are limited prospective randomized data on the role of hormonal stimulation in this clinical setting. The use of hormonal stimulation with agents such as gonadotropins, SERMs, aromatase inhibitors and assisted reproductive techniques should form part of the counselling process in this cohort of hypogonadal infertile men. Moreover, counselling men regarding the detrimental effects of TRT/AAS on fertility is very important, as is the need for robust randomized studies assessing the long-term effects of novel agents such as SARMs and the true efficacy of gonadotropins in promoting recovery of spermatogenesis.