Discontinuous reception (DRX), wherein a user equipment (UE) temporarily disables its receiver, is a critical power saving feature in modern cellular systems. DRX is likely to be aggressively used at mmWave and sub-THz frequencies due to the high front-end power consumption. A key challenge for DRX at these frequencies is blockage-induced link outages: a UE will likely need to track many directional links to ensure reliable multi-connectivity, thereby increasing the power consumption. In this paper, we explore bandit algorithms for link tracking in connected mode DRX that reduce power consumption by tracking only a fraction of the available links, but without adversely affecting the outage and throughput performance. Through detailed, system level simulations at 28 GHz (5G) and 140 GHz (6G), we observe that even sub-optimal link tracking policies can achieve considerable power savings with relatively little degradation in outage and throughput performance, especially with digital beamforming at the UE. In particular, we show that it is feasible to reduce power consumption by 75% and still achieve up to 95% (80%) of the maximum throughput using digital beamforming at 28 GHz (140 GHz), subject to an outage probability of at most 1%.