For any given network of detectors, and for any given integration time, even in the idealized limit of negligible instrumental noise, the intrinsic time variation of the isotropic component of the stochastic gravitational wave background (SGWB) induces a limit on how accurately the anisotropies in the SGWB can be measured. We show here how this sample limit can be calculated and apply this to three separate configurations of ground-based detectors placed at existing and planned sites. Our results show that in the idealized, best-case scenario, individual multipoles of the anisotropies at ℓ≤8 can only be measured to ∼10^{-5}-10^{-4} level over five years of observation as a fraction of the isotropic component. As the sensitivity improves as the square root of the observation time, this poses a very serious challenge for measuring the anisotropies of SGWB of cosmological origin, even in the case of idealized detectors with arbitrarily low instrumental noise.