Whispering gallery mode (WGM) resonators are an important platform for linear, nonlinear and quantum optical experiments. In such experiments, independent control of in- and out-coupling rates to different modes can lead to higher conversion efficiencies and greater flexibility in the generation of non-classical states based on parametric down conversion. In this work, we introduce a scheme that enables selective out-coupling of WGMs belonging to a specific polarization family, while the orthogonally polarized modes remain largely unperturbed. Our technique utilizes material birefringence in both the resonator and the coupler such that a negative (positive) birefringence allows for polarization-selective coupling to TE (TM) WGMs. We formulate a new coupling condition suitable for describing the case where the refractive indices of the resonator and the coupler are almost the same, from which we derive a criterion for polarization-selective coupling. Finally, we experimentally demonstrate our proposed method using a lithium niobate disk resonator coupled to a lithium niobate prism, where we show a \SI{22}{dB} suppression of coupling to TM modes relative to TE modes.