Overfitting has been widely studied in the context of classification and regression. In this paper, we study the overfitting in the context of dimensionality reduction. We show that the conventional wisdom of improving classification performance by maximising inter-class discrimination is not valid for high-dimensional datasets, and can lead to severe overfitting. In particular, we prove the theoretical existence of perfectly discriminative subspace projections, and show that for datasets with very high input dimensionality, inter-class discrimination should be reduced rather than maximised. This naturally leads to a simple dimensionality reduction technique, which we call Soft Discriminant Maps, which we use to show a direct relationship between the classification performance and the level of inter-class discrimination of feature extractors. Moreover, Soft Discriminant Maps consistently exhibit better classification performance than other comparable techniques.