We consider the evaporation of a thin liquid layer which consists of a binary mixture of volatile liquids. The mixture is on top of a heated substrate and in contact with the gas phase that consists of the same vapor as the binary mixture. The effects of thermocapillarity, solutocapillarity, and the van der Waals interactions are considered. We derive the long-wave evolution equations for the free interface and the volume fraction that govern the two-dimensional stability of the layer subject to the above coupled mechanisms and perform a linear stability analysis. Our results demonstrate two modes of instabilities, a monotonic instability mode and an oscillatory instability mode. We supplement our results from stability analysis with transient simulations to examine the dynamics in the nonlinear regime and analyze how these instabilities evolve with time. More precisely we discuss how the effect of relative volatility along with the competition between thermal and solutal Marangoni effect define the mode of instability that develops during the evaporation of the liquid layer due to preferential evaporation of one of the components.