Membrane fouling generated by small molecular-weight aromatic compounds with poor biodegradability is a major barrier to advanced petrochemical wastewater treatment using nanofiltration (NF) technology. In this study, the fouling behavior of ten BTEX with different substituent existing in petrochemical wastewater on the NF membrane was systematically investigated. By examining the effect of the number, position, and type of substituents on the permeability of NF membranes and membrane resistance analysis, combined with XDLVO theory and correlation analysis, we found that stronger dipole-dipole interactions of BTEX with higher polarity and hydrogen bonding effects between substituents and the membrane surface were verified to be the main forces driving the attachment to the surface of membranes. Furthermore, by analyzing the effect of common inorganic ions in petrochemical wastewater on membrane fouling, it was found that electron-donating substituents (-CH3, -C2H5, and -NH2) enhanced the electron cloud density of the benzene ring, a process that exacerbated membrane fouling by strengthening electrostatic interactions between the benzene ring and Ca2+ ions. The fouling potential of electron-withdrawing substituted (-NO2, -OH) BTEX exhibited the opposite trend. Overall, this study provides a theoretical basis for developing effective membrane fouling control strategies in NF advanced treatment of petrochemical wastewater.