Conserved lipid transfer proteins of the Ups/PRELI family regulate lipid accumulation in mitochondria by shuttling phospholipids in a lipid-specific manner across the intermembrane space. Here, we combine structural analysis, unbiased genetic approaches in yeast and molecular dynamics simulations to unravel determinants of lipid specificity within the conserved Ups/PRELI family. We present structures of human PRELID1–TRIAP1 and PRELID3b–TRIAP1 complexes, which exert lipid transfer activity for phosphatidic acid and phosphatidylserine, respectively. Reverse yeast genetic screens identify critical amino acid exchanges that broaden and swap their lipid specificities. We find that amino acids involved in head group recognition and the hydrophobicity of flexible loops regulate lipid entry into the binding cavity. Molecular dynamics simulations reveal different membrane orientations of PRELID1 and PRELID3b during the stepwise release of lipids. Our experiments thus define the structural determinants of lipid specificity and the dynamics of lipid interactions by Ups/PRELI proteins.