Dilute nitride InGaAsSbN layers grown by low-temperature liquid phase epitaxy are studied in comparison with quaternary InGaAsN layers grown at the same growth conditions to understand the effect of Sb in the alloy. The lattice mismatch to the GaAs substrate is found to be slightly larger for the InGaAsSbN layers, which is explained by the large atomic radius of Sb. A reduction of the band gap energy with respect to InGaAsN is demonstrated by means of photoluminescence (PL), surface photovoltage (SPV) spectroscopy and tight-binding calculations. The band-gap energies determined from PL and ellipsometry measurements are in good agreement, while the SPV spectroscopy and the tight-binding calculations provide lower values. Possible reasons for these discrepancies are discussed. The PL spectra reveal localized electronic states in the band gap near the conduction band edge, which is confirmed by SPV spectroscopy. The analysis of the power dependence of the integrated PL has allowed determining the dominant radiative recombination mechanisms in the layers. The values of the refraction index in a wide spectral region are found to be higher for the Sb containing layers.