Stars do not form in isolation but together with other stars, and often in a clustered environment. Depending on the initial conditions in these environments, such as initial density and substructure, the distances of encounters between stars will differ. These encounters can also affect just-formed exoplanetary systems. Using N-body simulations, we show the effect of a single fly-by on a common type of exoplanetary system: close-in Super-Earths/sub-Neptunes with or without a distant Giant planet. Even a single encounter can significantly modify the architecture of these exoplanetary systems over their long lifetimes. We test fly-bys with different characteristics, such as distance and mass, and show how they perturb the inner planets long after the encounter, leading to collisions and mutual inclination excitation, which can significantly modify the observed architecture of these systems in transit. We find that our initially four-planet inner systems reduce to three or two inner planets depending on their initial separation and that the mutual inclinations of these remaining planets can be high enough to reduce the number of observable, transiting planets. In our 500 Myr simulations, we show that this reduction in the number of transiting planets due to stellar fly-bys can contribute to the observed excess of single-transit systems.