Ancient layers of impact spherules provide a record of Earth's early bombardment history. Here, we compare different bombardment histories to the spherule layer record and show that 3.2-3.5 Ga the flux of large impactors (10-100 km in diameter) was likely 20-40 times higher than today. The E-belt model of early Solar System dynamics suggests that an increased impactor flux during the Archean is the result of the destabilization of an inward extension of the main asteroid belt (Bottke, W.F., Vokrouhlický, D., Minton, D., Nesvorný, D., Morbidelli, A., Brasser, R., Simonson, B., Levison, H.F., 2012. Nature 485, 78–81). Here, we find that the nominal flux predicted by the E-belt model is 7-19 times too low to explain the spherule layer record. Moreover, rather than making most lunar basins younger than 4.1 Gyr old, the nominal E-belt model, coupled with a corrected crater diameter scaling law, only produces two lunar basins larger than 300 km in diameter. We also show that the spherule layer record when coupled with the lunar cratering record and careful consideration of crater scaling laws can constrain the size distribution of ancient terrestrial impactors. The preferred population is main-belt-like up to ∼50 km in diameter transitioning to a steep distribution going to larger sizes.