Bacteria diversify into genetic clusters analogous to those observed in sexual eukaryotes, but the definition of bacterial species is an ongoing problem. Recent work has focused on adaptation to distinct ecological niches as the main driver of clustering, but there remains debate about the role of recombination in that process. One view is that homologous recombination occurs too rarely for gene flow to constrain divergent selection. Another view is that homologous recombination is frequent enough in many bacterial populations that barriers to gene flow are needed to permit divergence. Niche-specific gene pools have been proposed as a general mechanism to limit gene flow. We use theoretical models to evaluate additional hypotheses that evolving genetic architecture, specifically the effect sizes of genes and gene gain and loss, can limit gene flow between diverging populations. Our model predicts that i) in the presence of gene flow and recombination, ecological divergence is concentrated in few loci of large effect, and ii) high rates of gene flow plus recombination promote gene loss and favor the evolution of niche-specific genes. The results show that changing genetic architecture and gene loss can facilitate ecological divergence, even without niche-specific gene pools. We discuss these results in the context of recent studies of sympatric divergence in microbes.