Background: The microvasculature, the smallest blood vessels in the body, has key roles in maintenance of organ health as well as tumorigenesis. The retinal fundus is a window for human in vivo non-invasive assessment of the microvasculature. Large-scale complementary machine learning-based assessment of the retinal vasculature with phenome-wide and genome-wide analyses may yield new insights into human health and disease. Methods: We utilized 97,895 retinal fundus images from 54,813 UK Biobank participants. Using convolutional neural networks to segment the retinal microvasculature, we calculated fractal dimension (FD) as a measure of vascular branching complexity, and vascular density. We associated these indices with 1,866 incident ICD-based conditions (median 10y follow-up) and 88 quantitative traits, adjusting for age, sex, smoking status, and ethnicity. Results: Low retinal vascular FD and density were significantly associated with higher risks for incident mortality, hypertension, congestive heart failure, renal failure, type 2 diabetes, sleep apnea, anemia, and multiple ocular conditions, as well as corresponding quantitative traits. Genome-wide association of vascular FD and density identified 7 and 13 novel loci respectively, which were enriched for pathways linked to angiogenesis (e.g., VEGF, PDGFR, angiopoietin, and WNT signaling pathways) and inflammation (e.g., interleukin, cytokine signaling). Conclusions: Our results indicate that the retinal vasculature may serve as a biomarker for future cardiometabolic and ocular disease and provide insights on genes and biological pathways influencing microvascular indices. Moreover, such a framework highlights how deep learning of images can quantify an interpretable phenotype for integration with electronic health records, biomarker, and genetic data to inform risk prediction and risk modification.