Physical activity has been associated with lower risks of breast and colorectal cancer in epidemiological studies; however, it is unknown if these associations are causal or confounded. In two-sample Mendelian randomisation analyses, using summary genetic data from the UK Biobank and GWA consortia, we found that a one standard deviation increment in average acceleration was associated with lower risks of breast cancer (odds ratio [OR]: 0.51, 95% confidence interval [CI]: 0.27 to 0.98, P-value = 0.04) and colorectal cancer (OR: 0.66, 95% CI: 0.48 to 0.90, P-value = 0.01). We found similar magnitude inverse associations for estrogen positive (ER+ve) breast cancer and for colon cancer. Our results support a potentially causal relationship between higher physical activity levels and lower risks of breast cancer and colorectal cancer. Based on these data, the promotion of physical activity is probably an effective strategy in the primary prevention of these commonly diagnosed cancers.