Obesity has been consistently associated with a greater colorectal cancer risk, but this relationship is weaker among women. In the UK Biobank, we investigated the associations between body size (body mass index [BMI], height, waist circumference, and waist-to-hip ratio) and body fat composition (total body fat percentage and trunk fat percentage) measurements with colorectal cancer risk among 472,526 men and women followed for 5.6 years on average. Multivariable hazard ratios (HRs) and 95% confidence intervals (95%CI) for developing colorectal cancer (2,636 incident cases) were estimated using Cox proportional hazards models. Among men, when the highest and lowest fifths were compared, BMI (HR = 1.35, 95%CI: 1.13–1.61; Ptrend < 0.0001), waist circumference (HR = 1.66, 95%CI: 1.39–1.99; Ptrend < 0.0001), waist-to-hip ratio (HR = 1.58, 95%CI: 1.31–1.91; Ptrend < 0.0001), total body fat percentage (HR = 1.27, 95%CI: 1.06–1.53; Ptrend = 0.002), and trunk fat percentage (HR = 1.31, 95%CI: 1.09–1.58; Ptrend = 0.002) were associated with greater colorectal cancer risk. For women, only waist-to-hip ratio (HR for highest versus lowest fifth = 1.33, 95%CI: 1.08–1.65; Ptrend = 0.005) was positively associated with colorectal cancer risk. Greater body size (overall and abdominal adiposity) was positively associated with colorectal cancer development in men. For women, abdominal adiposity, rather than overall body size, was associated with a greater colorectal cancer risk.