Although physical activity is an established protective factor for cardiovascular diseases such as ischemic heart disease and stroke, less is known with regard to the association between specific domains of physical activity and heart failure, as well as the association between cardiorespiratory fitness and heart failure. We conducted a systematic review and meta-analysis of prospective observational studies to clarify the relations of total physical activity, domains of physical activity and cardiorespiratory fitness to risk of heart failure. PubMed and Embase databases were searched up to January 14th, 2020. Summary relative risks (RRs) were calculated using random effects models. Twenty-nine prospective studies (36 publications) were included in the review. The summary RRs for high versus low levels were 0.77 (95% CI 0.70–0.85, I2 = 49%, n = 7) for total physical activity, 0.74 (95% CI 0.68–0.81, I2 = 88.1%, n = 16) for leisure-time activity, 0.66 (95% CI 0.59–0.74, I2 = 0%, n = 2) for vigorous activity, 0.81 (95% CI 0.69–0.94, I2 = 86%, n = 3) for walking and bicycling combined, 0.90 (95% CI 0.86–0.95, I2 = 0%, n = 3) for occupational activity, and 0.31 (95% CI 0.19–0.49, I2 = 96%, n = 6) for cardiorespiratory fitness. In dose–response analyses, the summary RRs were 0.89 (95% CI 0.83–0.95, I2 = 67%, n = 4) per 20 MET-hours per day of total activity and 0.71 (95% CI 0.65–0.78, I2 = 85%, n = 11) per 20 MET-hours per week of leisure-time activity. Nonlinear associations were observed in both analyses with a flattening of the dose–response curve at 15–20 MET-hours/week for leisure-time activity. These findings suggest that high levels of total physical activity, leisure-time activity, vigorous activity, occupational activity, walking and bicycling combined and cardiorespiratory fitness are associated with reduced risk of developing heart failure.