Associations of total testosterone (T) and calculated free T with cardiovascular disease (CVD) remain poorly understood. Particularly how these associations vary according to race and ethnicity in a nationally representative sample of men. Data included 7058 men (≥20 years) from NHANES. CVD was defined as any reported diagnosis of heart failure (HF), coronary artery disease (CAD), myocardial infarction (MI), and stroke. Total T (ng/mL) was obtained among males who participated in the morning examination. Weighted multivariable-adjusted logistic regression models were conducted. We found associations of low T (OR = 1.57, 95% CI = 1.17–2.11), low calculated free T (OR = 1.53, 95% CI = 1.10–2.17), total T (Q1 vs Q5), and calculated free T (Q1 vs Q5) with CVD after adjusting for estradiol and SHBG. In disease specific analysis, low T increased prevalence of MI (OR = 1.72, 95% CI = 1.08–2.75) and HF (OR = 1.74, 95% CI = 1.08–2.82), but a continuous increment of total T reduced the prevalence of CAD. Similar inverse associations were identified among White and Mexican Americans, but not Blacks (OR = 0.93, 95% CI = 0.49–1.76). Low levels of T and calculated free T were associated with an increased prevalence of overall CVD and among White and Mexican Americans. Associations remained in the same direction with specific CVD outcomes in the overall population.