Visions of sustainable cities mostly conjure up well tended home and community gardens, where owners and residents plant fruits and vegetables that supply some of their livelihood needs. Indeed, home gardens can contribute to household food security but often fail to do so. Moreover, gardens can provide several additional ecosystem services and impact entire communities. This paper seeks to answer why these gardens often do not provide adequate services to make a substantial contribution to food security and identifies possible solutions. We undertook a case study in South Africa in a low-income former township area. The area is characterized by poverty, high levels of unemployment and food insecurity. We interviewed 140 respondents with home gardens to determine what role their own garden plays in household food security. Only 10% of households were found to be completely food secure. Of the rest, 39% experienced hunger that affected everyone in the household and 51% were at risk of hunger. Despite the fact that 72% of the respondents planted vegetables or fruits, the gardens did not contribute substantially to food security. The respondents mostly bought their food, with subsequent food shortages when they did not have enough money. The dietary diversity and consumption of vitamin A-rich fruits and vegetables were very low. The most important constraints inhibiting urban agriculture in the study area were cultural practices, such as the presence of large, bare, open spaces, or “lebala,” the focus of home gardeners on ornamental species and lawns; and a reliance on purchasing of foods.