Agricultural Land-Use Change (ALUC) is a major driver of global environmental change, not least via its direct impact on the sustainability and resilience of the rural economy. Its drivers are complex and have remained contentious, necessitating further empirical study. This study aims to derive context-specific evidence on the driving factors and effects of ALUC from different stakeholders’ perceptions. We carried out household surveys and participatory rural appraisal across Benue State, Nigeria. ALUC has economic, social, ecological, and institutional implications for farmers and on agricultural productivity. Farmers perceived that the main factors driving ALUC were land conflict, government land-use policies and infrastructural development. Stakeholders’ perceptions revealed that although the factors driving ALUC are diverse in nature, they are somewhat embedded within the broader issue of land-use conflict, which has led to cropland abandonment, clearing of forest vegetation, soil degradation, changes from large scale to subsistence farming, and farmers’ eventual loss of interest in agriculture. This suggest that the drivers and implications of ALUC go beyond simple changes to the extent of land used for agriculture, but also incorporates other regional socio-ecological changes. Our study highlights the importance of stakeholders’ perceptions in understanding complex socio-ecological issues if we are to provide clear direction into areas where policy interventions are most needed.