Large (vision) language models for autonomous vehicles: current trends and future directions
File(s)
Author(s)
Type
Journal Article
Abstract
As autonomous vehicles (AVs) advance, the integration of Large (Vision) Language Models (LLMs and VLMs) has emerged as a promising approach to enhance AV capabilities in perception, planning, decision-making, and data generation. However, the practical challenges of incorporating LLMs and VLMs into AV systems, including computational efficiency, real-time processing, and ethical considerations, remain underexplored. This survey aims to provide a comprehensive review of the current research on LLM and VLM applications in AVs, focusing on the following key areas: modular integration, end-to-end integration, data generation, evaluation platforms, datasets, and benchmarks. We systematically analyse 77 recent papers published before Sep 2025, detailing their methodologies and models. Our findings highlight the potential of LLMs and VLMs to improve AV system performance while acknowledging limitations. This survey offers researchers and practitioners a panoptic view of the classification and progression of LLMs and VLMs in the AV sphere, while systematically distilling models to their core components. We envision this survey as a central reference for AV researchers navigating this rapidly evolving landscape to accelerate future research.
Date Issued
2026-01-01
Date Acceptance
2025-11-01
Citation
IEEE Transactions on Intelligent Transportation Systems, 2026, 27 (1), pp.187-210
ISSN
1524-9050
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Start Page
187
End Page
210
Journal / Book Title
IEEE Transactions on Intelligent Transportation Systems
Volume
27
Issue
1
Copyright Statement
© 2025 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
License URL
Publication Status
Published
Date Publish Online
2025-11-21