Lines of thought in Large Language Models
File(s)
Accepted version
Author(s)
Sarfati, Raphael
Liu, Toni
Boulle, Nicolas
Earls, Christopher
Type
Conference Paper
Abstract
Large Language Models achieve next-token prediction by transporting a vectorized piece of text (prompt) across an accompanying embedding space under the action of successive transformer layers. The resulting high-dimensional trajectories realize different contextualization, or ‘thinking’, steps, and fully determine the output probability distribution. We aim to characterize the statistical properties
of ensembles of these ‘lines of thought.’ We observe that independent trajectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters extracted from data. We find it remarkable that the vast complexity of such large models can be reduced to a much simpler form, and we reflect on implications.
of ensembles of these ‘lines of thought.’ We observe that independent trajectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters extracted from data. We find it remarkable that the vast complexity of such large models can be reduced to a much simpler form, and we reflect on implications.
Date Acceptance
2025-01-22
Publisher
Curran Associates, Inc.
Copyright Statement
Subject to copyright. This paper is embargoed until publication.
Identifier
https://openreview.net/forum?id=zjAEa4s3sH
Source
International Conference on Learning Representations (ICLR 2025)
Publication Status
Accepted
Start Date
2025-04-24
Finish Date
2025-04-28
Coverage Spatial
Singapore