LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law
File(s)
Accepted version
Author(s)
Liu, Toni JB
Boulle, Nicolas
Sarfati, Raphael
Earls, Christopher J
Type
Conference Paper
Abstract
We study LLMs’ ability to extrapolate the be-
havior of various dynamical systems, including
stochastic, chaotic, continuous, and discrete
systems, whose evolution is governed by prin-
ciples of physical interest. Our results show
that LLaMA-2, a language model trained on
text, achieves accurate predictions of dynam-
ical system time series without fine-tuning or
prompt engineering. Moreover, the accuracy
of the learned physical rules increases with the
length of the input context window, revealing
an in-context version of a neural scaling law.
Along the way, we present a flexible and effi-
cient algorithm for extracting probability den-
sity functions of multi-digit numbers directly
from LLMs.
havior of various dynamical systems, including
stochastic, chaotic, continuous, and discrete
systems, whose evolution is governed by prin-
ciples of physical interest. Our results show
that LLaMA-2, a language model trained on
text, achieves accurate predictions of dynam-
ical system time series without fine-tuning or
prompt engineering. Moreover, the accuracy
of the learned physical rules increases with the
length of the input context window, revealing
an in-context version of a neural scaling law.
Along the way, we present a flexible and effi-
cient algorithm for extracting probability den-
sity functions of multi-digit numbers directly
from LLMs.
Date Acceptance
2024-09-19
Citation
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Journal / Book Title
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Copyright Statement
Subject to copyright. This paper is embargoed until publication. Once published the Version of Record (VoR) will be available on immediate open access.
Source
Empirical Methods in Natural Language Processing
Publication Status
Accepted
Start Date
2024-11-12
Finish Date
2024-11-16
Coverage Spatial
Miami, Florida
Rights Embargo Date
10000-01-01