Learning in unknown reward games: application to sensor networks
OA Location
Author(s)
Chapman, Archie C
Leslie, D
Rogers, Alex
Jennings, Nicholas R
Type
Journal Article
Abstract
This paper demonstrates a decentralised method for optimisation using game-theoretic multi-agent techniques, applied to a sensor network management problem. Our first major contribution is to show how the marginal contribution utility design is used to construct a unknown-reward potential game formulation of the problem. This formulation exploits the sparse structure of sensor network problems, and allows us to apply a bound to the price of anarchy of the Nash equilibria of the induced game. Furthermore, since the game is a potential game, solutions can be found using multiagent learning techniques. The techniques we derive use Q-learning to estimate an agent?s rewards, while an action adaptation process responds to an agent?s opponents? behaviour. However, there are many different algorithmic configurations that could be used to solve these games. Thus, our second major contribution is an extensive evaluation of several action adaptation processes. Specifically, we compare six algorithms across a variety of parameter settings to ascertain the quality of the solutions they produce, their speed of convergence, and their robustness to pre-specified parameter choices. Our results show that they each perform similarly across a wide range of parameters. There is, however, a significant effect from moving to a learning policy with sampling probabilities that go to zero too quickly for rewards to be accurately estimated.
Date Issued
2014
Citation
The Computer Journal, 2014, 57, pp.875-892
Start Page
875
End Page
892
Journal / Book Title
The Computer Journal
Volume
57
Identifier
http://eprints.soton.ac.uk/354633/
Subjects
Science & Technology
Technology
Computer Science, Hardware & Architecture
Computer Science, Information Systems
Computer Science, Software Engineering
Computer Science, Theory & Methods
Computer Science
COMPUTER SCIENCE, HARDWARE & ARCHITECTURE
COMPUTER SCIENCE, INFORMATION SYSTEMS
COMPUTER SCIENCE, SOFTWARE ENGINEERING
COMPUTER SCIENCE, THEORY & METHODS
potential games
learning in games
distributed optimization
POTENTIAL GAMES
FICTITIOUS PLAY
EQUILIBRIA
Computation Theory & Mathematics
08 Information And Computing Sciences
Article Number
6