A reinforcement-learning approach to proactive caching in wireless networks.
File(s)
Accepted version
Author(s)
Somuyiwa, Samuel
Gunduz, D
Gyorgy, Andras
Type
Journal Article
Abstract
We consider a mobile user accessing contents in a dynamic environment, where new contents are generated over time (by the user’s contacts), and remain relevant to the user for random lifetimes. The user, equipped with a finite-capacity cache memory, randomly accesses the system, and requests all the relevant contents at the time of access. The system incurs an energy cost associated with the number of contents downloaded and the channel quality at that time. Assuming causal knowledge of the channel quality, the content profile, and the user-access behavior, we model the proactive caching problem as a Markov decision process with the goal of minimizing the long-term average energy cost. We first prove the optimality of a threshold-based proactive caching scheme, which dynamically caches or removes appropriate contents from the memory, prior to being requested by the user, depending on the channel state. The optimal threshold values depend on the system state, and hence, are computationally intractable. Therefore, we propose parametric representations for the threshold values, and use reinforcement-learning algorithms to find near-optimal parametrizations. We demonstrate through simulations that the proposed schemes significantly outperform classical reactive downloading, and perform very close to a genieaided lower bound.
Index Terms—Markov decision process, proactive content caching, policy gradient methods, reinforcement learning.
Index Terms—Markov decision process, proactive content caching, policy gradient methods, reinforcement learning.
Date Issued
2018-06-07
Date Acceptance
2018-04-18
Citation
IEEE Journal on Selected Areas in Communications, 2018, 36 (6), pp.1331-1344
ISSN
0733-8716
Publisher
Institute of Electrical and Electronics Engineers
Start Page
1331
End Page
1344
Journal / Book Title
IEEE Journal on Selected Areas in Communications
Volume
36
Issue
6
Copyright Statement
© 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Sponsor
Commission of the European Communities
Identifier
https://ieeexplore.ieee.org/document/8374824
Grant Number
677854
Subjects
Science & Technology
Technology
Engineering, Electrical & Electronic
Telecommunications
Engineering
Markov decision process
proactive content caching
policy gradient methods
reinforcement learning
Networking & Telecommunications
0906 Electrical and Electronic Engineering
1005 Communications Technologies
0805 Distributed Computing
Publication Status
Published
Date Publish Online
2018-06-07