Offline reinforcement learning (RL) algorithms are
applied to learn performant, well-generalizing poli-
cies when provided with a static dataset of inter-
actions. Many recent approaches to offline RL
have seen substantial success, but with one key
caveat: they demand substantial per-dataset hyper-
parameter tuning to achieve reported performance,
which requires policy rollouts in the environment
to evaluate; this can rapidly become cumbersome.
Furthermore, substantial tuning requirements can
hamper the adoption of these algorithms in prac-
tical domains. In this paper, we present TD3 with
Behavioral Supervisor Tuning (TD3-BST), an al-
gorithm that trains an uncertainty model and uses
it to guide the policy to select actions within the
dataset support. TD3-BST can learn more effec-
tive policies from offline datasets compared to pre-
vious methods and achieves the best performance
across challenging benchmarks without requiring
per-dataset tuning.