We consider the problem of imitation learning from a finite set of expert
trajectories, without access to reinforcement signals. The classical approach
of extracting the expert's reward function via inverse reinforcement learning,
followed by reinforcement learning is indirect and may be computationally
expensive. Recent generative adversarial methods based on matching the policy
distribution between the expert and the agent could be unstable during
training. We propose a new framework for imitation learning by estimating the
support of the expert policy to compute a fixed reward function, which allows
us to re-frame imitation learning within the standard reinforcement learning
setting. We demonstrate the efficacy of our reward function on both discrete
and continuous domains, achieving comparable or better performance than the
state of the art under different reinforcement learning algorithms.