Software

We have made the following software available to download under various open-source licenses:

• The Pinball Domain
• The Fourier Basis
• Motion Planning Benchmarks

The Pinball Domain (GPL)

The Pinball domain is a fairly challenging 4-dimensional continuous and dynamic reinforcement learning domain. The goal is to maneuver the blue ball into the red hole, while avoiding (or using, since the ball is dynamic and collisions are elastic) the obstacles. The dynamics of the ball and the presence of obstacles result in a domain with sharp discontinuities, and the location and shape of the obstacles can be specified so you can make the domain as hard or as easy as you want.

The Pinball domain page contains a brief overview and Java source code, full documentation, an RL-Glue interface, and GUI programs for editing obstacle configurations, viewing saved trajectories, etc.

Other versions:

• Pierre-Luc Bacon has ported Pinball to Python.
• A Pinball implementation is included in RLPy.

The Pinball domain was originally described in:

• G.D. Konidaris and A.G. Barto. Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining. Advances in Neural Information Processing Systems 22, pages 1015-1023, December 2009.

The Fourier Basis

The Fourier basis is a simple, principled basis function scheme for linear value function approximation in reinforcement learning. It has performed well over a wide range of problems, despite (or perhaps because of) its simplicity. Please see our Fourier Basis page for a concise overview.

We contributed RL-Glue-compatible ε-greedy Sarsa(λ) Fourier Basis code to the RL-Library project. You can get the code, installation instructions, and a project overview at the RL Library community wiki. The code is in Java and is fully documented. (Apache License)

We have also made available a (GPLv3 licensed) Python implementation of the Fourier basis, along with code for ε-greedy Sarsa(λ) and two RLGym examples, here: github.com/brownirl/fourier_vfa.

Other versions:

• A Java implementation is included in BURLAP by James MacGlashan.
• A Python implementation is included in RLPy.

The Fourier Basis is described in:

• G.D. Konidaris, S. Osentoski and P.S. Thomas. Value Function Approximation in Reinforcement Learning using the Fourier Basis. In Proceedings of the Twenty-Fifth Conference on Artificial Intelligence, pages 380-385, August 2011.