![]() ![]() ![]() |
Intelligent Robot Lab
Brown University, Providence RIHome | People | Research | Publications | Joining | Software |
|||||||||||||||||
The Fourier BasisThe Fourier basis is a simple, principled basis function scheme for linear value function approximation in reinforcement learning. It has performed well over a wide range of problems, despite its simplicity, and I now use it almost exclusively. My advice to fellow researchers who require function approximation is that the Fourier basis should be the first thing you try. If a problem is small enough, the Fourier basis avoids the need for extensive problem-specific feature engineering. For example, we have achieved good results on the Acrobot problem using a Fourier basis of order between 5 and 9; contrast this with the two paragraphs describing a problem-specific tile-coding originally used to solve the Acrobot. For larger problems, the Fourier basis provides a generic but complete basis function scheme suitable for feature selection. For a single continuous state variable, , (scaled to ), the th order Fourier basis is the set of basis functions defined as: for . Thus, an order Fourier basis produces basis functions, the first having value 1 everywhere. The following pictures show a few example basis functions; note that, as increases, the basis functions become higher frequency.
For multivariate state spaces, where
where each cj is a length
A few
example basis functions for a 2D domain are shown below. Each element of the
One caveat: when performing
gradient-descent style value function approximation (such as TD(
An The paper describing the Fourier Basis is:
Source code for an RL-Glue-compatible epsilon-greedy Sarsa(λ) agent is available on the RL Library community wiki. The code is in Java and is fully documented. (Apache License) Here is a list of papers using the Fourier basis, courtesy of Google Scholar. | ||||||||||||||||||
![]() ![]() |