PyBrain Reinforcement Learning Input Buffer Incorrect

Question

I am trying to set up PyBrain for reinforcement learning, but keep on getting the same error when I try to get an action for the first time. This line in module.py is throwing an assert failure because the input buffer on the ActionValueTable (a child of module) we are using is not set properly.

 assert len(self.inputbuffer[self.offset]) == len(inpt)

Has anyone else experienced a similar issue or have insight on how to solve the problem?

Answer 1

I had the same problem. I replaced net.activate(tstdata) with net.activateOnDataset(tstdata) and it worked for me. That's the function they use in sample code as well.

Answer 2

I'm having a similar problem- I ran pdb to check indim and outdim, and they were both incorrect (i.e. not the values we set in our environment file.)

Make sure those values are correct and please follow up with any progress.

Answer 3

self.offset is the problem for me:

(49)activate()
-> self.inputbuffer[self.offset] = inpt
(Pdb) p self
<RecurrentNetwork 'RecurrentNetwork-13'>
(Pdb) p self.inputbuffer
array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]])
(Pdb) p inpt
array([ 0.36663106,  0.10664821, -0.09483858,  0.24661628, -0.33891044,
       -0.16277863, -0.46995505,  0.43191341,  0.46647206, -0.14306874])
(Pdb) p self.offset
3825
(Pdb) 

Edit: FIXED

net.offset = 0 # wtf pybrain
for inp, target in testDS:
    netOut.extend(net.activate(inp))

Context: I was printing out the results of the network after having trained it with the built-in GA of pybrain.

I've used recurrent networks before and had no problem (same dataset even) and so was curious as to what went wrong. I haven't delved into what the GA (or other thing I don't know about) did to the network but regardless setting the offset to 0 before entering a loop that involves net.activate() fixed it and now I'm getting proper activations (make sure to set it to 0 before the loop, not during).

Maybe this happened because I had trained it on separate data that it still thought was involved?

Good luck!

Answer 4

@Emi Nietfeld, I faced the same problem i.e. my indim and outdim values were incorrect. I checked the implementation of ActionValueTable(). ActionValueTable has been implemented in a file called interface.py and it's path is pybrain\rl\learners\valuebased\interface.py .

The constructor is as follows

def __init__(self, numStates, numActions, name=None)

and the call to the super constructor is as follows

Module.__init__(self, 1, 1, name)

So indim and outdim were always initialised to 1 I edited the call to super constructor as follows

Module.__init__(self, numStates, numActions, name)

This worked for me