1

I have a Stochastic Optimal Control problem that I wish to solve, using some type of Bayesian Simulation based framework. My problem has the following general structure:

s_t+1 = r*s_t(1 - s_t) - x_t+1 + epsilon_t+1
x_t+1 ~ Beta(u_t+1, w_t+1)
u_t+1 = f_1(u_t,w_t, s_t, x_t)
w_t+1 = f_2(u_t,w_t, s_t, x_t)
epsilon_t ~ Normal(0,sigma)
objective function: max_{x_t} E(Sigma_{t=0}^{T} V(s_t,x_t,c) * rho^t)

My goal is to explore different functional forms of f_1, f_2, and V to determine how this model differs w.r.t a non-stochastic model and another simpler stochastic model.

State variables are s_t, control variables are x_t with u_t and w_t representing some belief of the current state. The objective function is the discounted maximum from gains (function V) over the time period t=0 to t=T.

I was thinking of using Python, specifically PyMC to solve this, though I am not sure how to proceed, specifically how to optimize the control variables. I found a book, published 1967, Optimization of Stochastic Systems by Masanao Aoki, that references some bayesian techniques that may be useful, is there a current Python implementation that may help? Or is there a much better way to simulate a optimal path, using Python?

P_J
  • 67
  • 9

1 Answers1

1

The first guess coming to my mind is to try neural network packages like chainer or theano which can track derivative of your cost function with respect to control function parameters; they also have a bunch of optimization plug-in routines. You can use numpy.random to generate samples (particles), compose your control functions from the libraries components, and run them through explicit Euler scheme for first try. This will give you cost function on your particles and its derivative with respect to parameters, which can be fed to the optimizers.

The issue that can arise here is that solver's iterations will create a host of derivative-tracking objects.

update: Please see this example on Github

Also there is a number of hits on Github with keywords particle filter python:

https://github.com/strohel/PyBayes

https://github.com/jerkern/pyParticleEst

Also there is a manuscript around which mentions that the author implemented filters in Python, so you might want to contact them.

Dima Lituiev
  • 12,544
  • 10
  • 41
  • 58
  • Recurrent Neural Networks seem like a really nice approach, however how do I incorporate the stochasticity into the model? Also, the variable I wish to estimate/change is x_t, such that the objective function is optimized, what in this case will be my input state and output (observed state)? I am fairly new to neural networks, so any advise is much appreciated. – P_J Nov 27 '15 at 22:46
  • I posted an example on Github. Please see link above – Dima Lituiev Nov 28 '15 at 10:03
  • Thanks for the information, I ended up using a very large joint probability model implemented in PyMC. As the stochastic part was rather key, and could not get my head around how to incorporate that in the existing framework. But, definitely gonna look into this some more! Thanks again! – P_J Dec 10 '15 at 14:09