Tensorflow Adam with variable bounds

Question

I am using Tensorflow Adam method to optimize a stochastic function, which has (almost) nothing to do with neural networks, but rather with probabilistic inference.

Adam works pretty well in finding global optima of the cost function, however my variables are bounded and Adam has no way to implement bounds, as it is a unconstrained optimization method. In my case I would like to keep the variables all positive.

How is it possible to add bounds to stochastic gradient descent based methods in general? Are there already-implemented solutions to something that I think is a rather common problem?

score 3 · Answer 1 · answered Jan 21 '19 at 14:15

3

You can add dummy variables to enforce the bounds. For example, suppose you want a to be positive, what you can do is to introduce a variable b and compute

a = log(1 + exp(b))

Another example, if you want c to be within a and b, you can introduce d and compute

c = a + (b - a) / (1 + exp(-d))

answered Jan 21 '19 at 14:15

BlackBear

22,411
10
48
86

Interesting, do you have some reference to better undestand? – linello Jan 21 '19 at 16:05
@linello not really, sorry. I've seen this trick used every once in a while but never found anything talking about it specifically – BlackBear Jan 21 '19 at 16:09

Tensorflow Adam with variable bounds

1 Answers1