How to find minimum of a function with TensorFlow

Question

I try to implement a visualization for optimization algorithms in TensorFlow.

Therefore I started with the Beale's function

The global minimum is at

A plot of the Beale's function does look like this

I would like to start at the point f(x=3.0, y=4.0)

How do I implement this in TensorFlow with optimization algorithms?

My first try looks like this

import tensorflow as tf

# Beale's function
x = tf.Variable(3.0, trainable=True)
y = tf.Variable(4.0, trainable=True)
f = tf.add_n([tf.square(tf.add(tf.subtract(1.5, x), tf.multiply(x, y))),
          tf.square(tf.add(tf.subtract(2.25, x), tf.multiply(x, tf.square(y)))),
          tf.square(tf.add(tf.subtract(2.625, x), tf.multiply(x, tf.pow(y, 3))))])

Y = [3, 0.5]

loss = f
opt = tf.train.GradientDescentOptimizer(0.1).minimize(loss)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

for i in range(100):
    print(sess.run([x, y, loss]))
    sess.run(opt)

Obviously this doesn't work. I guess I have to define a correct loss, but how? To clearify: My problem is that I don't understand how TensorFlow works and I don't know much python (coming from Java, C, C++, Delphi, ...). My question is not on how this works and what the best optimization methods are, it's only about how to implement this in a correct way.

Answer 1

Oh I already could figure it out. The problem is that I need to clip the max and min values of x and y to -4.5 and 4.5 so that they don't explode to infinite.

This solution works:

import tensorflow as tf

# Beale's function
x = tf.Variable(3.0, trainable=True)
y = tf.Variable(4.0, trainable=True)
f = tf.add_n([tf.square(tf.add(tf.subtract(1.5, x), tf.multiply(x, y))),
          tf.square(tf.add(tf.subtract(2.25, x), tf.multiply(x, tf.square(y)))),
          tf.square(tf.add(tf.subtract(2.625, x), tf.multiply(x, tf.pow(y, 3))))])

opt = tf.train.GradientDescentOptimizer(0.01)
grads_and_vars = opt.compute_gradients(f, [x, y])
clipped_grads_and_vars = [(tf.clip_by_value(g, -4.5, 4.5), v) for g, v in grads_and_vars]

train = opt.apply_gradients(clipped_grads_and_vars)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

for i in range(100):
    print(sess.run([x, y]))
    sess.run(train)

If someone knows if it's possible to add multiple neurons / layers to this code, please feel free to write a answer.