8

I would like to train a GAN in Keras. My final target is BEGAN, but I'm starting with the simplest one. Understanding how to freeze weights properly is necessary here and that's what I'm struggling with.

During the generator training time the discriminator weights might not be updated. I would like to freeze and unfreeze discriminator alternately for training generator and discriminator alternately. The problem is that setting trainable parameter to false on discriminator model or even on its' weights doesn't stop model to train (and weights to update). On the other hand when I compile the model after setting trainable to False the weights become unfreezable. I can't compile the model after each iteration because that negates the idea of whole training.

Because of that problem it seems that many Keras implementations are bugged or they work because of some non-intuitive trick in old version or something.

Andrzej Pisarek
  • 271
  • 3
  • 9

3 Answers3

13

I've tried this example code a couple months ago and it worked: https://github.com/fchollet/keras/blob/master/examples/mnist_acgan.py

It's not the simplest form of GAN, but as far as I remembered, it's not too difficult to remove the classification loss and turn the model into a GAN.

You don't need to turn on/off the discriminator's trainable property and recompile. Simply create and compile two model objects, one with trainable=True (discriminator in the code) and another one with trainable=False (combined in the code).

When you're updating the discriminator, call discriminator.train_on_batch(). When you're updating the generator, call combined.train_on_batch().

Yu-Yang
  • 14,539
  • 2
  • 55
  • 62
0

Can you use tf.stop_gradient to conditionally freeze weights?

Alexandre Passos
  • 5,186
  • 1
  • 14
  • 19
  • `tf.stop_gradient` is stopping the gradient to flow and that's not what I want to achieve. I would like to make the gradient flow and compute gradients for weights, but not perform the update operation. – Andrzej Pisarek Jul 18 '17 at 09:51
  • Then you might be better off explicitly passing the list of variables you want to update to the tensorflow update op, instead of freezing / unfreezing weights all the time. – Alexandre Passos Jul 18 '17 at 16:07
  • You are right, but it's Tensorflow solution and Keras doesn't allow to do that. You have a model abstraction and you mainly have `fit` and `train_on_batch` methods, that's all. If there will be no solution in pure Keras then I'll switch to Tensorflow. – Andrzej Pisarek Jul 18 '17 at 17:20
0

Maybe your adversarial net(generator plus discriminator) are wrote in 'Model'. However, even you set the d.trainable=False, the independent d net are set non-trainable, but the d in the whole adversarial net is still trainable.

You can use the d_on_g.summary() before then after set d.trainable=False and you would know What I mean(pay attention to the trainable variables).

Munan Ning
  • 1