0

I have a trained a tensorflow seq2seq model for 30 epochs, and saved a checkpoint for each epoch. What I want to do now is combining the best X of those checkpoints (based on results on a development set). Specifically, I'm looking for a way that lets me average different model weights and merge them into a new model that can be used for decoding. However, there does not seem to be a set way for this, and loading different models can be a bit tricky. But even if this succeeds, I can't find a good answer on how to combine the weights in a new model.

Any help would be greatly appreciated.

Related questions (that do not sufficiently answer in my opinion):

Building multiple models in the same graph

How to load several identical models from save files into one session in Tensorflow

How to create ensemble in tensorflow?

Community
  • 1
  • 1
RNRug
  • 55
  • 5
  • Can you describe why you'd want to average the different model weights together? – saeta Mar 16 '17 at 21:11
  • I want to create an ensemble of the best X models. So, instead of choosing the best model, I want to average the best X models. This seems to increase the final results in Machine Translation tasks. The most logical way of doing this (to me) seems by averaging the weights of the model. – RNRug Mar 17 '17 at 09:18

2 Answers2

2

First, a bit of terminology:

  • In ensembles (as I understand them) you have N models at test time and you combine their predictions (by voting, or even better combining probabilistic distributions and using as input for further decoding in case of autoregressive seq2seq decoders). You can have independent ensembles (training each model independently from scratch, with different random initialization) or checkpoint ensembles (taking N last checkpoints, or possibly N checkpoints with best validation score). See e.g. Sennrich et al., 2017 for a comparison of these two types of ensembles.

  • In averaging you average the weights of N models, so at test time you have just one averaged model. This gives usually worse results than real ensembles, but it is much faster, so you can afford higher N. If the models are trained completely independently with different random initialization, averaging does not work at all. However, if the models share a reasonable number of initial training steps, averaging may work. A special case is checkpoint averaging where the last N checkpoints are averaged, but you can try even "forking" the training and using "semi-independent" models for averaging (in addition to checkpoint averaging). It may be very useful to use constant or cyclical learning rate, see Izmailov et al., 2018.

As for your question, how to do the averaging of Tensorflow checkpoints: See avg_checkpoints.py or t2t-avg-all.

Martin Popel
  • 2,671
  • 12
  • 22
-1

Doing an average of the weights of several models to produce a new one is unlikely to produce a useful result.

For a simple example, think about a classic CNN like AlexNet. Its first layer will contain a series of 2d filters looking for different image features. For every model you train from scratch, it's likely that similar features may show up in the filters but the orders they occur will be very different, and so just averaging weights will destroy most of the information.

Pete Warden
  • 2,866
  • 1
  • 13
  • 12
  • 1
    I don't agree. Averaging may produce useful results, if the models are not completely independent, see e.g. [Utans, 1996](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.55.7218&rep=rep1&type=pdf), [Junczys-Dowmunt et al., 2016](https://arxiv.org/pdf/1605.04809.pdf#page=7) and [Izmailov et al., 2018](https://arxiv.org/pdf/1803.05407.pdf). – Martin Popel May 09 '18 at 15:32