12

During training, at each epoch, I'd like to change the batch size (for experimental purpose). Creating a custom Callback seems appropriate but batch_size isn't a member of the Model class.

The only way I see would be to override fit_loop and expose batch_size to the callback at each loop. Is there a cleaner or faster way to do it without using a callback ?

0xmax
  • 503
  • 1
  • 6
  • 26

3 Answers3

14

For others who land here, I found the easiest way to do batch size adjustment in Keras is just to call fit more than once (with different batch sizes):

model.fit(X_train, y_train, batch_size=32, epochs=20)

# ...continue training with a larger batch size
model.fit(X_train, y_train, batch_size=512, epochs=10)
slund
  • 966
  • 8
  • 9
8

I think it will be better to use a custom data generator to have control over the data you pass to the training loop, so you can generate batches of different sizes, process data on the fly etc. Here is an outline:

def data_gen(data):
  while True: # generator yields forever
    # process data into batch, it could be any size
    # it's your responsibility to construct a batch
    yield x,y # here x and y are a single batch

Now you can train with model.fit_generator(data_gen(data), steps_per_epoch=100) which will yield 100 batches per epoch. You can also use a Sequence if you want to encapsulate this inside a class.

Ioannis Nasios
  • 8,292
  • 4
  • 33
  • 55
nuric
  • 11,027
  • 3
  • 27
  • 42
0

For most purposes the accepted answer is the best, don't change the batch size. There's probably a better way 99% of the time that this question comes up.

For those 1%-ers who do have an exceptional case where changing the batch size mid-network is appropriate there's a git discussion that addresses this well:

https://github.com/keras-team/keras/issues/4807

To summarize it: Keras doesn't want you to change the batch size, so you need to cheat and add a dimension and tell keras it's working with a batch_size of 1. For example, your batch of 10 cifar10 images was sized [10, 32, 32, 3], now it becomes [1, 10, 32, 32, 3]. You'll need to reshape this throughout the network appropriately. Use tf.expand_dims and tf.squeeze to add and remove a dimension trivially.

David Parks
  • 30,789
  • 47
  • 185
  • 328
  • 2
    "Don't change the batch size." --- [research begs to differ](https://arxiv.org/abs/1711.00489). Edit: Okay, I didn't fully read OP's question since I landed here by searching "Keras change batch size" –  Nov 02 '19 at 20:22
  • 2
    Good reference, good paper, though even that is still being debated. There are further papers on large batch training, here's a quick one I grabbed with a google search, https://arxiv.org/abs/1609.04836, which back the idea that large fixed batches might be a good thing still. It seems to be an open problem still, my take is that it probably depends on your problem. There are so hyperparameters and factors involved in these models it's challenging to disentangle their effects. – David Parks Nov 03 '19 at 17:17
  • 1
    The linked discussion talks about changing batch size for intermediate layers. I think that the OP is asking about changing the Batch size over epochs, which is quite different. Anyway, thanks for the answer. I upvoted it because it is a good contrib. – Duloren Mar 03 '22 at 17:08