1

I am programming a custom optimizer now, and the legth of bias in first dimension is not certain because last batch have no enough data to bulid a batch. So the initialization of weights with fixed batch_size do not satisfy for torch.add between the last batch and the fixed length weights.

bias = torch.randn(batch_size,units)
batch_data = generator(path)
# for example 
weights.shape # is (128,256)
# but the last batch has only 50 samples.
out = sigmoid(x*weights+bias) # where the length of first dimension is not mathed.

So, I wonder whether I can create a tensor where the lenght of some dimension could be variable, like variable length list.

Whisht
  • 681
  • 2
  • 6
  • 20

2 Answers2

0

Why do you want bias to depend on batch size? In test time, would you always test your net with batches of exactly the same size? If so, what is the meaning of a smaller batch?

If you still insist on using smaller batch, you can ignore the "un-used" entries of bias:

out = sigmoid(x * weights[:x.size(0), ...] + bias[:x.size(0), ...])
Shai
  • 111,146
  • 38
  • 238
  • 371
  • My bad, in my Derivation of the optimizer, I was using each **element** for computation simplicity. And when it turns to **vector** or **matrix** situatioin, I should aplly a sum operation to all the grad. – Whisht May 30 '19 at 08:13
0

This link may be helpful for you: How to initialize weights in PyTorch?

If you use Pytorch's built in data loader class in pytorch. It will generate an iterator to be used to automatically handle batching. The batch size should be explicitly set up front by passing the 'batch_size' keyword to your data loader.

The last batch will be smaller if the dataset isn't divisible by the batch size unless drop last is explicitly set to true for the data loader.

Bias's don't work like this and don't depend on the size of the dataset or batch size.

gordon macmillan
  • 123
  • 3
  • 13