0

I'm trying to make output of BLSTM deterministic, after investigation its appeared that my dropout layer creates not deterministic dropout masks, so I was researching about how to fix random seed in pytorch. I found this page and other suggestions though I put everything in code it did not help. Here is my code:

import sys 
import random 
import datetime as dt 

import numpy as np 
import torch 

torch.manual_seed(42) 
torch.cuda.manual_seed(42) 
np.random.seed(42)
random.seed(42) 
torch.backends.cudnn.deterministic = True  

ex = torch.ones(10) 
torch.nn.functional.dropout(ex, p=0.5, training=True)                  
# Out[29]: tensor([0., 0., 2., 0., 0., 0., 0., 0., 2., 2.])

torch.nn.functional.dropout(ex, p=0.5, training=True)                  
# Out[30]: tensor([0., 2., 0., 2., 2., 0., 0., 2., 2., 2.])

Please help me get deterministic output from dropout for the same input

Maksym Titov
  • 45
  • 1
  • 7
  • 1
    Does this answer your question? [Training PyTorch models on different machines leads to different results](https://stackoverflow.com/questions/67511658/training-pytorch-models-on-different-machines-leads-to-different-results) – iacob May 18 '21 at 09:07

1 Answers1

1

every time you want the same output you need to reset the random seed again so:

>>> import torch

>>> torch.manual_seed(42)
<torch._C.Generator object at 0x127cd9170>

>>> ex = torch.ones(10)

>>> torch.nn.functional.dropout(ex, p=0.5, training=True)
tensor([0., 0., 2., 2., 2., 2., 2., 0., 2., 0.])

>>> torch.manual_seed(42)
<torch._C.Generator object at 0x127cd9170>

>>> torch.nn.functional.dropout(ex, p=0.5, training=True)
tensor([0., 0., 2., 2., 2., 2., 2., 0., 2., 0.])

You may want to keep resetting all those random seeds in general though, you can find a lot of different randomness when building neural nets in python

jamied157
  • 61
  • 4
  • Good catch. I was confusing when first saw the question. The seed determines: the sequence of the random number is the same between different runs. For the same seed and run, each call will get a different random number which is expected. – chunyang.wen Nov 04 '21 at 01:55