0

I have a network which works as follows: The input is split in half; the first half is put through some convolutional layers l1, then the second half is put through the same layers l1 (after the output for the first half of the input has been computed), then the two output representations are concatenated and put through additional layers l2 at once. Now my question (similar to Can autograd in pytorch handle a repeated use of a layer within the same module? but not quite the same setting as in the other question, the same layer was reused in different depths of the computation graph, whereas here, the same layer is used twice within the same depth) is: does autograd handle this properly? I.e. is the backpropagation error for l1 computed with respect to both of its forward passes and the weights are adapted w.r.t. both of these at once?

joinijo
  • 353
  • 2
  • 9
  • Please provide a minimal example of what you think may be causing invalid results. From your description I don't see why this would be a problem for autograd. Gradients should be accumulated properly based on both instances where the layer was used. – jodag May 11 '22 at 16:40

1 Answers1

1

Autograd does not care how many times you "use" something. This is not how it works. it just builds a graph behind the scenes of the dependencies, using something twice just makes a graph that is not a line, but it will not affect its execution.

lejlot
  • 64,777
  • 8
  • 131
  • 164
  • Alright, that's what I was thinking but I just wasn't sure if that's how autograd works (or if the graph has to be a line). Thank you :) – joinijo May 12 '22 at 08:53