25

I am following the PyTorch tutorial here. It says that

x = torch.randn(3, requires_grad=True)

y = x * 2
while y.data.norm() < 1000:
    y = y * 2

print(y)

Out:    
tensor([-590.4467,   97.6760,  921.0221])

Could someone explain what data.norm() does here? When I change .randn to .ones its output is tensor([ 1024., 1024., 1024.]).

kmario23
  • 57,311
  • 13
  • 161
  • 150
bit_scientist
  • 1,496
  • 2
  • 15
  • 34
  • I don’t know PyTorch but after some searching I think that the norm() method could be related to [the mathematical norm](https://en.m.wikipedia.org/wiki/Norm_(mathematics)). I don’t know if this is even the same method but I also found a PyTorch doc with a norm() method [here](https://pytorch.org/docs/master/torch.html#torch.norm). – 0xCursor Jun 08 '18 at 05:13

4 Answers4

32

It's simply the L2 norm (a.k.a Euclidean norm) of the tensor. Below is a reproducible illustration:

In [15]: x = torch.randn(3, requires_grad=True)

In [16]: y = x * 2

In [17]: y.data
Out[17]: tensor([-1.2510, -0.6302,  1.2898])

In [18]: y.data.norm()
Out[18]: tensor(1.9041)

# computing the norm using elementary operations
In [19]: torch.sqrt(torch.sum(torch.pow(y, 2)))
Out[19]: tensor(1.9041)

Explanation: First, it takes a square of every element in the input tensor x, then it sums them together, and finally it takes a square root of the resulting sum. All in all, these operations compute the so-called L2 or Euclidean norm.

kmario23
  • 57,311
  • 13
  • 161
  • 150
6

Building on what @kmario23 says, the code multiplies the elements of a vector by 2 until the Euclidean magnitude (distance from origin) / L2 norm of the vector is at least 1000.

With the example of the vector with (1,1,1): it increases to (512, 512, 512), where the l2 norm is about 886. This is less than 1000, so it gets multiplied by 2 again and becomes (1024, 1024, 1024). This has a magnitude greater than 1000, so it stops.

Jonathan
  • 4,847
  • 3
  • 32
  • 37
1
y.data.norm()

is equivalent to

torch.sqrt(torch.sum(torch.pow(y, 2)))
Andy
  • 21
  • 4
1

Lets break it step wise to understand the code better.

The below block of code creates a tensor x of shape (1,3)

x = torch.ones(3, requires_grad=True)
print(x)

>>> tensor([1., 1., 1.], requires_grad=True)

The below block of code creates a tensor y by multiplying each element of x by 2

y = x * 2
print(y)
print(y.requires_grad)

>>> tensor([2., 2., 2.], grad_fn=<MulBackward0>)
>>> True

TORCH.data returns a tensor whose requires_grad is set to false

print(y.data)
print('Type of y: ', type(y.data))
print('requires_grad: ', y.data.requires_grad)

>>> tensor([2., 2., 2.])
>>> Type of y:  <class 'torch.Tensor'>
>>> requires_grad:  False

TORCH.norm() Returns the matrix norm or vector norm of a given tensor. By default it returns a Frobenius norm aka L2-Norm which is calculated using the formula

.

In our example since every element in y is 2, y.data.norm() returns 3.4641 since is equal to 3.4641

print(y.data.norm())

>>> tensor(3.4641)

The below loop is run until the norm value is less than 1000

while y.data.norm() < 1000:
    print('Norm value: ', y.data.norm(), 'y value: ', y.data )
    y = y * 2

>>> Norm value:  tensor(6.9282) y value:  tensor([4., 4., 4.])
>>> Norm value:  tensor(3.4641) y value:  tensor([2., 2., 2.])
>>> Norm value:  tensor(13.8564) y value:  tensor([8., 8., 8.])
>>> Norm value:  tensor(27.7128) y value:  tensor([16., 16., 16.])
>>> Norm value:  tensor(55.4256) y value:  tensor([32., 32., 32.])
>>> Norm value:  tensor(110.8512) y value:  tensor([64., 64., 64.])
>>> Norm value:  tensor(221.7025) y value:  tensor([128., 128., 128.])
>>> Norm value:  tensor(443.4050) y value:  tensor([256., 256., 256.])
>>> Norm value:  tensor(886.8100) y value:  tensor([512., 512., 512.])
>>> 
>>> Final y value:  tensor([1024., 1024., 1024.], grad_fn=<MulBackward0>)
IM80554Coding
  • 86
  • 4