Pytorch1.6 What is the actual learning rate during training?

Question

I'd like to know the actual learning rate during training, here is my code.

learning_rate = 0.001
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[1, 2], gamma=0.1)


def train(epoch):
    train_loss = 0

    for batch_idx, (input, target) in enumerate(train_loader):
        predict_label = net(input)
        loss = criterion(predict_label, target)
        train_loss += loss.item()

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print(optimizer.param_groups[0]['lr'])

    scheduler.step()
    print(scheduler.state_dict()['_last_lr'])
    print(optimizer.param_groups[0]['lr'])

the output is 0.001, 0.0001, 0.0001. So what is the actual lr during optimizer.step()? 0.001 or 0.0001? Thanks.

score 0 · Accepted Answer · answered Mar 26 '21 at 03:44

The important part is here:

for batch_idx, (input, target) in enumerate(train_loader):
    ...

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print(optimizer.param_groups[0]['lr']) #### CURRENT LEARNING RATE

scheduler.step() #step through learning rate
print(scheduler.state_dict()['_last_lr']) #### NEW LEARNING RATE
print(optimizer.param_groups[0]['lr']) #### NEW LEARNING RATE

Because you step your scheduler after your epoch, then the first epoch will have your initial value which is set to 0.001. If you run for multiple epochs then it will continue to be annealed.

Pytorch1.6 What is the actual learning rate during training?

1 Answers1