1

Can I get a Gradient Clipping function in Chainer?

I found some codes in Pytorch Documentation : https://pytorch.org/docs/stable/_modules/torch/nn/utils/clip_grad.html

Is there anything like alternative function in Chainer? I just found chainer.optimizer_hooks.GradientClipping, but this is hard to utilize.

Thanks in advance.

Cordon Raj
  • 11
  • 1
  • why do you actually need a separate function for it? Basically, you want to utilize gradient clipping during the optimization process, that's why it is implemented as an optimizer hook. – DiKorsch Oct 09 '19 at 07:53

1 Answers1

0

how about trying this. just I just re-wrote pyTorch function in Chainer style.

import cupy
def clip_grad_norm(model, max_norm, norm_type=2):
    params  = list( filter(lambda p : p.grad is not None ,  model.params()) )
    max_norm = float(max_norm)
    norm_type = float(norm_type)
    total_norm = 0.0
    for p in params:
        g = p.grad
        norm = cupy.linalg.norm(g)
        total_norm += norm**(norm_type)
    total_norm = total_norm **(1/norm_type)
    clip_coef = max_norm / (total_norm + 1e-6)
    if clip_coef < 1:
        for p in params:
            g = p.grad
            p.grad = g * clip_coef
yeachan park
  • 154
  • 2
  • 4