lrn's equation is as follows:
y = x * pow(lrn_k + lrn_alpha/lrn_size * sum(Xj * Xj), -lrn_beta)
the cudnn's lrn backward api is
cudnnStatus_t cudnnLRNCrossChannelBackward(
cudnnHandle_t handle,
cudnnLRNDescriptor_t normDesc,
cudnnLRNMode_t lrnMode,
const void *alpha,
const cudnnTensorDescriptor_t yDesc,
const void *y,
const cudnnTensorDescriptor_t dyDesc,
const void *dy,
const cudnnTensorDescriptor_t xDesc,
const void *x,
const void *beta,
const cudnnTensorDescriptor_t dxDesc,
void *dx)
I find there is no workspace in order to save scale which is implemented in caffe, which is
scale = lrn_k + lrn_alpha/lrn_size * sum(Xj * Xj)
this variable is useful in order to calculate lrn backward. So I think maybe in the backward kernel will just calculate from y, x and lrn_beta in order to calculate this variable, the equation is similar like this:
y/x = pow(scale, -lrn_beta)
y/x = pow2(-lrn_beta * log2(scale))
log2(scale) = -1/lrn_beta * log2(y/x)
scale = pow2(-1/lrn_beta * log2(y/x))
when I use these equations to calculate scale, cuda kernel generate NAN, is there other method which calculate lrn backward without saving scale value in forward? And why my method is not stable?
