Inaccurate result with opencv DFT (FW and backward)

Question

I wanted to perform some denoising operations using opencv's DFT by doing forward fourier performing some operations and the doing the inverse fourier. But I started to notice that in some cases noise appears out of nowhere. So I tested just doing forward backward with no operation in between and I don't correctly recover the original signal, does anyone know why? All my zeros should remain zeros or at least very low values, here my result is at the power of -5 -6, which is not that good. Even if I work with CV_64F, results are not much better. In MATLAB this same operation gives me the right result.

Here is a closeup to my input image

And here is a closeup to the output

And this is the code I used:

void func(cv::Mat &I,cv::Mat &G){
    const int n1=G.rows;
    const int m1=G.cols;

    const int nl = static_cast<int>(n1/2);
    const int ml = static_cast<int>(m1/2);

    const int n = I.rows + 2*nl;
    const int m = I.cols + 2*ml; // on the border add zero values

    cv::Mat paddedI;
    cv::Mat planesI[] = {cv::Mat_<float>(paddedI), cv::Mat::zeros(paddedI.size(), CV_32F)};
    merge(planesI, 2, paddedI);         // Add to the expanded another plane with zeros
    cv::dft(paddedI, paddedI, cv::DFT_COMPLEX_OUTPUT);
    cv::dft(paddedI, paddedI, cv::DFT_INVERSE | cv::DFT_SCALE);//I could just output the real part
    split(paddedI, planesI);

    planesI[0].rowRange(nl,nl+I.rows+1).colRange(ml,ml+I.cols+1).copyTo(I);
}