How to add `accumulateWeighted` support to OpenCV.js?

Question

Context: This is related to "How to build a custom opencv.js with opencv_contrib ?" in a general sense.

This question is a very specific variant: I'd like to use accumulateWeighted from the imgproc module with OpenCV.js

My attempt so far involved getting the hang of where things are and so I can try and tweak the emscripten setup a bit. As far as I understand most of files I need to work with live in:

For example I can see this section in opencv_js.config.py:

imgproc = {'': ['Canny', 'GaussianBlur', 'Laplacian', 'HoughLines', 'HoughLinesP', 'HoughCircles', 'Scharr','Sobel', \
                'adaptiveThreshold','approxPolyDP','arcLength','bilateralFilter','blur','boundingRect','boxFilter',\
                'calcBackProject','calcHist','circle','compareHist','connectedComponents','connectedComponentsWithStats', \
                'contourArea', 'convexHull', 'convexityDefects', 'cornerHarris','cornerMinEigenVal','createCLAHE', \
                'createLineSegmentDetector','cvtColor','demosaicing','dilate', 'distanceTransform','distanceTransformWithLabels', \
                'drawContours','ellipse','ellipse2Poly','equalizeHist','erode', 'filter2D', 'findContours','fitEllipse', \
                'fitLine', 'floodFill','getAffineTransform', 'getPerspectiveTransform', 'getRotationMatrix2D', 'getStructuringElement', \
                'goodFeaturesToTrack','grabCut','initUndistortRectifyMap', 'integral','integral2', 'isContourConvex', 'line', \
                'matchShapes', 'matchTemplate','medianBlur', 'minAreaRect', 'minEnclosingCircle', 'moments', 'morphologyEx', \
                'pointPolygonTest', 'putText','pyrDown','pyrUp','rectangle','remap', 'resize','sepFilter2D','threshold', \
                'undistort','warpAffine','warpPerspective','warpPolar','watershed', \
                'fillPoly', 'fillConvexPoly'],
           'CLAHE': ['apply', 'collectGarbage', 'getClipLimit', 'getTilesGridSize', 'setClipLimit', 'setTilesGridSize']}

I can simply add accumulateWeighted to the list, however I feel this should also adjust bindings.cpp / core_bindings.cpp appropriately. This is where some of confusion lies as I pickup experience with emscripten.

For example adding:

// C++: void accumulateWeighted(InputArray src, InputOutputArray dst, double alpha, InputArray mask=noArray() )
void accumulateWeighted_wrapper(const cv::Mat& src, const cv::Mat& dst, double alpha, cv::Mat& mask) {
    return cv::accumulateWeighted(src, dst, alpha, mask);
}

void accumulateWeighted_wrapper_1(const cv::Mat& src, const cv::Mat& dst, double alpha) {
    return cv::accumulateWeighted(src, dst, alpha);
}
// ...
function("accumulateWeighted", select_overload<void(const cv::Mat&, const cv::Mat&, double , cv::Mat&)>(&Wrappers::accumulateWeighted_wrapper));

function("accumulateWeighted", select_overload<void(const cv::Mat&, const cv::Mat&, double)>(&Wrappers::accumulateWeighted_wrapper_1));

to bindings.cpp

I'm not sure if I'm missing anything else ? I'm hoping devs with more OpenCV + emscripten experience can provide guidance.

Update I've compiled OpenCV.js using Docker and tried calling cv.accumulateWeighted, but I'm getting an error:

cv.accumulateWeighted(src, dst, 0.001)
opencv.js:9 Uncaught 6587800
___resumeException  @   opencv.js:9
(anonymous) @   02086862:0x1621d4
(anonymous) @   02086862:0x1c1f8
dynCall_viid    @   02086862:0x365dc
dynCall_viiid   @   02086862:0x37296
Module.dynCall_viiid    @   opencv.js:9
dynCall_viiid_532   @   VM1966:4
accumulateWeighted  @   VM3269:10
proto.<computed>    @   opencv.js:9
(anonymous) @   VM5257:1

I'm not 100% sure what I've got wrong. Here's a snippet which includes a link to the compiled script:

function onOpenCvReady(){
  cv.then(test);
}

function test(cv){
  console.log("cv",cv.getBuildInformation());
  
  src = cv.Mat.ones(3,3, cv.CV_8UC1);
  dst = cv.Mat.ones(3,3, cv.CV_8UC1);
  mask = cv.Mat.zeros(3,3, cv.CV_8UC1);


  console.log("dst before", dst.data);

// throws error
  try{
    cv.accumulateWeighted(src, dst, 0.001, mask);
  }catch(err){
    console.warn("error running accumulateWeighted")
    console.warn(err.stack)
  }
  console.log("dst after", dst);
}

<script async src="https://lifesine.eu/so/opencv_js_ubuntu/opencv.js" onload="onOpenCvReady();" type="text/javascript"></script>

Update I've updated the snippet above to use OpenCV.js compiled on Ubuntu. Additionally I had a typo in the accumulateWeighted_wrapper and accumulateWeighted_wrapper_1 functions, however I'm still getting a very similar error which leads me to believe there is something else I'm missing in the bindings code.

What is the correct way of adding accumulateWeighted support to OpenCV.js ?

I don't think there is a function cv::accumulateWeighted(src, dst, alpha) you need mask as the last parameter https://github.com/opencv/opencv/blob/1913482cf5d257d7da292bb2c15f22d0588d34dc/modules/imgproc/src/accum.cpp#L611 — user2258152, Sep 25 '20 at 01:49
@user2258152 No worries: good point on clarity. I did update the post above to test the same thing with a mask argument. — George Profenza, Sep 25 '20 at 09:27
They have emscripten example maybe it can help you https://github.com/opencv/opencv/blob/7722a2b8a838f80d3e01c2a60036a29759b2b64b/modules/js/src/core_bindings.cpp — user2258152, Sep 25 '20 at 14:32
@user2258152 Yup, as you can see above, I've attempted to add `accumulateWeighted_wrapper` and `accumulateWeighted_wrapper_1` to `core_bindings.cpp` then recompiled OpenCV.js. No c++ compiler errors, just the weird JS one. Notice that if you attempt to run the snippet above with a different version of opencv.js (e.g. https://docs.opencv.org/4.4.0/opencv.js) you'll get a different error because `accumulateWeighted` will be missing — George Profenza, Sep 25 '20 at 18:04
I know it's old but what about this https://github.com/ganwenyao/opencv_js — user2258152, Sep 25 '20 at 18:42
@user2258152 Thank you for the due diligence. Surpringly I did compile that a few days ago as I provided a related [answer](https://stackoverflow.com/questions/53151595/how-to-build-a-custom-opencv-js-with-opencv-contrib/63986128#63986128). I've tweaked [core_bindings.cpp](https://github.com/ganwenyao/opencv_js/blob/master/opencv/modules/js/src/core_bindings.cpp) in his version (because accumulateWeighted wasn't part of it), but got the same js error as above. (cleared cache, tried different browser just in case). either I didn't get the c++ types right or I'm missing something else — George Profenza, Sep 25 '20 at 19:31

score 0 · Answer 1 · answered Oct 03 '20 at 02:22

For the scope of this question one workaround is to not modify OpenCV.js and recompile, to implement the functionality in JS. The function roughly has two parts (and a gotcha):

accumulating: adding a new image on top of an existing one
weighing the new image inversely proportional to old image
the catch is using an image to act as an accumulator: otherwise it's simply linearly interpolating/ lerping / cross fading between two images:

cv.lerp = function(lerpFromMat, lerpToMat, lerpResult, amount){
    // TODO: args safety check (including constraining amount)
    if (lerpToMat.cols === 0) {
      lerpFromMat.copyTo(lerpResult);
    } else if (lerpFromMat.cols === 0) {
      lerpToMat.copyTo(lerpResult);
    } else {
      cv.addWeighted(lerpFromMat, amount, lerpToMat, 1.0 - amount, 0.0, lerpResult);
    }
  }

// super simplified alias, skipping mask for now
cv.accumulateWeighted = function(newMat, accumulatorMat, alpha){
   p5cv.lerp(accumulatorMat, newMat, accumulatorMat, alpha);
}

I still look forward to another general solution: a breakdown of the process to add new c++ functions to OpenCV.js via emscripten (be it accumulateWeighted or something else)

How to add `accumulateWeighted` support to OpenCV.js?

1 Answers1