How to dynamically remove residual block or reduce residual block size in Ceres Solver

Question

Can I add or remove a Residual block between optimization iterations? I am trying to use auto differentiation.

I am using Ceres-Solver for a camera tracking problem. I want to find the best relative pose by minimizing the photometric errors of feature points. However every iteration as I update the pose, some reprojection points fall outside the camera view and therefore cannot be used in the optimization. So far each feature point is a residual block.

Answer 1

No you can't. And there is a good reason for it. Because adding and removing residual blocks as the optimization is being performed, changes the objective function and Ceres Solver would not be able to check and guarantee that that the steps taken by the solver actually reduce the objective function.