2

I have a picture and I want to find the crop with the minimal area that at the same time retains a certain percentage of the edge energy.

My take on this was to formulate this as a optimization problem and let scipy's constrained optimizers solve it [code see below]. This apparently is problematic since it is an integer problem (croping takes the integer coordinates of the top-left and down-right corners as parameters). And indeed fmin_cobyla fails to find a solution after some 20s runtime, while fmin_slsqp fails after one iteration with "Singular matrix C in LSQ subproblem (Exit mode 6)".

Any ideas on how I might tackle this problem otherwise? Is there by chance a library that handles optimization problems on images?

from skimage.filters import sobel
from PIL import Image
from scipy.optimize import fmin_slsqp

def objective(x):
    # minimize the area
    return abs((x[2] - x[0]) * (x[3] - x[1]))

def create_ratio_constr(img):
    def constr(x):
        # 81% of the image energy should be contained
        x = tuple(map(int, x))
        crop = img.crop((x[0], x[1], x[2], x[3]))
        area_ratio = round(sum(list(crop.getdata())) /
                           float(sum(list(img.getdata()))), 2)
        if area_ratio == 0.81:
            return 0.0
        return -1
    return constr

def borders_constr(x):
    x = tuple(map(int, x))
    # 1st point is up and left of 2nd point
    rectangle = x[0] < x[2] and x[1] < x[3]

    # only positive values valid
    positive = x[0] > 0 and x[1] > 0

    if rectangle and positive:
        return 0.0
    return -1

img = Image.open("/some/path.jpg")
# get the edges
edges = Image.fromarray(sobel(img.convert("L")))

ratio_constr = create_ratio_constr(edges)
x = fmin_slsqp(objective,
               (0, 0, edges.size[0]-1, edges.size[1]-1),
               [borders_constr, ratio_constr],
               disp=1)

print x
Community
  • 1
  • 1
Zakum
  • 2,157
  • 2
  • 22
  • 30
  • Try writing constraint functions that tell the optimizer by how much the constraint is violated. Binary-valued constraints are not useful, because the optimizer assumes continuous constraint functions. It however may work for functions that are "continuous except for pixel quantization". You also cannot specify an equality constraint like that as inequality, because of continuity assumptions. COBYLA has equality constraints. – pv. Nov 16 '15 at 08:48

1 Answers1

0

I would probably ignore the integer requirement for the corners of the cropped area and solve the relaxed problem. Then consider moving the cropped edges in or out to the nearest whole pixel. If it is for aesthetic purposes, the +/- part pixel probably doesn't matter. If it has to be correct, there are only four edges to consider in two places each so should not be a big deal to find the one out of 16 choices which is best.

TimChippingtonDerrick
  • 2,042
  • 1
  • 12
  • 13
  • That is what I am doing in the provided code. The minimization algorithms work on float values, and there output values are only rounded down in the constraint functions (because the cropping wont work on floats). However even this seems to be enough of a problem to render this insolvable, if I understand the linked question: [Why can't I rig SciPy's constrained optimization for integer programming?](http://stackoverflow.com/questions/15793381/why-cant-i-rig-scipys-constrained-optimization-for-integer-programming?lq=1) correctly. – Zakum Nov 15 '15 at 19:47
  • 1
    Note that your problem formulations is not the relaxation to a continuous problem, since you have binary-valued constraints. The constraints should be written so that they are at least continuous functions (preferably differentiable), if you neglect the pixel quantization. – pv. Nov 16 '15 at 08:51