0

I need to solve an assignment problem between the pixels of two images. That means, I want to find the pixel from the left image that matches best to a given pixel in the right image. But not on a per pixel basis, but considering the overall cost of all assignments.

Usually, you build a cost matrix for that and then lower on a row and column basis until you get at least one zero in each column and row. Those zeros are optimal assignments then. However, the cost matrix for a 1920 * 1080 pixel image would be roughly 4TB in memory, which I can't handle.

Is there an alternative that uses less space to solve the assignment problem?

Cœur
  • 37,241
  • 25
  • 195
  • 267
danijar
  • 32,406
  • 45
  • 166
  • 297
  • Just to be clear, you're seeking the permutation of A which is closest (in some sense) to B? – Oliver Charlesworth Sep 03 '14 at 21:22
  • 1
    Do you need an exact one-to-one match here? Perhaps you could subdivide the image into blocks and match each block up, then match up the pixels in each block? – templatetypedef Sep 03 '14 at 21:22
  • I don't know the exact context but you could build a sparse matrix where each pixel in picture A can only be matched to some pixels in picture B. (the n'ths spatialy closest pixels for example) – Seb Sep 03 '14 at 21:26
  • @OliCharlesworth Right, in a very basic case I can assign every pixel from the left image to the closest pixel in the right image, in terms of color difference. However, this will not always result in the overall nearest approximation. I need to find the mapping so that the overall cost, thus sum over all color differences, is minimal. – danijar Sep 03 '14 at 21:26
  • @templatetypedef This is a good idea. The problem is with images with very low frequencies, those would be transferred to the final image. My goal is to approximate an image with the shuffled pixels of a reference photo. – danijar Sep 03 '14 at 21:29
  • @Seb How would I determine the set of pixels I want to consider? I explained the context in the comment above. – danijar Sep 03 '14 at 21:30
  • If the spatial distance between pixels from A and B is element of your cost function, you can use a threshold on distance. If it is not, you can only consider pixels couples under a certain cost. The solution may be sub-optimal in some cases. – Seb Sep 03 '14 at 21:35
  • @Seb The spatial difference isn't part of the cost, just the difference in the red, green and blue channels. I think the threshold would depend on the overall difference of the two images. Two similar images need low threshold so that not all costs can pass while very different images need high threshold so that not all option get discarded. Isn't there another algorithm to solve that problem? When I wouldn't have three dimensions but just luminance, I'd just sort pixels of both images by that. – danijar Sep 03 '14 at 21:44
  • Ok I'm starting to get it ;) it mean a flipped picture will admit a zero cost solution regardless the fact the pixels can be really far from each others. My second option will preserve the optimal solution in the solutions space. If you want a suggestion you should use HSI representation to calculate costs in your matrix – Seb Sep 03 '14 at 23:00
  • @Seb Thanks for the advice of using HSI, that sounds reasonable. However, the main problem is that I can't store the whole matrix in memory and not even on disk. – danijar Sep 04 '14 at 05:59

1 Answers1

1

The modifications made by the Hungarian algorithm to the cost matrix are to add/subtract constants from whole rows/columns. Instead of storing the whole matrix, you can store just the row/column deltas (i.e., the potentials) and, when retrieving a matrix element, add the appropriate one of each to the base cost (recomputed as needed). I expect that the running time still would be prohibitive, however.

David Eisenstat
  • 64,237
  • 7
  • 60
  • 120
  • Even with storing just the deltas, the matrix would be 1920 times 1080 squared. This is 3.9TB even if I use only one byte per entry. Or do I misunderstand your answer? – danijar Sep 04 '14 at 05:57
  • @danijar You would need 1920 \* 1080 \* 2 = 4147200 entries, one for each row and each column of the cost matrix. The base costs are computed on demand and thus not stored. – David Eisenstat Sep 04 '14 at 06:13
  • So you mean I don't store costs but just which value I subtracted from each row or column, right? – danijar Sep 04 '14 at 06:20
  • @danijar Right. The initial value of each entry in the big matrix is computed from the pair of colors. The updates are rank-1. – David Eisenstat Sep 04 '14 at 06:21
  • Couldn't I just store the minimum value of each row and each column right ahead then? Then I just had to compare which one is smaller and choose that. – danijar Sep 04 '14 at 06:38
  • @danijar Perhaps, but I would have expected the updates to be too complicated for a simpler scheme to work. I haven't implemented the Hungarian algorithm recently, though. – David Eisenstat Sep 04 '14 at 07:04