How to combine large list of lists (of lists again) under certain condition

Question

I am trying to solve a problem coming from the area of biology in which I have to combine local sub-optimal solutions from each big element such that each sub particle is unique. The problem is that the possibilities could scale up to +4.000 local sub-optimal solutions and to +30.000 for the elements. Cartesian product is not an option as combining the lists is a n*m*p*... problem, impossible without an algorithm beyond itertools.

The general schema is:

[
  [ [a,b,c],[d,e,a],[f], ...],
  [ [f,e,t],[a,b,t],[q], ...],
  [ [a,e,f],[],[p], ... up to 4.000],
  ... up to 30.000
]

[ [a,b,c],[d,e,a],[f],.....], -> group of sub-optimal solutions for elem. #1

I want to find as fast as possible

• First: one solution, which means a combination of one sub-optimal solution for each element (could include blank lists) such that there are no duplicates. For example [[a,b,c],[f,e,t][p]].

• Second: all the compatible solutions.

I know is an open questions, however I need some guidance or general algorith to confront this problem, I can investigate further if I have something to start with.

I am using python for the rest of the lab work, however I am open to other languages.

We can start from a basic solver that handle less possibilities in terms of total sub-optimal and number of list of lists.

Best.

EDIT 1

A very-short-real example:

[[[1,2,3][1,2,4],[1,2,5],[5,8]],
[[1,3][7,8],[6,1]],
[[]],
[[9,10][7,5],[6,9],[6,10]]]

OPTIMAL SOLUTION (FROM ROW #):

#1 [1,2,3]
#2 [7,8]
#3 [9,10]

Output: [[1,2,3],[7,8],[9,10]]

Can see here https://pastebin.com/qq4k2FdW

Answer 1

Here are a couple of algorithms. You can do a brute force search over all possible combinations with itertools very simply:

from itertools import product, chain

def get_compatible_solutions(subsolutions):
    for sol in product(*subsolutions):
        if len(set(chain.from_iterable(sol))) == sum(map(len, sol)):
            yield sol

# Test
example = [
    [[1, 2, 3], [1, 2, 4], [1, 2, 5], [5, 8]],
    [[1, 3], [7, 8], [6, 1]],
    [[]],
    [[9, 10], [7, 5], [6, 9], [6, 10]]
]

# Get one solution
print(next(get_compatible_solutions(example)))
# ([1, 2, 3], [7, 8], [], [9, 10])

# Get all solutions
print(*get_compatible_solutions(example), sep='\n')
# ([1, 2, 3], [7, 8], [], [9, 10])
# ([1, 2, 3], [7, 8], [], [6, 9])
# ([1, 2, 3], [7, 8], [], [6, 10])
# ([1, 2, 4], [7, 8], [], [9, 10])
# ([1, 2, 4], [7, 8], [], [6, 9])
# ([1, 2, 4], [7, 8], [], [6, 10])
# ([1, 2, 5], [7, 8], [], [9, 10])
# ([1, 2, 5], [7, 8], [], [6, 9])
# ([1, 2, 5], [7, 8], [], [6, 10])
# ([5, 8], [1, 3], [], [9, 10])
# ([5, 8], [1, 3], [], [6, 9])
# ([5, 8], [1, 3], [], [6, 10])
# ([5, 8], [6, 1], [], [9, 10])

Another possibility is to do a recursive search one row at a time. This will explore less candidate solutions than the Cartesian product, as once a suboptimal solution is excluded from a search path no combinations including it will be processed.

def get_compatible_solutions(subsolutions):
    current = [None] * len(subsolutions)
    seen = set()
    yield from _get_compatible_solutions_rec(subsolutions, current, 0, seen)

def _get_compatible_solutions_rec(subsolutions, current, i, seen):
    if i >= len(subsolutions):
        yield tuple(current)
    else:
        for subsol in subsolutions[i]:
            if any(s in seen for s in subsol):
                continue
            seen.update(subsol)
            current[i] = subsol
            yield from _get_compatible_solutions_rec(subsolutions, current, i + 1, seen)
            seen.difference_update(subsol)