Find intersection group of sorted integer arrays

Question

Let's we have some integer short sorted arrays and we need to find intersection equal or more then predefined constant. Here is code and it demonstrates what i want to do better then i can explain it in words. The problem is SPEED. My code is working very slow. It takes about 15 sec on 2000 elements array(on my slow machine). Ofcourse i can implement my own intersection method and parallize code but it give a very limited improvement. Execution time growing as N^2 or something and already for 500k arrays it takes a very very long time. So how can i rewrite algorithm for better perfomance? I am not limited c# language maybe CPU or GPU has good special instructions for such job.

Example:

Input:
1,3,7,8
2,3,8,10
3,10,11,12,13,14

minSupport = 1

Output:

1 and 2: 2, 8
1 and 3: 3
2 and 3: 3, 10

---

    var minSupport = 2;
    var random = new Random(DateTime.Now.Millisecond);

    // Numbers is each array are unique
    var sortedArrays = Enumerable.Range(0,2000)
    .Select(x => Enumerable.Range(0,30).Select(t => random.Next(1000)).Distinct()
    .ToList()).ToList();
    var result = new List<int[]>();
    var resultIntersection = new List<List<int>>();

    foreach (var array in sortedArrays)
    {
        array.Sort();
    }

    var sw = Stopwatch.StartNew();

    //****MAIN PART*****//

    for (int i = 0; i < sortedArrays.Count-1; i++)
    {
        for (int j = i+1; j < sortedArrays.Count; j++)
        {
            var intersect = sortedArrays[i].Intersect(sortedArrays[j]).ToList();
            if(intersect.Count()>=minSupport)
            {
                result.Add( new []{i,j});
                resultIntersection.Add(intersect);
            }
        }
    }

    //*****************//

    sw.Stop();

    Console.WriteLine(sw.Elapsed);

EDIT:

Now it takes about 9 sec vs 15 sec with old algorithm on 2000 elements. Well...ofcourse it is not fast enough.

//****MAIN PART*****//

    // This number(max value which array can contains) is known
    var maxValue = 1000;

    var reverseIndexDict = new Dictionary<int,List<int>>();

    for (int i = 0; i < maxValue; i++)
    {
        reverseIndexDict[i] = new List<int>();
    }

    for (int i = 0; i < sortedArrays.Count; i++)
    {
        for (int j = 0; j < sortedArrays[i].Count; j++)
        {
            reverseIndexDict[sortedArrays[i][j]].Add(i);
        }
    }

    var tempArr = new List<int>();
    for (int i = 0; i < sortedArrays.Count; i++)
    {
        tempArr.Clear();
        for (int j = 0; j < sortedArrays[i].Count; j++)
        {
            tempArr.AddRange(reverseIndexDict[j]);
        }

        result.AddRange(tempArr.GroupBy(x => x).Where(x => x.Count()>=minSupport).Select(x => new[]{i,x.Key}).ToList());

    }

    result = result.Where(x => x[0]!=x[1]).ToList();


    for (int i = 0; i < result.Count; i++)
    {
        resultIntersection.Add(sortedArrays[result[i][0]].Intersect(sortedArrays[result[i][1]]).ToList());
    }



    //*****************//

EDIT:

Some improvent.

//****MAIN PART*****//

    // This number(max value which array can contains) is known
    var maxValue = 1000;

    var reverseIndexDict = new List<int>[maxValue];

    for (int i = 0; i < maxValue; i++)
    {
        reverseIndexDict[i] = new List<int>();
    }

    for (int i = 0; i < sortedArrays.Count; i++)
    {
        for (int j = 0; j < sortedArrays[i].Count; j++)
        {
            reverseIndexDict[sortedArrays[i][j]].Add(i);
        }
    }



    for (int i = 0; i < sortedArrays.Count; i++)
    {
        var tempArr = new Dictionary<int, List<int>>();

        for (int j = 0; j < sortedArrays[i].Count; j++)
        {
            var sortedArraysij = sortedArrays[i][j];


            for (int k = 0; k < reverseIndexDict[sortedArraysij].Count; k++)
            {
                if(!tempArr.ContainsKey(reverseIndexDict[sortedArraysij][k]))
                {
                    tempArr[reverseIndexDict[sortedArraysij][k]] = new[]{sortedArraysij}.ToList();
                }
                else
                {
                   tempArr[reverseIndexDict[sortedArraysij][k]].Add(sortedArrays[i][j]);
                }

            }
        }


        for (int j = 0; j < reverseIndexDict.Length; j++)
        {
            if(reverseIndexDict[j].Count>=minSupport)
            {
                result.Add(new[]{i,j});
                resultIntersection.Add(reverseIndexDict[j]);
            }
        }

    }

    // and here we are filtering collections

    //*****************//

Answer 1

There are two solutions:

1. Let us suppose you have 3 sorted arrays and you have to find the intersection between them. Traverse the first array and run a binary search on the rest of the two arrays for the element in first array. If the respective binary search on two list gave positive, then increment the counter of intersection.

   result = List
   for element in Array1:
       status1 = binarySearch(element, Array2)
       status2 = binarySearch(element, Array2)
       status = status & status
       if status == True:
           count++
           if count == MAX_INTERSECTION:
               result.append(element)
               break
   

   Time Complexity : N * M * Log(N),
   where,
   N = Number of element in the array
   M = Number of arrays

2. This solution works only if the number in the arrays are positive integers. Calculate the maximum and the minimum number out of the total elements in all the sorted arrays. As it is sorted, we can determine it by surveying the start and end element of the sorted arrays given. Let the greatest number be max and the lowest number be min. Create an array of size max - min and fill it with zero. Let us suppose you have 3 Arrays, now start traversing the first array and and go to the respective index and increment the value in the previously created array. As mentioned below:

   element is 5 in Array 1, the New_array[5]+=1
   

   Traverse all the three sorted list and perform the operation mentioned above. At the end traverse the new_array and look for value equal to 3, these indexes are the intersection result.
   
   Time Complexity : O(N) + O(N) + .. = O(N)
   Space Complexity : O(maximum_element - minimum_element)
   where,
   N = number of elements in the array.