How can I compare spliced multidimensional arrays (with huge amount of data)?

Question

I have a huge array $properties with about 500.000 items:

  array(470000) {
    ["12345"]=>
    array(5) {
      ["dateTime"]=>
      string(19) "2016-10-12 19:46:25"
      ["fileName"]=>
      string(46) "monkey.jpg"
      ["path"]=>
      string(149) "Volumes/animals/monkey.jpg"
      ["size"]=>
      string(7) "2650752"
    }
    ["678790"]=>
    array(5) {
      ["dateTime"]=>
      string(19) "2016-10-12 14:39:43"
      ["fileName"]=>
      string(45) "elephant.jpg"
      ["path"]=>
      string(171) "Volumes/animals/elephant.jpg"
      ["size"]=>
      string(7) "2306688"
    }

... and so on.

So to make performance better, I spliced it into parts:

$splice_size = 10000;
        $count_arr = (count($properties)/$splice_size)-1;


        For($i=0; $i<$count_arr; $i++){
            $res[] = array_splice($properties, 0,$splice_size); 
        }
        $res[] = array_splice($properties, 0,count($properties));

Now my array looks like this:

array(4) {
  [0]=>
  array(10000) {
    ["12345"]=>
    array(5) {
      ["dateTime"]=>
      string(19) "2016-10-12 19:46:25"
      ["fileName"]=>
      string(46) "monkey.jpg"
      ["path"]=>
      string(149) "Volumes/animals/monkey.jpg"
      ["size"]=>
      string(7) "2650752"
    }
    ["678790"]=>
    array(5) {
      ["dateTime"]=>
      string(19) "2016-10-12 14:39:43"
      ["fileName"]=>
      string(45) "elephant.jpg"
      ["path"]=>
      string(171) "Volumes/animals/elephant.jpg"
      ["size"]=>
      string(7) "2306688"
    }

   ... and so on.
   }
  [1]=>....
  and so on....
}

I want now compare two of these arrays:

  function array_diff_assoc_recursive($array1, $array2)
                {
                    foreach($array1 as $key => $value)
                    {
                        if(is_array($value))
                        {
                            if(!isset($array2[$key]))
                            {
                                $difference[$key] = $value;
                            }
                            elseif(!is_array($array2[$key]))
                            {
                                $difference[$key] = $value;
                            }
                            else
                            {
                                $new_diff = array_diff_assoc_recursive($value, $array2[$key]);
                                if($new_diff != FALSE)
                                {
                                    $difference[$key] = $new_diff;
                                }
                            }
                        }
                        elseif(!isset($array2[$key]) || $array2[$key] != $value)
                        {
                            $difference[$key] = $value;
                        }
                    }
                    return !isset($difference) ? 0 : $difference;
                }


                echo "<pre>";
                print_r(array_diff_assoc_recursive($new, $res));
                echo "</pre>";

But the system crashes. Too much data. So my question is, their must be an advantage of splicing the array (like making chunks) that I still do not get, to get better performance.

if there is a mistake for example. In one array it is `monkey` and in the other array it is `munkey`. Should be able to find it. Works well with smaller arrays. But big ones make problems — peace_love, May 21 '18 at 12:20
Using [`array_chunk`](http://php.net/manual/en/function.array-chunk.php) will make your code shorter, but I'm not convinced it's necessary. You end up duplicating the array into chunks which actually doubles the memory requirement. What I'd do is just a single loop through the already existing arrays — apokryfos, May 21 '18 at 12:27
Why would you have the same values at more than one offset in the array? If you have "monkey" and "munkey" how do you know which one is wrong? Your explanation makes no sense. — symcbean, May 21 '18 at 12:32
@symcbean I'm assuming they have a 2nd array which has the same keys and might have slightly different values to some of the matching keys. That's the only way this makes sense to me — apokryfos, May 21 '18 at 12:34

apokryfos · Accepted Answer · 2018-05-23T09:29:57.100

1

If I were you I would just do:

$different = [];
$missingFrom2 = [];

foreach ($array1 as $key => $value) {
    if (!isset($array2[$key])) { $missingFrom2[] = $key; }
    if ($array2[$key] != $value) { $different[] = $key; }
}
$missingFrom1 = array_diff(array_keys($array2), array_keys($array1));

$different will be all keys which are different.

What you're doing seems a bit over-engineered for not particular benefit

Examples: http://sandbox.onlinephpfunctions.com/code/7ff02f562e0591e8afb45ea51799b847fbc4063b http://sandbox.onlinephpfunctions.com/code/402926605ba8a195d2dfc667be146654117cd078

edited May 23 '18 at 09:29

answered May 21 '18 at 12:33

apokryfos

38,771
9
70
114

tested your code but unfortunately my result is `array(0) { }` even if in array1 I wrote `monkey` and in array2 I wrote `monkei`. – peace_love May 23 '18 at 09:25
@Jarla http://sandbox.onlinephpfunctions.com/code/7ff02f562e0591e8afb45ea51799b847fbc4063b can you share what you've tried and didn't work? – apokryfos May 23 '18 at 09:29
http://sandbox.onlinephpfunctions.com/code/3f3ce1faf28ba539fca6d75ecbd752a481c0ce1e – peace_love May 23 '18 at 09:44
Is that the already chunked array? This answer works if you don't chunk the array – apokryfos May 23 '18 at 09:45
Ah, ok, this is the chunked array – peace_love May 23 '18 at 09:48
Need solution for chunked array – peace_love May 23 '18 at 09:50
1

I just wanted to point out in this answer that you don't need to chunk your array. It's unnecessary complexity and consumes more memory. – apokryfos May 23 '18 at 10:41
If I do not make the array into parts, I cannot compare it because it is too big. System crashes – peace_love May 23 '18 at 12:11
I will try your solution – peace_love May 23 '18 at 12:28

How can I compare spliced multidimensional arrays (with huge amount of data)?

1 Answers1