How to find multiple duplicates in a PodioItemCollection response?

Question

I have an array with some students in who have enrolled on a course. There are multiple duplicates and should be only one student per course.

Example array:

'item_id'=> 1, 'student'=> 'Bob', 'course'=> 'Learn Piano', 'address'=>''
'item_id'=> 2, 'student'=> 'Sam', 'course'=> 'Learn Piano', 'address'=> 'foo street'
'item_id'=> 3, 'student'=> 'Bob', 'course'=> 'Learn Guitar', 'address'=>''
'item_id'=> 4, 'student'=> 'Sam', 'course'=> 'Learn Piano', 'address'=>''
'item_id'=> 5, 'student'=> 'Bob', 'course'=> 'Learn Guitar', 'address'=> 'bla bla street'
'item_id'=> 6, 'student'=> 'Sam', 'course'=> 'Learn Piano', 'address'=>''
'item_id'=> 7, 'student'=> 'John', 'course'=> 'Learn Guitar', 'address'=>''

Data is accessed via API (otherwise this whole thing would be a simple SQL query!).

The raw data looks like below:

object(PodioItemCollection)#287 (5) { ["filtered"]=> int(45639) ["total"]=> int(45639) ["items"]=> NULL ["__items":"PodioCollection":private]=> array(10) { [0]=> object(PodioItem)#3 (5) { ["__attributes":"PodioObject":private]=> array(16) { ["item_id"]=> int(319357433) ["external_id"]=> NULL ["title"]=> string(12) "Foo Bar" ["link"]=> string(71) "https://podio.com/foo/enrolments/apps/applications/items/123" ["rights"]=> array(11) ...

The challenge is that I can't just use array_unique or similar because i need to:

1. Find all the duplicates for a student + course
2. Evaluate the found duplicates against each other and retain the item with the most amount of supplementary information (or merge them)
3. Obtain the un-needed "item_id" for the duplicates and use the API to delete the items.

Further constraints:

• I have no control over the API.
• There are 44,000 records
• There could be as many as 100 duplicates per person + course
• The API returns a nested hierarchy of objects, so 44,000 records uses 27GB of RAM (the server has 144GB to play with) and yes php_memory limit is set to a ridiculous level!!! This is a single project and measures will be taken to correct the server variables afterwards.
• Because of the large RAM usage things such as array_intersect are going to be a less popular choice

The final output should be:

    'item_id'=> 1, 'student'=> 'Bob', 'course'=> 'Learn Piano', 'address'=>''
    'item_id'=> 2, 'student'=> 'Sam', 'course'=> 'Learn Piano', 'address'=> 'foo street'
    'item_id'=> 5, 'student'=> 'Bob', 'course'=> 'Learn Guitar', 'address'=> 'bla bla street'
    'item_id'=> 7, 'student'=> 'John', 'course'=> 'Learn Guitar', 'address'=>''

But i also need access to 'item_id's 3,4,6 so i can call a delete routine via an API.

Any ideas how to tackle this multi-duplicate mess?

Answer 1

It looks like you're trying to fix XY problem by parsing all the items at the same time, when actually you just need to read about Podio API and how to work with PodioObjects using their API.

So instead of traversing through the whole array, traverse through the object (PodioItemCollection in this case) by sorting it via key field (such as student), e.g.:

PodioItem::filter(31060, array('limit' => 20, 'offset' => 20, 'sort_by' => 'student'));

Then (assuming the sort criteria) you know that if there are some duplicates (student + course), they're in consecutive order. So as far you've the same student name in the next row, treat it as duplicate and ignore the other rows (unless you want to process them as part of the current student object), then go to the next item.

In this case, you can avoid to work with the all records at the same, so you won't have any memory issues.

If the memory issue would be still a problem, use NOSQL solution such as memcached or redis, so store the parsed items in the memory, free-up the variables from PHP, go to the next one, and at the end check the consistency (e.g. count the items). Consider also disabling any extra PHP extensions which you're having and could slow down the processing (such as xdebug and similar).

See also:

• Working with items - Podio API
• Aggregating responses for a PodioItem:filter() call at SO

Answer 2

Following function will do the job for you:

$apiData = array(
   array('item_id'=> 1, 'student'=> 'Bob', 'course'=> 'Learn Piano', 'address'=>''),
   array('item_id'=> 2, 'student'=> 'Sam', 'course'=> 'Learn Piano', 'address'=> 'foo street'),
   array('item_id'=> 3, 'student'=> 'Bob', 'course'=> 'Learn Guitar', 'address'=>''),
   array('item_id'=> 4, 'student'=> 'Sam', 'course'=> 'Learn Piano', 'address'=>''),
   array('item_id'=> 5, 'student'=> 'Bob', 'course'=> 'Learn Guitar', 'address'=> 'bla bla street'),
   array('item_id'=> 6, 'student'=> 'Sam', 'course'=> 'Learn Piano', 'address'=>''),
   array('item_id'=> 7, 'student'=> 'John', 'course'=> 'Learn Guitar', 'address'=>'')
);

function resolveDuplicate($apiData = null)
{
  if(!$apiData) return false;

  foreach ($apiData as $key => $arr) {
    $key = $arr['student'] . ':' . $arr['course'];
    if(!$newArr[$key]['address']){
       if($newArr[$key]) $itemIds[] = $newArr[$key]['item_id'];
       $newArr[$key] = $arr;
    }
    else{
       $itemIds[] = $arr['item_id'];
    }
  }

  if($newArr){
     foreach ($newArr as $value) {
       $finalArr[] = $value;
     }
  }

  $result['student']    = $finalArr;
  $result['duplicates'] = $itemIds;
  return $result;
}

$res = resolveDuplicate($apiData);
echo '<pre>';
print_r($res);

Output

Array
(
    [student] => Array
        (
            [0] => Array
                (
                    [item_id] => 1
                    [student] => Bob
                    [course] => Learn Piano
                    [address] => 
                )

            [1] => Array
                (
                    [item_id] => 2
                    [student] => Sam
                    [course] => Learn Piano
                    [address] => foo street
                )

            [2] => Array
                (
                    [item_id] => 5
                    [student] => Bob
                    [course] => Learn Guitar
                    [address] => bla bla street
                )

            [3] => Array
                (
                    [item_id] => 7
                    [student] => John
                    [course] => Learn Guitar
                    [address] => 
                )

        )

    [duplicates] => Array
        (
            [0] => 4
            [1] => 3
            [2] => 6
        )

)