0

I am sending a get request and receiving a JSON object with an array of around 50000 elements. I need to loop through the 50000 array elements and group together all elements with the same username then add that data to another array in my code so it can be displayed in a UI-grid.

My first solution was to loop through the received 50000 elements and check if they were in the ui-grid array which required me to loop through the ui-grid array, so for each element in the array from the GET request I had to loop through the ui-grid array. If I found a element in the ui-grid array with the same username then I just added the integer values together i.e.

(uiGridArray[i].views += GetRequestArray[j].views).

My second attempt I tried to use and array of key values. The usernames were the key values. I created a temp array to store these. I would loop through the GET request array as before but instead of looping through the ui-grid array to check for usernames I would first check if an element with the value tempArray[GetRequestArray[i].username] existed if not I would create one if one did exist I would just go directly to that element using the key value i.e.

(tempArray[GetRequestArray[i].username].views += GetRequestArray[i].views)

Finally I would loop through the tempArray once and add it to the ui-grid array.

GetRequestArray -> tempKeyNameArray -> uiGridArray

callAPI.get(options).then(function(value) {
          let dataLength = uiGrid.data.length;

          // formates data from the graylog query into an object and stores the formated
          // objects in a hash table to increase performace while formating the data
          for (let i = 0; i < value.length; i++) {
              $scope.addMessageToHashTableArray(value[i]);
          }

          // add clinical hash table to ui-grid data array
          for (let key in $scope.tempArray) {
            uiGrid.data.push($scope.tempArray[key]);
          }
      })


$scope.addMessageToHashTableArray = function(value) {

      let row = angular.copy($scope.templateRow);
      row.username = value.message.username;
      row.email = value.message.email;
      row.group = $scope.getGroupName(value.message);

      let key = row.username;

      // if a row already exists with the current username and group
      // then update that row with the current values
      if ($scope.tempArray.hasOwnProperty(key)) {
        $scope.tempArray[key].data_view += row.data_view;
        $scope.tempArray[key].records += row.records;
        $scope.tempArray[key].crosstabs += row.crosstabs;
        $scope.tempArray[key].explorer += row.explorer;
        $scope.tempArray[key].bookmarks += row.bookmarks;
        $scope.tempArray[key].reports += row.reports;
        $scope.tempArray[key].total_views += row.total_views;

      }
      // if the row does not exist then create one
      else {
        $scope.tempArray[key] = angular.copy(row);
      }
  }

From my tests the key pair value array seemed to be 4 seconds faster but the process still takes 30-40 seconds. Is there a way I can optimize this? I would like the process to take 10 seconds i.e. reduce the load time by 20 - 30 seconds

  • Except for the angular.copy part, it should take under 1s. Are you sure you are meassuring the time correctly (30s pass between let dataLength and for (let key)? – juvian Jun 26 '19 at 15:46
  • this is the code that I am assuming is taking up the most time `for (let i = 0; i < value.length; i++) { $scope.addMessageToHashTableArray(value[i]); }` – user8685657 Jun 27 '19 at 08:45

0 Answers0