1

My deployed app has a function that makes a call to a database whenever the app.get("/") route is called, or when the user adds or deletes a file to the database. My service is being limited by my web host when this function is called, because it exceeds my memory allotment, causing a failure in one of my npm packages (sharp) not having the processor threads available to process an image.

I've studied the metrics provided by my web host, but aside from telling me a fault has occurred, it does not show how large the memory usage is, or where. Using the Chrome DevTools task manager I can see that the live javascript memory does in fact gain slowly over the runtime of the app, but not more than other webpages I've looked at.

I have no mentors in the field, and after some exhaustive research on memory leaks, what causes them, and how to troubleshoot them, I'm left lost and confused, but I do believe I've narrowed it down to the culprit of the problem. I just don't know what to do from here.

Here is the function in question:

let artWorks = [];

async function listFiles() {
  artWorks = [];   //clears the array if it has content
  const [files] = await storage.bucket(bucketName).getFiles();    //call for url names of files in DB
  files.forEach(file => {   //loops through all files and pushes them to the array, separating the URL string into usable data
    function splitStr(str) {
      var string = str.split("/");
      artWorks.push(
        {
          imgCategory: string[0],
          subCat: string[1].replace(/-/g, ' '),
          imgSrc: bucket + file.name,
          thumbnail: thumbBucket + file.name.split('.')[0] + "-thumb.jpg",
          alt: string[2]
        });
    }
    var str = file.name;
    splitStr(str);
  });
  console.log("files listed");
}

The artWorks array is the crux of this whole app. Every route uses the contents of this array to populate the ejs template pages. Once a user adds or deletes a file, the listFiles() function is called again to repopulate the array.

I have not noticed any slow down in performance, and if it wasn't for my web host capping my memory, I never would have known there was a leak. Please offer me guidance, hopefully not getting too involved in terminology, but in terms a new developer like myself can understand.

PsiKai
  • 1,803
  • 1
  • 5
  • 19
  • If `artWorks` is used everywhere, any code that uses it can be the memory leak by holding on an "old" array. Make a heap snapshot an inspect what exactly is retaining the objects. – Bergi Oct 11 '20 at 19:47
  • Chrome devtools measure client-side memory usage, that has nothing to do with server-side code. How many files are there in the array? – Guy Incognito Oct 11 '20 at 19:48
  • @GuyIncognito There are a variable number of files, but on average there are ~100. Each file represents an object in the array with the key:value pairs shown in the function. – PsiKai Oct 11 '20 at 19:52
  • Not a memory leak, but having your splitStr inside your forEach will put more strain on the memory manager. Your basically creating a new function for each file when 1 would do. – Keith Oct 11 '20 at 19:58
  • @Bergi Analyzing the heap snapshot is part of what I'm confused on. In this screengrab of the snapshot results, maybe you can tell me where to look? I've expanded the tabs but I don't recognize any of the processes. https://imgur.com/a/9uJuMZg – PsiKai Oct 11 '20 at 19:59
  • @Kieth Thank you for that, it may not solve my leak issue, but it certainly solves some potential bloat issues. – PsiKai Oct 11 '20 at 20:04
  • 2
    Again, that screenshot shows your browser's memory usage. It has nothing to do with the Node.js code. You need to take the snapshot on the server. – Guy Incognito Oct 11 '20 at 20:04
  • @GuyIncognito Thank you for clarifying that. Can you recommend a good program to use for that? I keep seeing outdated npm packages in my search results, such as heapdump which appears to be deprecated. – PsiKai Oct 11 '20 at 20:11
  • I haven't needed to do that myself, sorry. – Guy Incognito Oct 11 '20 at 20:13
  • @PsiKai what is the reason to declare the function `splitStr` inside the for each? you basically declare it per each item you are iterating over. declare it outside the scope of the forEach and use it as a callback for each item. – Yakir Fitousi Oct 11 '20 at 20:42

1 Answers1

-1

Notice you are declaring the splitStr function on each iteration. I wrote you a small suggestion for a cleaner and easier way of reading this code snippet.

Hope you'd like it.

 const filesMapper = ({name}) => {
    const string = name.split('/')
    return { 
       imgCategory: string[0],
       subCat: string[1].replace(/-/g, ' '),
       imgSrc: bucket + name,
       thumbnail: thumbBucket + name.split('.')[0] + "-thumb.jpg",
       alt: string[2]
    }
}

async function listFiles() {
    const [files] = await storage.bucket(bucketName).getFiles(); 

    return files.map(filesMapper)
  });
}
Yakir Fitousi
  • 537
  • 3
  • 12