using createWriteStream with xmlbuilder consume a lot of ram

Question

We need to create a XML file with some items in there, so we are using xmlbuilder module for this.

The array have 50k of products, the memory usage processing the file is 200mb of ram, but at the moment of creating the file with xmlbuilder, the memory usage go to more that 1gb of ram, right now we can handle but the server will run out of memory in some moments.

here is the functions that i'm using it, i need to find a way of how to process the files without consuming a lot of ram. the time to generate the file is not a problem, we can wait, but the ram usage is a BIG PROBLEM.

const fs = require("fs");

const products = [...new Array(50000)].map((item, i) => ({
  _id: i,
  name: "productName",
  description: "productDescription",
  currency: "USD",
  salePrice: 100,
  img: ["https://productLink.com"],
  productAvailability: true,
  slug: "product-slug",
}));

let items = [];

for await (const product of products) {
  const item = {
    id: product._id,
    title: product.name,
    description: product.description,
    image: product.img[0],
    availability: product.productAvailability ? "in stock" : "out of stock",
    price: `${product.salePrice} ${product.currency}`,
  };

  items = [...items, item];
}

let obj = {
  rss: {
    channel: {
      item: items,
    },
  },
};

const root = xmlbuilder.create(obj, { encoding: "utf-8" });
const writeStream = fs.createWriteStream("file.xml");
const writer = xmlbuilder.streamWriter(writeStream);
const xml = root.end(writer);

(obj = {}), (items = []), (xmldoc = ""), (products = []); // Declare all to empty to reduce node cache

writeStream.end();
writeStream
  .on("error", (error) => {
    throw error;
  })

  .on("finish", async () => {
    console.log("done");
  });

If have any idea of how to create this file, will be nice, thanks

using createWriteStream with xmlbuilder consume a lot of ram

0 Answers0