Merging arrays that hold CSV data into one object array

Question

I've taken influence from this post to store CSV files in individual arrays, and it works. I'm looking to add about 8 more arrays that each hold the contents of a CSV file.

But how can I merge these arrays into one array of objects? Having an array of objects is critical for my subsequent application of these arrays. What I have so far is below, but I'm not sure if it is correct or not?

Any help would be appreciated.

var dataCSV;
var dataCSV2;

var collection = {dataCSV, dataCSV2};

d3.csv("data/csv1.csv", function(data) {
    data.forEach(function(d) {
        d.word = d.word;
        d.frequency = +d.frequency;
})
    dataCSV = data;
    console.log(dataCSV);
});

d3.csv("data/csv2.csv", function(data) {
    data.forEach(function(d) {
        d.word = d.word;
        d.frequency = +d.frequency;
    })
    dataCSV2 = data;
    console.log(dataCSV2);
});

function merge(data1, data2) {
    collection = {data1, data2};
    console.log(collection);
}

merge(dataCSV, dataCSV2);

Ideally, I'd like the output of collection to be:

collection {
    dataCSV {
        word: ..., frequency: ...,
        word: ..., frequency: ...
    },
    dataCSV2 {
        word: ..., frequency: ...,
        word: ..., frequency: ...
    }
 }

Are you bound to using an old version of d3/d3-queue? You could make your life easier if you upgraded to v5. — altocumulus, Sep 13 '18 at 15:25
I had no idea V5 was available... I'll look into it. What a solution for what i need look like in V5? — dumbquestionsbydumbpeople, Sep 13 '18 at 15:27
Have a look at my [answer](https://stackoverflow.com/a/49240175/4235784) to [*"Load multiple files using the d3-fetch module"*](/q/49239474). `d3-fetch` module replaces the now deprecated module `d3-queue` in v5. — altocumulus, Sep 13 '18 at 16:12
@altocumulus This is confusing, i may need to post another question — dumbquestionsbydumbpeople, Sep 13 '18 at 19:19
The solution from my answer I linked to is basically the same as the one proposed by @OriDrori's answer. However, v5 makes your life easier as there is no need to create new Promises yourself as d3 makes use of the Fetch API which will return Promises on its own, i.e. `d3.csv` returns a Promise. This is what you can see in my answer. Same technique, just shorter. — altocumulus, Sep 13 '18 at 19:40

Ori Drori · Accepted Answer · 2018-09-13T15:11:37.387

1

Instead of repeating yourself, create a single method that gets the data, formats it, and returns a promise. Then collect the results of several promises using Promise.all(). Then you can collect both arrays in an object.

Example (not tested):

const getCsv = (url) => new Promise((resolve) => {
  d3.csv(url, function(data) {
    data.forEach(function(d) { // you can use a Array.map() instead for a cleaner solution
      d.word = d.word;
      d.frequency = +d.frequency;
    })

    resolve(data);
  });
});

Promise.all([ // load both csvs, and wait for the results of both
  getCsv('data/csv1.csv'),
  getCsv('data/csv1.csv')
])
.then(([dataCSV, dataCSV]) => { 
  const collection = { dataCSV, dataCSV2 };
  console.log(collection );
});

edited Sep 13 '18 at 15:11

answered Sep 13 '18 at 13:44

Ori Drori

183,571
29
224
209

Hey man, just want to say this works great. But is it possible to have `collection` in the global scope when using `promises`? – dumbquestionsbydumbpeople Sep 13 '18 at 19:19
You can assign the data inside the function to a global variable (like you do in your original code), but it's not sure that it'll we'll there when you'll want it. In your original code, add a `console.log(dataCSV1)` after the `d3.csv();`. The result will always be `undefined`. Because of the async call, code after the call is executed before the result arrives, and the only way to know when it's done is using a callback, promise, generator or async/await (which uses promises). My advice to you is to learn how javascript works. – Ori Drori Sep 13 '18 at 19:25

Merging arrays that hold CSV data into one object array

1 Answers1