Loading local data for visualization using D3.js

Question

I am working on a project which requires me to visualize a rather complicated type of data (see this older question). In short, I have a large chunk of data which I can export to JSON, CSV or some other arbitrary flat format although I prefer to avoid XML if possible (see the linked question above for in-detail explanation of the underlying data).

I have started working on a visualization using D3, the layout I wrote seems to work OK so far when I test it with some very simple data that I hardcode in the Javascript as an array. The tutorials I read on data binding in D3 have been a bit confusing in the sense that some use JSON and some use TXT/CSV format while some others use hardcoded arrays/matrices.

In the case of JSON, I watched a tutorial where the narrator firmly advises to host the JSON file on a webserver and get it using a HTTP request instead of a local file read. I realize that this is due to cross domain request limitations, which I believe I have to work-around somehow. At this point I am not sure how to proceed since:

1. The D3-powered visualization will be on a series of HTML reports which are created as results of an analysis tool I wrote. The analysis is done on the users computer, and the HTML reports are also created locally on the client-side.

2. The intended users are most definitely not tech-savvy, so it is not an option to instruct them to run an webserver on their computer to be able to serve JSON or any other type or resource via localhost

For the record, I have tried running the python SimpleHTTPServer module to try it out, and again everything works fine. I then tried to hard-code the data in the generated HTML reports, then call on the JSON object from my script which uses D3,

//d3.json("mydata.json", function(json){
d3.json(myjson, function(json){
    nodeData = json.elements;
....
}

which fails since in that case I end up sending in a JSON object while D3.js is expecting a URL.

What can I do to avoid/solve this problem?

Answer 1

To load data local files without using a server, append a script tag to the body with the source data assigned to a variable or object element that you are expecting. In the example below, I load data for a certain country from a [COUNTRY_ISO3CODE].js file. Specifically, an Afghanistan data file might have the content in the format:

data[AFG] = {"name": "Afghanistan", "estimate": 9.003, ... }

The calling file will have something like:

if (!data[iso3]) { //only load the needed data-script as needed

    // if using jQuery
    $('body').append("<script type='text/javascript' src='jdb/"+ iso3 +".js'></script>")

    //If not using jQuery
    /*var script   = document.createElement("script");
    script.type  = "text/javascript";
    script.src   = "jdb/"+iso3+".js"; //?_="+ (new Date()); URL might have to be randomized to avoid cached data
    document.body.appendChild(script);
    */

    // call the data-handling function only after 
    // the desired data has fully loaded
    // in this case, I check every 100 milliseconds
    var intervalId = setInterval( function () {
        if (data[iso3]) {    //once data is detected, process data
            clearInterval(intervalId); //stop checking for data
            prepData(mainWrapper, iso3)
            drawCharts(mainWrapper, iso3)
        }
    }, 100)
}
else drawCharts(mainWrapper, iso3)

Answer 2

For JSON:

var data = JSON.parse(JavascriptStringVariableThatContainsJSON);

For CSV:

 var data = d3.csv.parseRows(JavascriptStringVariableThatContainsMyCSVdata);

//then add data to the graph and call enter, something like:

 var dataEnter = svg.selectAll("rect").data(data).enter();

Answer 3

So you want to load the local data through a FileReader call, the stuff below is taken from Reading local files in JavaScript. However, building on the example from above, this code loads a image file into an svg without linking the file, but actually inserting the raster data into the svg. And feel free to replace raster data with any other data, just add the correct processing to it...

First off the supporting html elements in the <body> section:

<input type="file" id="files" name="files[]" multiple />

Then you want to link the <input ... /> element to some code in the <script> section:

document.getElementById('files').addEventListener('change', handleFileSelect, false);

Now the code behind the file read in the event handling code (Now in this case I want to load a local image into a D3JS svg):

function handleFileSelect(evt) {
    reader = new FileReader();
    reader.onabort = function(e) {alert('File read cancelled');};
    var imageObj = new Image();  //image object to hold the local file contents.

    //and given the asynchronous nature, set up the event so that when the image is loaded, do something with it:
    imageObj.onload = function() {image.datum(imageObj.src).attr("xlink:href", function(d) {return d})};

    //similarly for the file reading:
    reader.onload = function(e) {imageObj.src = reader.result};

    //With all the events set up, lets start reading the file in.
    //the secret sauce is the DataURL
    reader.readAsDataURL(evt.target.files[0]);
}

And for completeness the D3JS sauce behind the image:

var svg = d3.select("body").append("svg");

var image = svg.append("defs")
    .append("pattern")
    .attr("id", "venus")
    .attr('patternUnits', 'userSpaceOnUse')
    .attr("width", 200)
    .attr("height", 200)
    .append("image")
    .attr("width", 200)
    .attr("height", 200);

var image2 = svg.append("rect")
    .attr("x", "0")
    .attr("y", "0")
    .attr("width", 200)
    .attr("height", 200)
    .attr("fill", "url(#venus)");

Answer 4

If you are generating HTML each time, you can put data as JSON either in right in your HTML or in a .js file that you can reference from the HTML, probably via a generated unique URL.

Answer 5

D3 has some great tools for importing data and manipulating it. Having always served json in the past (and that is sometimes a hassle to create) I'm going more towards serving csv and manipulating it in javascript/d3 but would be interested to see how others respond to your question.

This is a simple example showing csv to nested json here: https://gist.github.com/3053667

Also lots of tools for manipulating data: https://github.com/mbostock/d3/wiki/Arrays

And then there is Crossfilter if you want to slice and dice. Looks well worth getting to grips with but am waiting for the dummies guide to come out! http://square.github.com/crossfilter/

Answer 6

d3.json loads external json files via ajax calls - the myjson variable in your example is already a javascript object so you don't need to load it, just use it directly in your nodeData assignment.

nodeData = myjson.elements;