best method of turning millions of x,y,z positions of particles into visualisation

Question

I'm interested in different algorithms people use to visualise millions of particles in a box. I know you can use Cloud-In-Cell, adaptive mesh, Kernel smoothing, nearest grid point methods etc to reduce the load in memory but there is very little documentation on how to do these things online.

i.e. I have array with:

x,y,z
1,2,3
4,5,6
6,7,8
xi,yi,zi

for i = 100 million for example. I don't want a package like Mayavi/Paraview to do it, I want to code this myself then load the decomposed matrix into Mayavi (rather than on-the-fly rendering) My poor 8Gb Macbook explodes if I try and use the particle positions. Any tutorials would be appreciated.

Answer 1

Analysing and creating visualisations for complex multi-dimensional data is complex. The best visualisation almost always depends on what the data is, and what relationships exists within the data. Of course, you are probably wanting to create visualisation of the data to show and explore relationships. Ultimately, this comes down to trying different posibilities.

My advice is to think about the data, and try to find sensible ways to slice up the dimensions. 3D plots, like surface plots or voxel renderings may be what you want. Personally, I prefer trying to find 2D representations, because they are easier to understand and to communicate to other people. Contour plots are great because they show 3D information in a 2D form. You can show a sequence of contour plots side by side, or in a timelapse to add a fourth dimension. There are also creative ways to use colour to add dimensions, while keeping the visualisation comprehensible -- which is the most important thing.

I see you want to write the code yourself. I understand that. Doing so will take a non-trivial effort, and afterwards, you might not have an effective visualisation. My advice is this: use a tool to help you prototype visualisations first! I've used gnuplot with some success, although I'm sure there are other options.

Once you have a good handle on the data, and how to communicate what it means, then you will be well positioned to code a good visualisation.

UPDATE

I'll offer a suggestion for the data you have described. It sounds as though you want/need a point density map. These are popular in geographical information systems, but have other uses. I haven't used one before, but the basic idea is to use a function to enstimate the density in a 3D space. The density becomes the fourth dimension. Something relatively simple, like the equation below, may be good enough.

The point density map might be easier to slice, summarise and render than the raw particle data.

The data I have analysed has been of a different nature, so I have not used this particular method before. Hopefully it proves helpful.

PS. I've just seen your comment below, and I'm not sure that this information will help you with that. However, I am posting my update anyway, just in case it is useful information.