How to efficiently store past depth pixel data from depthMapRealWorld() method?

Question

I can't get around a peculiar problem with SimpleOpenNI for Processing ao I'm asking for your help.

I'd like to store snapshots of pixel depth data (returned by .depthMapRealWorld() method as PVector arrays) on discrete time intervals, then process them further for a presentation. I tried adding them in an ArrayList, but it seems that the depthMapRealWorld() method is returning only a reference to a current depth data, not a real array. I tried in this sequence:

1. Just getting the data and adding it in an arraylist. On every call of the update() method the whole arraylist contained the same PVector array, even if the array at the zero position was added many iterations away!

2. Then I made the PVector array, along with its creation time, part of a class. Rewrote the sketch a little, but it didn't help. All of the arrays in the arraylist werw still the same.

3. Finally, in the constructor of the class, I "manually" copied the xyz coordinates of every vector from the PVector array into a int array. That seemed to solve the problem - the int arrays in the arraylist are now different from each other. But this solution introduced serious performance problems.

The question is: is there a more efficient way of storing these PVector arrays and retaining their value?

code:

import processing.opengl.*;
import SimpleOpenNI.*;

SimpleOpenNI kinect;


float rotation = 0;
int time = 0;
ArrayList dissolver;
ArrayList<Integer> timer;
int pSize = 10;
Past past;

void setup()  {
  dissolver = new ArrayList();
  timer = new ArrayList();
  size(1024, 768, OPENGL);
  kinect = new SimpleOpenNI(this);
  kinect.enableDepth();
  translate(width/2, height/2, -100);
  rotateX(radians(180));
  stroke(255);
}

void draw() {
  background(0);
  translate(width/2, height/2, 500);
  rotateX(radians(180));
  kinect.update();
  stroke (255, 255, 255);

  past = new Past (kinect.depthMapRealWorld(), time);
  if (dissolver.size() == pSize) {  //remove the oldest arraylist element if when list gets full
    dissolver.remove(0); //
  }  

  if (time % 20 == 0) {
    dissolver.add (past);
    Past p1 = (Past) dissolver.get (0);
    float [][] o2 = p1.getVector(); 
    println ("x coord of a random point at arraylist position 0: " + o2[50000][0]); //for testing
  }
  if (dissolver.size() == pSize-1) {
    //dissolve ();  
  }
  time ++;
}

void dissolve () { //from the previous nonworking version; ignore
  for (int offset = 0; offset < pSize-1; offset ++) {
    PVector[] offPoints = (PVector[]) dissolver.get (offset);
    int offTime =  timer.get(offset);
    for (int i = 0; i < offPoints.length; i+=10) {
      int col = (time-offTime)*2; //why??
      stroke (255, 0, col);
      PVector currentPoint = offPoints[i];
      if (currentPoint.z <1500) {
        point(currentPoint.x, currentPoint.y, currentPoint.z); // - 2*(time-offTime) + random(0, 100)
      }
    }
  }
}

class Past {
  private PVector [] depth; //should contain this, not int 
  private float [][] depth1;
  private  int time;

  Past (PVector [] now, int t) {
    //should be like this: depth = now;
    //clumsy and performancewise catastrophic solution below
    depth1 = new float [now.length][3];
    for (int i = 0; i< now.length; i+=10) {
      PVector temp = now[i];
      depth1 [i][0] = temp.x;
      depth1 [i][1] = temp.y;
      depth1 [i][2] = temp.z;

    }
    //arrayCopy(now, depth); this didn't work either
    time = t;
  }

  float [][] getVector () {
    return depth1;
  }

  int getTime () {
    return time;
  }
}

Answer 1

If I understood correctly, you want to store the 3D positions(ArrayList of PVectors) for each frame, right ? If so, you should be able to simply store PVectors and reference them later. Here's a basic sketch to illustrate this:

import processing.opengl.*;
import SimpleOpenNI.*;

SimpleOpenNI kinect;
ArrayList<ArrayList<PVector>> frames = new ArrayList<ArrayList<PVector>>();
ArrayList<PVector> frame;
boolean isRecording = true;
boolean isRecFrame;

void setup()  {
  size(1024, 768, OPENGL);
  kinect = new SimpleOpenNI(this);
  kinect.enableDepth();
  stroke(255);
}

void draw() {
  background(0);
  translate(width/2, height/2, 500);
  rotateX(PI);
  translate(0,0,-1000); 

  kinect.update();

  if(isRecording){

    isRecFrame = (frameCount % 20 == 0);//record every 20 frames
    int[]   depthMap = kinect.depthMap();
    int     steps   = 5;  // to speed up the drawing, draw every N point
    int     index;
    PVector realWorldPoint;
    if(isRecFrame) frame = new ArrayList<PVector>();

    for(int y=0;y < kinect.depthHeight();y+=steps)
    {
      for(int x=0;x < kinect.depthWidth();x+=steps)
      {
        index = x + y * kinect.depthWidth();
        if(depthMap[index] > 0)
        { 
          realWorldPoint = kinect.depthMapRealWorld()[index];
          point(realWorldPoint.x,realWorldPoint.y,realWorldPoint.z);
          if(isRecFrame) frame.add(realWorldPoint.get());
        }
      } 
    }
    if(isRecFrame) frames.add(frame); 

  }else{//playback
    ArrayList<PVector> currentFrame = frames.get(frameCount%frames.size());//playback is faster than recording now for testing purposes - add a decent frame counter here at some point
    for(PVector p : currentFrame) point(p.x,p.y,p.z);
  }

}

void keyPressed(){
  if(key == ' ') isRecording = !isRecording;
}

Use the SPACE key to toggle between recording and playback. The main thing to note is I'm storing a copy of the real world position for each depth pixel (frame.add(realWorldPoint.get());). Another thing to keep in mind is that currently you're storing these coordinates in memory which at some point will fill. If you only store a limited number of frames that should be fine, if not you might want to save to the points to disk. This way you can reuse recordings with other sketches. A basic way would be to sore them in a csv file:

void saveCSV(ArrayList<PVector> pts){
  String csv = "x,y,z\n";
  for(PVector p : pts) csv += p.x + "," + p.y + "," + p.z + "\n";
  saveStrings("frame_"+frameCount+".csv",csv.split("\n"));
}

Another would be to use a more suitable format for point clouds, like PLY. Saving an ASCII PLY is fairly straight forward:

void savePLY(ArrayList<PVector> pts){
  String ply = "ply\n";
  ply += "format ascii 1.0\n";
  ply += "element vertex " + pts.size() + "\n";
  ply += "property float x\n";
  ply += "property float y\n";
  ply += "property float z\n";
  ply += "end_header\n";
  for(PVector p : pts)ply += p.x + " " + p.y + " " + p.z + "\n";
  saveStrings("frame_"+frameCount+".ply",ply.split("\n"));
}

You can later open/explore/process these files with tools like MeshLab.