HTML Canvas Sniff Webgl Data

Question

This question might be weird but suppose we have a canvas which for example draws some 3D content like this experiment.

Disregarding using ThreeJS, Babylon or any other library to achieve same effect, is it possible to set some interval that copies the birth of every voxel and repeat (redraw) it later.

Simply I want to record the canvas draw process and replay it, without using RTC , video, or images sequence.

What Have been done?

I have been trying with WebGl Context and Stream Capture, but unfortunately could not achieve the desired result.

Can anyone help with this?

How did you tried the Stream Capture ? It works well for me in the given [example link](http://www.laplace2be.com/lab/PixelParticles/). Copy this in the console : `let s = mycanvas.captureStream(), r = new MediaRecorder(s), chunks = []; r.ondataavailable = e => chunks.push(e.data); r.onstop = e => { window.open(URL.createObjectURL(new Blob(chunks))); }; r.start(); setTimeout(_=>r.stop(), 3000);` — Kaiido, Jan 19 '17 at 04:13
@Kaiido thanks a lot for your comment, I did that, but this blob is actually a webm video which actually is not an optimum solution in my case, as the size will be huge. would you please add this as answer ? — ProllyGeek, Jan 19 '17 at 05:00

gman · Accepted Answer · 2017-01-19T03:26:00.677

You can wrap the WebGL context and capture all the function calls. An example of wrapping the WebGL context would be something like

const rawgl = document.querySelector("canvas").getContext("webgl");
const gl = wrapContext(rawgl);

gl.viewport(0, 0, gl.canvas.width, gl.canvas.height);
gl.enable(gl.SCISSOR_TEST);
gl.scissor(40, 50, 200, 60);
gl.clearColor(0,1,1,1);
gl.clear(gl.COLOR_BUFFER_BIT);
gl.scissor(60, 40, 70, 90);
gl.clearColor(1,0,1,1);
gl.clear(gl.COLOR_BUFFER_BIT);

function wrapContext(gl) {
  const wrapper = {};
  for (let name in gl) {
    var prop = gl[name];
    if (typeof(prop) === 'function') {
      wrapper[name] = wrapFunction(gl, name, prop);
    } else {
      wrapProperty(wrapper, gl, name);
    }
  }
  return wrapper;
}

function wrapFunction(gl, name, origFn) {
  // return a function that logs the call and then calls the original func
  return function(...args) {
    log(`gl.${name}(${[...args].join(", ")});`);
    origFn.apply(gl, arguments);
  };
}

function wrapProperty(wrapper, gl, name) {
  // make a getter because these values are dynamic
  Object.defineProperty(wrapper, name, {
    enumerable: true,
    get: function() {
      return gl[name];
    },
  });
}

function log(...args) {
  const elem = document.createElement("pre");
  elem.textContent = [...args].join(" ");
  document.body.appendChild(elem);
}

canvas { border: 1px solid black; }
pre { margin: 0; }

<canvas></canvas>

In your case instead of logging the calls you'd add them to some array of calls only on the frames you want captured.

You then need to somehow keep track of all the resources (buffers, textures framebuffers, renderbuffers, shaders, programs) and all their parameters (like filtering settings on textures) and you also need to track uniform settings etc.

The WebGL-Inspector does this and can playback frames so it might be a good example. There's also this webgl-capture library.

What you need to capture for your program is up to your program. For example if you know your buffers and textures never change and they're still in memory when you want to playback then maybe you don't need to try to capture the state of buffers and textures which both of the above examples have to do.

What do you mean? [It scrolls for me](http://imgur.com/a/qkhY1). If it's not scrolling for you I suggest you [file a bug](http://meta.stackoverflow.com). Otherwise any code block which i longer than 40 lines automatically gets a scrollbar — gman, Jan 19 '17 at 03:29
Thanks for your help @gman , your answer made me discover lot of stuff I did not realize before. — ProllyGeek, Jan 19 '17 at 19:02
@gman is capture webgl your library ? if so , would you please add some examples, it really sounds promising. — ProllyGeek, Jan 19 '17 at 19:28
Yes I wrote it. I mostly pointed it out as an example but when actually using it it generates way large files because of all the data it has to write out (textures and buffers) so it's better an an example. There's a working example [here](https://greggman.github.io/webgl-capture/examples/twgl-cube.html). It captures the first frame and then displays the generated HTML above the sample. If you copy and paste that HTML to another file you get [this](https://greggman.github.io/webgl-capture/examples/twgl-cube-example-capture.html). — gman, Jan 20 '17 at 02:43

score 1 · Answer 2 · answered Jan 19 '17 at 05:06

I don't know how you did tried with the captureStream method, but on your example page this code does work.

let s = mycanvas.captureStream(),
r = new MediaRecorder(s),
chunks = [];
r.ondataavailable = e => chunks.push(e.data);
r.onstop = e => {
  let videoURL = URL.createObjectURL(new Blob(chunks));
  doSomethingWith(videoURL);
  };
r.start();
setTimeout(_=>r.stop(), 3000); // records 3 seconds

Now that you've got a valid blobURL pointing to your canvas' record, you can play it in a <video> element, and then draw it in your webgl context.

HTML Canvas Sniff Webgl Data

2 Answers2