3

UPDATE: This entire issue ended up being a problem with the systems graphics driver, and not (seemingly) a browser / API issue. The torn frames came down to the actual display updating. Thank you again to those who were a part of the discussion and attempts to help.


I have page that uses a canvas and 2d context to display a pre-rendered frame at 720p. I'm rendering the frames separately and updating a variable with the new ImageData. Then, within requestAnimationFrame I simply do context.putImageData(cached_image_data);. Despite having the frame fully rendered in advance and effectively double buffered, I still get tearing far too often. There are a few other questions along these lines that I've found on SO, but they all end in "Use RAF". The code comes down to this:

var canvas = document.getElementById("canvas");
var cached_frame = new ImageData(new Uint8ClampedArray(canvas.width * canvas.height * 4), canvas.width, canvas.height);
var context = canvas.getContext("2d");
var framerate = 30;

function draw() {
    if (cached_frame)
        context.putImageData(cached_frame, 0, 0);

    requestAnimationFrame(draw);
}

setInterval(function() {
    var frame = context.getImageData(0, 0, canvas.width, canvas.height);

    // Do things to manipulate frame.data.

    // Save the resultant pixel data for the cached_frame.
    cached_frame.data = frame.data;
}, 1000 / framerate);

draw();

Is there anything more that I can do without turning to webgl?

Any suggestions appreciated. TY all :D

The Busy Wizard
  • 956
  • 1
  • 7
  • 11
  • It's so simple that I didn't include code initially. Added for clarity. – The Busy Wizard Feb 26 '16 at 07:20
  • 1
    Is the shearing at the same spot all the time or does it move around randomly? Is it just one machine, or all machines you have tried it on? Is it browser dependent? You should not expect any shearing so we need more info to work out what is going on. – Blindman67 Feb 26 '16 at 07:35
  • 1
    canvas is already double buffered. Avoid putImageData and getImageData for anything that needs to happen this often (super slow). Create a new canvas through code and just draw the result of that. – ericjbasti Feb 26 '16 at 18:31
  • 2
    this whole portion of code `setInterval(function() { cached_frame = new ImageData(new Uint8ClampedArray(pixel_data)` is going to kill the memory. The Garbage collector is going to be firing all the time because your creating a new Image and Uint8ClampedArray every time the interval fires. Save these as variables and reuse them at the very least. – ericjbasti Feb 26 '16 at 18:34
  • 1
    *Just some quick thoughts building on @ericjbasti's good comments:* if possible, avoid `getImageData` / `putImageData` for performance reasons. rAF will help your shearing. Do both getImageData and putImageData operations in the same rAF. Use the timestamp argument that's auto-fed into `draw(timestamp)` to do timed activities. Put `requestAnimationFrame(draw)` at the bottom (not top) of draw(). Canvas is natively double-buffered, but you may need to buffer additional frames if you're presenting video. – markE Feb 26 '16 at 20:08
  • @Blindman67 Shearing location (both row and time) are inconsistent. It is happening in FF, Chrome, and Chromium. It does vary by machine. Running on an i7-6500U it doesn't seem to have issues. This is the Intel HD 520 gfx, with proper drivers. On various other machines (including a Mac) it appears fairly often. I've noticed that on an Intel NUC with i7-5557U and Iris gfx *without* proper driver support, it is happening very consistently. (Not really surprising IMO.) – The Busy Wizard Feb 27 '16 at 08:26
  • @ericjbasti / markE I should be reusing the buffer and ImageData. I had a reason for not originally, though you're right, I need to address that. I hate that I have to use getImageData and it is as slow as it is. This seems like something that would be highly optimized by the browser, though I suppose that's where webGL comes in. I'm not aware of any better way to get pixel data from a canvas, or better yet, directly from the – The Busy Wizard Feb 27 '16 at 08:32
  • Gee I must be going blind. Just saw you are using setInterval. Sorry I did not see that last time. That is the cause of the shearing. Never use `setInterval` for any reason for too many reasons to type here. If you need that done every 30th of a second do it every second frame with requestAnimationFrame – Blindman67 Feb 27 '16 at 08:45
  • @Blindman67 rAF is *NOT* consistent (in reality and in the spec). I know that setInterval is inaccurate and non-optimal, but it is within reason for my purposes of rendering the frames themselves. Note that I'm making a distinction between rendering and drawing. The drawing portion is as short and simple as I can make it (to my knowledge); all it does is draw the pre-rendered ImageData onto the canvas. When the cached frame is updated, the var referencing it is reassigned directly, so it is not a matter of being mid-update when it is drawn (from rAF / draw() ). – The Busy Wizard Feb 27 '16 at 09:03
  • @IanWizard: Why do you say rAF is not "consistent" -- It works consistently in my apps. :-// ... And as we say: rAF will help your tearing problems while setInterval allows tearing. – markE Feb 27 '16 at 20:22
  • 1
    @markE LOL and the irony is I specialise in computer graphics. :) – Blindman67 Feb 27 '16 at 22:22
  • @markE rAF is *extremely* consistent.. when it's mean to be. It also knows not to bother firing it when the user isn't going to see the result, such as if the user is on another tab and the page isn't visible. To be more accurate, there simply are no animation frames when the page isn't visible, so it fires on the next animation frame when it is visible. I need to be rendering (not drawing) the frames regardless so I chose to render them with an interval and then stuff the results into cached_frame. Then whenever rAF fires, it draws that frame. (My) rendering would break if run on every rAF. – The Busy Wizard Feb 28 '16 at 18:17
  • @IanWizard. Ahhhh, I see now: rAF runs *consistently*, but rAF does not run *constantly* when a different browser tab is in focus. You are allowing `draw()` to `putImageData` based on the existence of `cached_frame`. It exists even in its incomplete state which allows your tearing. Instead, create a global true/false flag that is trued when the newest `cached_frame` is fully complete. *An Aside:* stealing CPU time from another browser tab? Thief!! :-) – markE Feb 29 '16 at 03:18
  • @markE That's the thing though: The cached_frame is reassigned, not updated. Unless there is actually a deem flaw in the JS engine, it seems incredibly improbably that it would be partially reassigned. That said, there may be a flaw somewhere in my code that is sharing a Buffer that it shouldn't be, which is allowing it to be partially updated. I will review and clean all of the code today and update this post. Aside: Yes... yes I am :( – The Busy Wizard Feb 29 '16 at 15:18

1 Answers1

3

I don't think the code is doing what you think it's doing

First off, as far as I know you can't assign new data to an ImageData so this line

cached_frame.data = frame.data;

Doesn't do anything. We can test that which shows it doens't work

var ctx = document.createElement("canvas").getContext("2d");
document.body.appendChild(ctx.canvas);

var imageData = ctx.getImageData(0, 0, ctx.canvas.width, ctx.canvas.height);
var data = new Uint8ClampedArray(imageData.length);

// fill imageData.data with red
fillWithColor(255, 0, 0, 255, imageData.data);

// fill data with green
fillWithColor(0, 255, 0, 255, data);

// assign imageData.data to data
imageData.data = data;

// Draw. If assigning imageData.data works result will
// be green, if not result will be red
ctx.putImageData(imageData, 0, 0);

function fillWithColor(r, g, b, a, dst) {
  for (ii = 0; ii < dst.length; ii += 4) {
    dst[ii + 0] = r;
    dst[ii + 1] = g;
    dst[ii + 2] = b;
    dst[ii + 3] = a;
  }
}
  

Second, your draw function is drawing continuously, at least from the code you posted cached_frame is set on line 2 so it's always going to be true and always going to be drawing. If you're somehow partially updating the actual data in cached_frame then it's going to draw when there are only partial results.

I think you want something like this instead

var canvas = document.getElementById("canvas");
var context = canvas.getContext("2d");
var frame;
var framerate = 30;

function draw() {
   context.putImageData(frame, 0, 0);
}

setInterval(function() {
    frame = context.getImageData(0, 0, canvas.width, canvas.height);

    // Do things to manipulate frame.data

    // frame is ready, draw it at next rAF
    requestAnimationFrame(draw);
}, 1000 / framerate);

You might want to check if a draw it is already queued if you think decoding will ever happen faster than raf. I don't think you actually need rAF in this case though. I'm pretty sure you could just draw at the end of your setInterval and it will show up the next frame, no tearing.

Here's a test, it's not tearing for me.

var canvas = document.getElementById("canvas");
var context = canvas.getContext("2d");
var frame;
var framerate = 30;
var frameCount = 0;

setInterval(function() {
  ++frameCount;
  frame = context.getImageData(0, 0, canvas.width, canvas.height);
  var data = frame.data;
  var width = frame.width;
  var height = frame.height;
  // Do things to manipulate frame.data
  for (var yy = 0; yy < height; ++yy) {
    for (var xx = 0; xx < width; ++xx) {
      var offset = (yy * width + xx) * 4;
      data[offset + 0] = ((xx >> 2 & 0x1) ^ frameCount & 0x1) ? 255 : 0;
      data[offset + 3] = 255;
    }
  }

  // frame is ready, draw it at next rAF
  context.putImageData(frame, 0, 0);
}, 1000 / framerate);
<canvas id="canvas" width="1280" height="720"></canvas>
gman
  • 100,619
  • 31
  • 269
  • 393
  • You are correct, the ImageData cannot be updated, which is something that I just ran into moments ago. Also, the check for cached_frame was an oversight when I updated the code. It isn't needed in the scenario above. You're also right that the call to rAF should go at the end of setInterval as there is no need unless the frame has been updated. I put the actual call to putImageData() in a rAF rather than the setInterval so as to avoid calling it unnecessarily (such as when the page is out of focus, or simply under heavy CPU load). I am going through everything now and will test it soon. – The Busy Wizard Feb 27 '16 at 09:43
  • Did your tearing disappear? – gman Feb 27 '16 at 09:47
  • Using `setInterval` risks some very bad behaviour on devices that can not handle the load. As setInterval places calls on the call stack irrespective of whether the last call has exited, it will slowly fill up the call stack and eventually crash, and there are many reason a page can slow. There is also the problem that rAF can stop firing while setTimeout does not. You can not guarantee that each interval will call the draw function via rAF. Use setTimeout if you have a real need to use a timer. – Blindman67 Feb 27 '16 at 22:37
  • I don't know where you got your info but setInterval does not fill up any call stacks. That is not the way it works. As for rAF vs setInterval vs setTimeout each have their place. The specific example above is simulating receiving XHR responses so it's a perfect fit. – gman Feb 28 '16 at 06:38
  • Regarding using an interval vs rAF: I *need* to be rendering constantly. rAF does not fire in many situations where I want to be rendering the frames. This is (half of) the point of rAF; it allows you to not do any rendering / drawing when the user won't see it. For these reasons, I'm using setInterval to handle the rendering behind the scenes, and rAF for the actual drawing portion. – The Busy Wizard Feb 28 '16 at 18:07
  • Regarding your code above (red and black lines). It is tearing, which I would expect to happen when drawing at random times. This is the other half of rAF; it allows you to synchronize drawing / rendering in such a way that you and the browser can appropriately vsync. The tearing is hard to notice with the red and black lines moving such as they are, but it is indeed tearing for me. I checked in FF and Chrome on the i7-6500U (the device which isn't tearing with my code). – The Busy Wizard Feb 28 '16 at 18:10
  • That's now how the browser works. If it's tearing for you that's an issue in your browser. When you update anything that changes what is displayed in your browser the browser marks the page as "needs to be re-rendered". It does this re-render on the next vblank. It doesn't matter if you do the change in a raf or a setTimeout or on a mouse event. The update to what's visible on the page won't happen until the next vblank unless (a) the browser has a bug or (b) can't sync to vblank because of bugs in your driver or (c) is just not implemented in your browser. – gman Feb 29 '16 at 04:56
  • What rAF allows you to do is (a) get called everytime the browser re-draws, something other events don't and therefore (b) make sure all your updates to various parts of the page happen in the same event so they'll all be seen at the same time when the page is rendered. In the example above, since there is only 1 event there no way to get out of sync with other events but there will still be no tearing. What platform are you on? What does your "about:gpu" in chrome say? Also there is one more reason you'd a tear. If you turned off vsync in your browser settings – gman Feb 29 '16 at 05:08
  • @gman If you're scanning through the pixels and augmenting them, then the tearing is entirely possible, regardless of what event you're hooking on. You're absolutely right, it shouldn't be an issue for me because the canvas is being updated in a single call to putImageData(). If internally that sets the pixels one-by-one, then it may cause this issue, but I'd like to believe that that wouldn't be an issue across three different browsers, especially since it is (should be) double-buffered internally. – The Busy Wizard Feb 29 '16 at 15:24
  • @gman GREAT SCOTT!! 2d canvas acceleration and accelerated decoding are disabled according to about:gpu. That's upsetting, but more importantly... that's on the laptop that runs it just fine. I will have to wait until later today to check on a device that is having problems. – The Busy Wizard Feb 29 '16 at 15:26
  • If you `putImageData` your half augmented pixels then yes you can get tearing. My example is not doing that. It's augmenting all the pixels (which are not the actual pixels but a copy. `putImageData` then copies all those augmented pixels back into the canvas which then gets copied to a texture, when then gets recomposited on to the page the next vblank. no tearing unless vsync is off or compositing is not hardware accelerated – gman Mar 01 '16 at 01:55