How to remove rolling shutter effect in camera preview?

Question

I'm trying to send data to my phone by filming a blinking LED. This works fine most of the time, but for some frames I get a rolling shutter effect in my camera preview, which messes up my data transfer. Is there a way to remove this rolling shutter effect?

I am working with Google's cameraAPI2 and a Nexus 6 (but I've seen the same effect on a Samsung S7). I record at 20 fps and the LED blinks at a frequency of 20 Hz. My images have a format of 320x240 with YUV_420_888. Here are my current settings for a CaptureRequest:

CONTROL_AE_MODE CONTROL_AE_MODE_OFF

CONTROL_AWB_MODE CONTROL_AWB_MODE_OFF

EDGE_MODE EDGE_MODE_OFF

CONTROL_MODE CONTROL_MODE_OFF

SENSOR_EXPOSURE_TIME 500000

SENSOR_FRAME_DURATION 50000000

I've already tried playing with exposure time and frame duration to no effect. The same effect can be observed by simply opening the stock camera app and filming a LED close up. Example

The right half of the picture shows the old frame and the left half shows the new frame.

Is there a way to fix this? Does Android have camera vsync?

Edit: I changed the frequencies to 20Hz.

Edit 2: I got it working by performing Canny Edge Detection and then Standard Hough Line Transformation on the image using opencv. After getting the line coordiantes from the Hough Line Transformation, I checked if the line is horizontal. Then I split the image along this line and only used the larger area for my calculations.

Answer 1

Rolling shutter effects stem from the fact that cameras with CMOS image sensor most often read out the pixel data sequentially, row by row (or column by column), so separate rows/columns that make up one image frame are not captured at the same point in time.

There is no "camera vsync" since rolling shutter is a physical property of how the image sensor is built.

Apart from switching to a global shutter image sensor (like a CCD or some newer CMOS sensors) you can try increasing the shutter speed/reducing exposure time of your camera to minimize the problem.

You can consult this question as a concrete direction how to find out available shutter speeds for your camera.

Update: what you can try is setting the shutter speed of your camera to the fastest setting and then matching your LED blinking frequency with your camera frame rate (1/30s). This way the blinking of the LED should always be at the same time point of the image acquisition and you shouldn't get any problems with rolling shutter effects anymore.

Update2: as discussed in the comments, your setup now seems to work like this. You have a LED that can change it's state (on or off) every 50 ms. You have a camera that captures a scene over the course of 50 ms to produce one image. Leftmost column is at the beginning of these 50 ms, rightmost column is at the end of these 50 ms (or vice versa, you need to figure that out). Since the 50 ms period of the LED is not synchronized with the 50 ms capture time of the camera, a state change of the LED will probably happen in the middle of the camera image that is produced, this splits the image apart in a dark and a bright half.

Here is how a random LED sequence might look like on the image:

LED sequence:   ... 0  |  1  |  1  |  0  |  1  |  0  |  0  |  0 ...

Image sequence: ... | 0 1 | 1 1 | 1 0 | 0 1 | 1 0 | 0 0 | 0 0 | ...

The pipes denote when the LED may change state and when the image finishes one frame capture.

As you can see in the drawing, assuming the frequencies are the same the divide in the image between old and new state is always going to stay at the same image column. All you have to do is detect where the line divides the image (should be doable with some computer vision) and then only always look at one half of the image to see if that half changes. Ideally you should look at the half which is recorded later since that one is giving you the most up to date information from the LED.

I hope that helps.