I am now trying to interpolate video using optical flow. I was able to interpolate the video by referring to this question and using it as a reference. So my question is: Is it possible to interpolate the video in 1/4 units even finer using the original frame and the optical flow frame? Thank you in advance.
I tried halving the optical flow frame and remapped it to the original image, but it did not work. (It's obvious, of course, but...).
I will try solving this problem with the help of an example. Suppose my previous image is
and my next image is 
The next image was created by translating the previous image towards right (I hope it is visible).
Now, let's calculate the optical flow between the two images and plot the flow vectors.
optical_flow = cv2.calcOpticalFlowFarneback(img1, img2, None, pyr_scale = 0.5, levels = 3, winsize = 200, iterations = 3, poly_n = 5, poly_sigma = 1.1, flags = 0)
I have taken the previous image and drawn flow vectors on top of it.
This image shows that, previous image was formed by translating the next image towards the left.
If you followed this, it is pretty easy to accomplish the task at hand .i.e interpolate frames between these two frames
Essentially, optical flow vectors tell us for each pixel, where has the pixel come from the previous image. Lets talk about a particular pixel, which has moved from (100, 100) to (200, 100) .i.e moved in the right direction by 100 pixels.
The question is where will this pixel be if there was uniform motion between these 2 frames?
Example: if there were 4 frames in between these two frames, the pixel will be at locations
initial location = (100, 100)
interpolated frame 1 = (120, 100)
interpolated frame 2 = (140, 100)
interpolated frame 3 = (160, 100)
interpolated frame 4 = (180, 100)
final location. = (200, 100)
Enough talking, let me show you some code.
def load_image(image_path):
"Load the image and convert it into a grayscale image"
img = cv2.imread(image_path)
return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
def plot_flow_vectors(img, flow):
"""
Plots the flow vectors
"""
img = img.copy()
h, w = img.shape[:2]
step_size = 40
cv2.imshow("original_image", img)
for y in range(0, h, step_size):
for x in range(0, w, step_size):
# Get the flow vector at this point
dx, dy = flow[y, x]
# Draw an arrow to represent the flow vector
cv2.arrowedLine(img, (x, y), (int(x + dx), int(y + dy)), (0, 255, 0), 1, cv2.LINE_AA, tipLength=0.8)
cv2.imshow("image_with_flow_vectors", img)
cv2.waitKey(0)
cv2.destroyAllWindows()
img1 = load_image('first.jpeg') // load previous image
img2 = load_image('second.jpeg') // load next image
optical_flow = cv2.calcOpticalFlowFarneback(img1, img2, None, pyr_scale = 0.5, levels = 3, winsize = 200, iterations = 3, poly_n = 5, poly_sigma = 1.1, flags = 0) // calculate optical flow
plot_flow_vectors(img1, optical_flow) // plot flow vectors
// Generate frames in between
num_frames = 10
h, w = optical_flow.shape[:2]
for frame_num in range(1, num_frames+1):
alpha = frame_num / num_frames
flow = alpha * optical_flow
flow[:,:,0] += np.arange(w)
flow[:,:,1] += np.arange(h)[:,np.newaxis]
interpolated_frame = cv2.remap(img2, flow, None, cv2.INTER_LINEAR)
cv2.imshow("intermediate_frame", interpolated_frame)
cv2.waitKey(0)
PS: optical flow vectors also show an motion in the y direction (when there isn't). I think it is an artifact of the large motion between the two frames. This is generally not the case for frames in a video.
Edit: I created a github repository to interpolate frames in between two video frames to increase the video fps or get a slow motion video out of the input video. Check it out here
