Parsing frames from raspberry pi video stream

Question

Problem

A project I am working on requires me to parse frames and do some very simple image processing from the raspberry pi camera at around 10Hz.

I tried using both the raspistill and the raspiyuv (raw still) applications to produce images, however neither of them were able to reach the 10Hz framerate that I was looking for (even at the low resolution of 128x64 which is all I need, in timelapse mode it was talking ~1/2 a second per image with a setting of 100ms delay for the timelapse).

The raspivid application allows me to output video at higher than the necessary 10 frames/second, however I'm not really sure what the best way to go about grabbing frames for image processing from the .h264 stream.

What I've looked at

I found Capturing jpegs from an h264 stream with gstreamer on a Raspberry Pi where someone managed to get jpegs at 1Hz from the .h264 camera stream using gstreamer, which would suggest that it's possible to simply parse raw frames using gstreamer.

I tried copying what was done, but when I attempt to call his unmodified pipeline, just to ensure it is working correctly before moving on to attempting to modify it it gives me a pipeline error linking from videorate0 to jpegenc0.

Questions

Is what I am attempting possible using gstreamer and available plugins (or should I look at other tools, such as ffmpeg)? If so, could anyone offer assistance with getting this working? Finally, would it be possible to use this tool further to pipe these raw frames straight into a c program for parsing? (Possibly by using ! filesink location=/dev/stdout?)

Answer 1

@MarkTolley I believe I'm using gstreamer 1.0, though I should note that I'm not very experienced using gstreamer, so I used the build script within this pastebin, which seemed to run correctly, and install the required plugins, etc.. Would this potentially be outdated?

It is outdated, but it is still gstreamer-1.0. Looks like the script is using the 1.0.x branch due to a bug in 1.1.x. Unfortunately I don't know if that bug was fixed in 1.2.x or not.

After digging through the mailing list thread the author links to, I found the post where he finally got it working: http://gstreamer-devel.966125.n4.nabble.com/Capturing-jpegs-from-an-h264-stream-tp4660254p4660459.html

raspivid -n -t 1000000 -vf -b 2000000 -fps 25 -o - | gst-launch-1.0 fdsrc ! video/x-h264,framerate=25/1,stream-format=byte-stream ! decodebin ! videorate ! video/x-raw,framerate=10/1 ! videoconvert ! jpegenc ! multifilesink location=img_%04d.jpg

Try the pipeline that he uses in that post. If you're still having problems I suggest posting to the mailing list or visiting the IRC channel, as those places are much better for troubleshooting than SO.

---

I'll try to answer the other questions from your original post:

Is what I am attempting possible using gstreamer and available plugins

Absolutely! Gstreamer also has support for hardware encoding/decoding h264 on the rpi, which I don't know if other tools have.

Finally, would it be possible to use this tool further to pipe these raw frames straight into a c program for parsing

I see you're already using fdsrc, there is a corresponding fdsink that can write to stdout.

But if you're going to be writing a C program anyway, you might be better off implementing an appsink and having gstreamer feed buffers directly to your code.