I would start with avisynth it's a simple scripting language for video and can use avidemux to edit video interactively or x264 and ffmpeg to read/write files automatically.
edit:
see http://forum.videohelp.com/threads/184560-AviSynth-Chroma-Key
The avisynth interpreter loads your file itself, it then spits out the processed frames. If you run the script with the avisynth proxy (included with avidemux) then players like avidemux can connect to the proxy as if they were opening a file and play the processed video.
The x264 commandline H264 codec can also run avisynth scripts, you just supply the script as the input filename (remember the actual source file is inside the script) and it will output the processed, and then H264 coded file to an mp4
the tutorials make more sense than I do .......