I am trying to use the overlay
filter with multiple input sources, for an Android app. Basically, I want to overlay multiple video sources on top of a static image.
I have looked at the sample that comes with ffmpeg and implemented my code based on that, but things don't seem to be working as expected.
In the ffmpeg filtering sample there seems to be a single video input. I have to handle multiple video inputs and I am not sure that my solution is the correct one. I have tried to find other examples, but looks like this is the only one.
Here is my code:
AVFilterContext **inputContexts;
AVFilterContext *outputContext;
AVFilterGraph *graph;
int initFilters(AVFrame *bgFrame, int inputCount, AVCodecContext **codecContexts, char *filters)
{
int i;
int returnCode;
char args[512];
char name[9];
AVFilterInOut **graphInputs = NULL;
AVFilterInOut *graphOutput = NULL;
AVFilter *bufferSrc = avfilter_get_by_name("buffer");
AVFilter *bufferSink = avfilter_get_by_name("buffersink");
graph = avfilter_graph_alloc();
if(graph == NULL)
return -1;
//allocate inputs
graphInputs = av_calloc(inputCount + 1, sizeof(AVFilterInOut *));
for(i = 0; i <= inputCount; i++)
{
graphInputs[i] = avfilter_inout_alloc();
if(graphInputs[i] == NULL)
return -1;
}
//allocate input contexts
inputContexts = av_calloc(inputCount + 1, sizeof(AVFilterContext *));
//first is the background
snprintf(args, sizeof(args), "video_size=%dx%d:pix_fmt=%d:time_base=1/1:pixel_aspect=0", bgFrame->width, bgFrame->height, bgFrame->format);
returnCode = avfilter_graph_create_filter(&inputContexts[0], bufferSrc, "background", args, NULL, graph);
if(returnCode < 0)
return returnCode;
graphInputs[0]->filter_ctx = inputContexts[0];
graphInputs[0]->name = av_strdup("background");
graphInputs[0]->next = graphInputs[1];
//allocate the rest
for(i = 1; i <= inputCount; i++)
{
AVCodecContext *codecCtx = codecContexts[i - 1];
snprintf(args, sizeof(args), "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
codecCtx->width, codecCtx->height, codecCtx->pix_fmt,
codecCtx->time_base.num, codecCtx->time_base.den,
codecCtx->sample_aspect_ratio.num, codecCtx->sample_aspect_ratio.den);
snprintf(name, sizeof(name), "video_%d", i);
returnCode = avfilter_graph_create_filter(&inputContexts[i], bufferSrc, name, args, NULL, graph);
if(returnCode < 0)
return returnCode;
graphInputs[i]->filter_ctx = inputContexts[i];
graphInputs[i]->name = av_strdup(name);
graphInputs[i]->pad_idx = 0;
if(i < inputCount)
{
graphInputs[i]->next = graphInputs[i + 1];
}
else
{
graphInputs[i]->next = NULL;
}
}
//allocate outputs
graphOutput = avfilter_inout_alloc();
returnCode = avfilter_graph_create_filter(&outputContext, bufferSink, "out", NULL, NULL, graph);
if(returnCode < 0)
return returnCode;
graphOutput->filter_ctx = outputContext;
graphOutput->name = av_strdup("out");
graphOutput->next = NULL;
graphOutput->pad_idx = 0;
returnCode = avfilter_graph_parse_ptr(graph, filters, graphInputs, &graphOutput, NULL);
if(returnCode < 0)
return returnCode;
returnCode = avfilter_graph_config(graph, NULL);
return returnCode;
return 0;
}
The filters
argument of the function is passed on to avfilter_graph_parse_ptr
and it can looks like this: [background] scale=512x512 [base]; [video_1] scale=256x256 [tmp_1]; [base][tmp_1] overlay=0:0 [out]
The call breaks after the call to avfilter_graph_config
with the warning:
Output pad "default" with type video of the filter instance "background" of buffer not connected to any destination
and the error Invalid argument
.
What is it that I am not doing correctly?
EDIT: The are two issues that I have discovered:
Looks like the description of
avfilter_graph_parse_ptr
is a bit vague. Theouputs
parameter represents a list of the current outputs of the graph, in my case that being thegraphInputs
variable, because these are the outputs from thebuffer
filter. Theinputs
parameter represents a list of the current inputs of the graph, in this case this is thegraphOutput
variable, because it represents the input to thebuffersink
filter.I did some testing with a
scale
filter and a single input. It seems that the name of theAVFilterInOut
structure required byavfilter_graph_parse_ptr
needs to bein
. I have tried with different versions:in_1
,in_link_1
. None of them work and I have not been able to find any documentation related to this.
So the issue still remains. How do I implement a filter graph with multiple inputs?