Mixing Audio Using OpenGL

Question

I want to mix two (or more) 16bit audio streams using OpenGL and I need a bit of help

Basically what I want to do is to put the audio data into texture which I draw to a frame buffer object and then read back. This is not a problem, however drawing the data in way that gives correct results is a bit more problematic.

I have basically two questions.

In order to mix the data by drawing i need to use blending (alpha = 0.5), however the result should not have any alpha channel. So if I render to e.g. a frame buffer with the format RGB will alpha blending still work as I expect and the resulting alpha will not be written to the fbo? (I want to avoid having to read back the fbo for each render pass)

texture |sR|sG|sB|

framebuffer(before) |dR|dG|dB|

framebuffer(after) |dR*0.5+sR*0.5|dG*0.5+sG*0.5|dB*0.5+sB*0.5|

The audio samples are signed 16bit integer values. Is it possible do signed calculations this way? Or will I need to first convert the values to unsigned on the cpu, draw them, and then make them signed again on the cpu?

EDIT:

I was a bit unclear. My hardware is restricted to OpenGL 3.3 hardware. I would prefer to not use CUDA or OpenCL, since I'm alrdy using OpenGL for other stuff.

Each audio sample will be rendered in seperate passes, which means that it has to "mix" with whats already been rendered to the frame buffer. The problem is how the output from the pixel shader is written to the framebuffer (this blending is not accessible through programmable shaders, as far as i know, and one has to use glBlendFunc).

EDIT2:

Each audio sample will be rendered in different passes, so only one audio sample will be available in the shader at a time, which means that they need to be accumulated in the FBO.

foreach(var audio_sample in audio_samples)
     draw(audio_sample);

and not

for(int n = 0; n < audio_samples.size(); ++n)
{
      glActiveTexture(GL_TEXTURE0 + n);
      glBindTexture(audio_sample);
}
draw_everything();

qdot · Answer 1 · 2010-12-15T15:35:31.293

4

Frankly, why wouldn't you just use programmable pixel shaders for that?

Do you have to use OpenGL 1 Fixed Functionality Pipeline?

I'd just go with a programmable shader operating on signed 16bit grayscale linear textures.

Edit:

foreach(var audio_sample in audio_samples) 
 blend FBO1 + audio_sample => FBO2
 swap FBO2, FBO1

It ought to be just as fast, if not faster (thanks to streaming pipelines).

edited Dec 15 '10 at 15:35

answered Dec 15 '10 at 14:54

qdot

6,195
5
44
95

A pixel shader would probably be useful. However, the problem is how I get the result from the pixel shader written correctly to the framebuffer, since it "should" mix with what's already in the framebuffer. Maybe I was a bit unclear. But each "audio sample" will be rendered in seperate passes. Soo I will only have access to that in the shader. – ronag Dec 15 '10 at 14:59
2

You can have a pixel shader render into a PBO or FBO, without bothering yourself with the alpha channel of the PBO (you can just disable them and blend two textures directly by a shader) – qdot Dec 15 '10 at 15:04
Yes, but I will not have access to more than one texture in the shader since each audio sample will be rendered in different passes. They need to be accumulated pass by pass in the FBO. – ronag Dec 15 '10 at 15:11
1

You're definitely approaching the problem from a nonstandard angle. I guess you can do the bouncing - render from one FBO to another FBO and swap, each pass. – qdot Dec 15 '10 at 15:12
+1, that's an excellent suggestion and seems to be compilant with what the OP asks for., – Kos Dec 15 '10 at 16:56

score 2 · Answer 2 · answered Dec 15 '10 at 14:57

2

I agree with QDot. Could you however inform us a bit about the hardware restrictions you are facing? If you have reasonable modern hardware I might even suggest to go the CUDA or OpenCL route, in stead of going through OpenGL.

answered Dec 15 '10 at 14:57

Bart

19,692
7
68
77

Is there a really convincing reason to go through OpenGL? Do you need to interface with something in particular? – Bart Dec 15 '10 at 15:31
Well the main reason is that I don't want to add another dependency if not necessary... it doesn't seem justifiable to involve OpenCL just for this... especially if I can do the job with OpenGL which is alrdy used extensively in my project. – ronag Dec 15 '10 at 23:20

score 1 · Accepted Answer · answered Dec 15 '10 at 16:15

You should be able to do blending even if the destination buffer does not have alpha. That said, rendering to non-power-of-two sizes (rgb16 = 6bytes/pixel) usually incurs performance penalties.
Signed is not your typical render target format, but it does exist in the OpenGL 4.0 specification (Table 3.12, called RGB16_SNORM or RGB16I, depending on whether you want a normalized representation or not).

As a side note, you also have glBlendFunc(GL_CONSTANT_ALPHA,GL_ONE_MINUS_CONSTANT_ALPHA) to not even have to specify an alpha per-pixel. That may not be available on all GL implementations though.

Mixing Audio Using OpenGL

3 Answers3