Sound in videos is full of static

Question

I'm trying to play sound from an FFMpegFrameGrabber by getting the Frame and sending the audio samples to a SourceDataLine. Here's what I have so far:

Creating the SourceDataLine:

int channels = _grabber.getAudioChannels();
int format = _grabber.getSampleFormat();
AudioFormat fmt = new AudioFormat(_grabber.getSampleRate(), format, channels, true, true);
_sourceDataLine=(SourceDataLine)AudioSystem.getLine(new DataLine.Info(SourceDataLine.class, fmt));
_sourceDataLine.open(fmt);
_sourceDataLine.start();

Attempting to play sound (images are handled in the else block):

org.bytedeco.javacv.Frame f = _grabber.grabFrame();

if (f.samples != null && f.samples.length > 0)
{
    byte[] bytes = new byte[4096];
    for (Buffer buffer : f.samples)
    {
        FloatBuffer floatBuffer = (FloatBuffer) buffer;
        ByteBuffer byteBuffer = ByteBuffer.allocate(floatBuffer.capacity() * 4);
        byteBuffer.asFloatBuffer().put(floatBuffer);
        byteBuffer.rewind();
        byteBuffer.get(bytes);
        _sourceDataLine.write(bytes, 0, bytes.length);
    }
}

(Note: I tried a few different versions of this and they all have static. The versions I tried included combining the buffers into one large buffer, only trying to play one sample instead of each channel, and changing the audio format to many different permutations.)

The problem is the sound is full of static, and almost completely unintelligible. This is my first time doing any audio programming, so I'm sure I'm doing something completely ridiculous.

I appreciate any help. Thank you.

EDIT

In response to Radiodef, I tried a number of AudioFormats, and I couldn't find one that worked for PCM_FLOAT. I found an example that used this:

fmt = new AudioFormat(AudioFormat.Encoding.PCM_FLOAT, _grabber.getSampleRate(), format, channels, channels, _grabber.getSampleRate(), true);

Note: I tried a few different values for the framesize from examples: channels * format / 8, channels * 8 with a hardcoded samplerate of 64, channels * 4 with a hardcoded samplerate of 32, and any combinations of those

But it give me this exception:

java.lang.IllegalArgumentException: No line matching interface SourceDataLine supporting format PCM_FLOAT 44100.0 Hz, 8 bit, stereo, 2 bytes/frame,  is supported.
    at javax.sound.sampled.AudioSystem.getLine(Unknown Source)
    at com.enplug.player.video.Video.<init>(Video.java:52) <- where I get the SourceDataLine
    ...

EDIT 2

Sorry for the delay. I appreciate all the help Radiodef.

Here is some output from the FFMpegGrabber that is automatically output.

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'C:\Users\Shawn\AppData\Roaming\Enplug Display\Download\Resource\c7cb496d-96ea-4be8-a238-5ffd50955a3e.mp4':
  Metadata:
    major_brand     : qt
    minor_version   : 0
    compatible_brands: qt
    creation_time   : 2014-10-02 07:14:38
  Duration: 00:00:31.13, start: 0.000000, bitrate: 2412 kb/s
    Stream #0:0(und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 246 kb/s (default)
    Metadata:
      creation_time   : 2014-10-02 07:14:38
      handler_name    : Core Media Data Handler
    Stream #0:1(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720, 2157 kb/s, 30 fps, 30 tbr, 600 tbn, 1200 tbc (default)
    Metadata:
      creation_time   : 2014-10-02 07:14:38
      handler_name    : Core Media Data Handler
      encoder         : H.264

I have two videos I'm testing with, and the first one (which is the one in the example above) has the following:

Bit rate: 247 kbps
Channels: 2 (stereo)
Audio sample rate: 44 kHz

And the second is:

Bit rate: 161 kbps
Channels: 2 (stereo)
Audio sample rate: 48 kHz

They're both mp4s, and I can provide any details about the video itself if needed.

As for the library, yeah I'm pretty locked into JavaCV. We already have videos running without sound, but we're now trying to add sound to our program.

When I run the sample program from your JSR link I get:

PCM_UNSIGNED unknown sample rate, 8 bit, mono, 1 bytes/frame,
PCM_SIGNED unknown sample rate, 8 bit, mono, 1 bytes/frame,
PCM_SIGNED unknown sample rate, 16 bit, mono, 2 bytes/frame, little-endian
PCM_SIGNED unknown sample rate, 16 bit, mono, 2 bytes/frame, big-endian
PCM_UNSIGNED unknown sample rate, 8 bit, stereo, 2 bytes/frame,
PCM_SIGNED unknown sample rate, 8 bit, stereo, 2 bytes/frame,
PCM_SIGNED unknown sample rate, 16 bit, stereo, 4 bytes/frame, little-endian
PCM_SIGNED unknown sample rate, 16 bit, stereo, 4 bytes/frame, big-endian
PCM_UNSIGNED unknown sample rate, 8 bit, mono, 1 bytes/frame,
PCM_SIGNED unknown sample rate, 8 bit, mono, 1 bytes/frame,
PCM_SIGNED unknown sample rate, 16 bit, mono, 2 bytes/frame, little-endian
PCM_SIGNED unknown sample rate, 16 bit, mono, 2 bytes/frame, big-endian
PCM_UNSIGNED unknown sample rate, 8 bit, stereo, 2 bytes/frame,
PCM_SIGNED unknown sample rate, 8 bit, stereo, 2 bytes/frame,
PCM_SIGNED unknown sample rate, 16 bit, stereo, 4 bytes/frame, little-endian
PCM_SIGNED unknown sample rate, 16 bit, stereo, 4 bytes/frame, big-endian
PCM_UNSIGNED unknown sample rate, 8 bit, mono, 1 bytes/frame,
PCM_SIGNED unknown sample rate, 8 bit, mono, 1 bytes/frame,
PCM_SIGNED unknown sample rate, 16 bit, mono, 2 bytes/frame, little-endian
PCM_SIGNED unknown sample rate, 16 bit, mono, 2 bytes/frame, big-endian
PCM_UNSIGNED unknown sample rate, 8 bit, stereo, 2 bytes/frame,
PCM_SIGNED unknown sample rate, 8 bit, stereo, 2 bytes/frame,
PCM_SIGNED unknown sample rate, 16 bit, stereo, 4 bytes/frame, little-endian
PCM_SIGNED unknown sample rate, 16 bit, stereo, 4 bytes/frame, big-endian
PCM_UNSIGNED unknown sample rate, 8 bit, mono, 1 bytes/frame,
PCM_SIGNED unknown sample rate, 8 bit, mono, 1 bytes/frame,
PCM_SIGNED unknown sample rate, 16 bit, mono, 2 bytes/frame, little-endian
PCM_SIGNED unknown sample rate, 16 bit, mono, 2 bytes/frame, big-endian
PCM_UNSIGNED unknown sample rate, 8 bit, stereo, 2 bytes/frame,
PCM_SIGNED unknown sample rate, 8 bit, stereo, 2 bytes/frame,
PCM_SIGNED unknown sample rate, 16 bit, stereo, 4 bytes/frame, little-endian
PCM_SIGNED unknown sample rate, 16 bit, stereo, 4 bytes/frame, big-endian

You'll have to find out what the audio format actually is. Inspect the file outside of this API. — Radiodef, Feb 17 '15 at 19:16
I want to make it work independent of the file's details using the contents of javacv. However, inspecting the file doesn't have a framesize value. There are only Bitrate, channels, and audio sample. — Shawn Blakesley, Feb 17 '15 at 20:12
*"I want to make it work independent of the file's details"* You should use a library that's more clearly documented. If you are forced to use JavaCV, what you need to find out is what the return value of `getSampleFormat` means. It's not the bits per sample so you can't pass it directly to AudioFormat. (If you want it to work for all formats, why does your code assume it's float? You have a lot more work to do.) — Radiodef, Feb 17 '15 at 20:39
Haha yeah I know I have more work to do, but I need to get it to work first then make it work for everything. If I can get one thing working that would help — Shawn Blakesley, Feb 17 '15 at 21:56

Radiodef · Answer 1 · 2015-02-17T20:50:43.617

1

FFMpeg aside, which I don't know much about, it looks like there is a problem with the way you are constructing the AudioFormat. The constructor you are using assumes integer LPCM but you are writing floats to it. You should use a constructor that takes an AudioFormat.Encoding. If you know it's going to be float, you should be using AudioFormat.Encoding.PCM_FLOAT.

Again, if you know that the file is 32-bit float, you can hardcode the format to verify it works as follows:

new AudioFormat(
    AudioFormat.Encoding.PCM_FLOAT,
    _grabber.getSampleRate(),
    32,
    channels,
    channels * 4,
    _grabber.getSampleRate(),
    true // you should also verify whether this is true
)

That should get you a valid line. If that doesn't get you a line, then it's not a format the platform can play back.

edited Feb 17 '15 at 20:50

answered Feb 17 '15 at 18:28

Radiodef

37,180
14
90
125

I tried using PCM_FLOAT, but every value I enter gives me an error. I'll add it to the main question because it is too long for a comment. Thank you. – Shawn Blakesley Feb 17 '15 at 19:01
So I tried that and it doesn't work either. But it must be in a format that my platform is able to play back because I can play the video on my desktop successfully. – Shawn Blakesley Feb 17 '15 at 21:54
Whatever you are playing the video with is performing its own conversion, or maybe Java sound does not have full access to the provider. If you can examine the file and post here exactly what the format is we can give you a good idea whether or not it's supported. [JSR also has sample code](http://www.jsresources.org/faq_audio.html#supported_formats) to find out what formats are supported by the platform. That's if JavaCV does not perform its own conversion which I can't determine. – Radiodef Feb 17 '15 at 22:04

score 0 · Answer 2 · answered Oct 15 '17 at 08:49

I had added sound in my program.Recently i am tring to play video with sound using javaCV.You can grab frame which sample type is FloatBuffer when you play video with aac format audio.The float value range is -1 to 1.So you can simply convert them to short value ref:

    private ShortBuffer convert(FloatBuffer arr){
    int len = arr.capacity();
    float f;
    ShortBuffer res = ShortBuffer.allocate(len);
    for(int i=0;i<len;i++){
        f = arr.get(i)*32768;
        if(f>32767.0f) f = 32767.0f;
        if(f<-32768.0f) f = 32768.0f;
        res.put(i,(short)f);
    }
    return res;
}

then you can create a AudioFormat if your aac is stereo as following:

AudioFormat af = new AudioFormat(AudioFormat.Encoding.PCM_SIGNED,audioSampleRate,16,2,4,audioSampleRate,true);

note: endian is true or false depend on you how to convert float to byte,if setting is opposite,result a noise.

but Buffer[] length is two,every Buffer is a channel,so you should mix them to a stream when you have convert shortBuffer to byte as flowwing:

byte[] buf0,buf1;
byte[] stream = new byte[4];
for(int i=0;i<buf0.length;i=i+2){//mix the two channel
    for(int j=0;j<2;j++){
        stream[j] = buf0[i+j];
        stream[j+2] = buf1[i+j];
         }
    sourceDataLine.write(stream,0,4);
}

Sound in videos is full of static

2 Answers2