1

I've tried to combine 4 separate byte arrays into a single file but I'm only getting null pointer exceptions and I'm not sure why. My audio format is 16 bit PCM signed and I know I should be using short instead of bytes but honestly I'm quite lost in it all.

private short[] mixByteBuffers(byte[] bufferA, byte[] bufferB) {
    short[] first_array = new short[bufferA.length/2];
    short[] second_array = new short [bufferB.length/2];
    short[] final_array = null;

    if(first_array.length > second_array.length) {
        short[] temp_array = new short[bufferA.length];

        for (int i = 0; i < temp_array.length; i++) {
            int mixed=(int)first_array[i] + (int)second_array[i];
            if (mixed>32767) mixed=32767;
            if (mixed<-32768) mixed=-32768;
            temp_array[i] = (short)mixed;
            final_array = temp_array;
        }
    }
    else {
        short[] temp_array = new short[bufferB.length];

        for (int i = 0; i < temp_array.length; i++) {
            int mixed=(int)first_array[i] + (int)second_array[i];
            if (mixed>32767) mixed=32767;
            if (mixed<-32768) mixed=-32768;
            temp_array[i] = (short)mixed;
            final_array = temp_array;
        }        
    }
    return final_array;
}

This is what I'm trying at the moment but it's returning with java.lang.ArrayIndexOutOfBoundsException: 0 at line

int mixed = (int)first_array[i] + (int)second_array[i];

My arrays aren't all the same length, this is how I call the function:

public void combineAudio() {
    short[] combinationOne = mixByteBuffers(tempByteArray1, tempByteArray2);
    short[] combinationTwo = mixByteBuffers(tempByteArray3, tempByteArray4);
    short[] channelsCombinedAll = mixShortBuffers(combinationOne, combinationTwo);
    byte[] bytesCombined = new byte[channelsCombinedAll.length * 2];
    ByteBuffer.wrap(bytesCombined).order(ByteOrder.LITTLE_ENDIAN)
        .asShortBuffer().put(channelsCombinedAll);

    mixedByteArray = bytesCombined;
}

There's got to be a better way than what I'm doing currently, it's driving me absolutely crazy.

Hendrik
  • 5,085
  • 24
  • 56
o.no
  • 70
  • 6
  • 1) See [What is a stack trace, and how can I use it to debug my application errors?](http://stackoverflow.com/q/3988788/418556) & [What is a Null Pointer Exception, and how do I fix it?](http://stackoverflow.com/q/218384/418556) 2) For better help sooner, [edit] to add a [MCVE] or [Short, Self Contained, Correct Example](http://www.sscce.org/). 3) Always copy/paste error and exception output! – Andrew Thompson Mar 15 '20 at 05:59
  • Formatting the code to make the nesting more visible would help a lot. – Phil Freihofner Mar 16 '20 at 04:39
  • With "combine", do you mean "mix"? In other words: How many channels has your input, how many has your output? – Hendrik Mar 16 '20 at 10:11
  • @Hendrik sorry for not being more clear, that's definitely a me problem. I'm trying to combine 4 channels into a single output channel. – o.no Mar 16 '20 at 21:35

2 Answers2

2

The temp_array.length value in the else clause for loop is bufferB.length. But the value in the if clause is bufferA.length/2. Did you overlook dividing by 2 in the else clause?

Regardless, it's usual to just process audio data (signals) as streams. With each line open, grab a predefined buffer's worth of byte values from each, enough to get the same number of PCM values from each line. If one line runs out before the others, you can fill out that line with 0 values.

Unless there is a really strong reason for adding arrays of unequal length, I think it's best to avoid doing so. Instead, use pointers (if you are drawing from arrays) or progressive read() methods (if from AudioInput lines) to get the fixed number of PCM values each loop iteration. Otherwise, I think you are asking for trouble, needlessly complicating things.

I've seen workable solutions where just one PCM value is processed from each source at a time, to more, such as 1000 or even a full half second (22,050 if at 44100 fps). The main thing is get the same number of PCM from each source on each iteration, and fill in with 0's if a source runs out of data.

Phil Freihofner
  • 7,645
  • 1
  • 20
  • 41
2

To mix two byte arrays with 16 bit sound samples, you should first convert those arrays to int arrays, i.e., sample-based arrays, then add them (to mix) and then convert back to byte arrays. When converting from byte array to int array, you need to make sure you use the correct endianness (byte order).

Here's some code that lets you mix two arrays. At the end there is some sample code (using sine waves) that demonstrates the approach. Note that this is may not be the ideal way of coding this, but a working example to demonstrate the concept. Using streams or lines, as Phil recommends is probably the smarter overall approach.

Good luck!

import javax.sound.sampled.AudioFileFormat;
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;

public class MixDemo {

    public static byte[] mix(final byte[] a, final byte[] b, final boolean bigEndian) {
        final byte[] aa;
        final byte[] bb;

        final int length = Math.max(a.length, b.length);
        // ensure same lengths
        if (a.length != b.length) {
            aa = new byte[length];
            bb = new byte[length];
            System.arraycopy(a, 0, aa, 0, a.length);
            System.arraycopy(b, 0, bb, 0, b.length);
        } else {
            aa = a;
            bb = b;
        }

        // convert to samples
        final int[] aSamples = toSamples(aa, bigEndian);
        final int[] bSamples = toSamples(bb, bigEndian);

        // mix by adding
        final int[] mix = new int[aSamples.length];
        for (int i=0; i<mix.length; i++) {
            mix[i] = aSamples[i] + bSamples[i];
            // enforce min and max (may introduce clipping)
            mix[i] = Math.min(Short.MAX_VALUE, mix[i]);
            mix[i] = Math.max(Short.MIN_VALUE, mix[i]);
        }

        // convert back to bytes
        return toBytes(mix, bigEndian);
    }

    private static int[] toSamples(final byte[] byteSamples, final boolean bigEndian) {
        final int bytesPerChannel = 2;
        final int length = byteSamples.length / bytesPerChannel;
        if ((length % 2) != 0) throw new IllegalArgumentException("For 16 bit audio, length must be even: " + length);
        final int[] samples = new int[length];
        for (int sampleNumber = 0; sampleNumber < length; sampleNumber++) {
            final int sampleOffset = sampleNumber * bytesPerChannel;
            final int sample = bigEndian
                    ? byteToIntBigEndian(byteSamples, sampleOffset, bytesPerChannel)
                    : byteToIntLittleEndian(byteSamples, sampleOffset, bytesPerChannel);
            samples[sampleNumber] = sample;
        }
        return samples;
    }

    private static byte[] toBytes(final int[] intSamples, final boolean bigEndian) {
        final int bytesPerChannel = 2;
        final int length = intSamples.length * bytesPerChannel;
        final byte[] bytes = new byte[length];
        for (int sampleNumber = 0; sampleNumber < intSamples.length; sampleNumber++) {
            final byte[] b = bigEndian
                    ? intToByteBigEndian(intSamples[sampleNumber], bytesPerChannel)
                    : intToByteLittleEndian(intSamples[sampleNumber], bytesPerChannel);
            System.arraycopy(b, 0, bytes, sampleNumber * bytesPerChannel, bytesPerChannel);
        }
        return bytes;
    }

    // from https://github.com/hendriks73/jipes/blob/master/src/main/java/com/tagtraum/jipes/audio/AudioSignalSource.java#L238
    private static int byteToIntLittleEndian(final byte[] buf, final int offset, final int bytesPerSample) {
        int sample = 0;
        for (int byteIndex = 0; byteIndex < bytesPerSample; byteIndex++) {
            final int aByte = buf[offset + byteIndex] & 0xff;
            sample += aByte << 8 * (byteIndex);
        }
        return (short)sample;
    }

    // from https://github.com/hendriks73/jipes/blob/master/src/main/java/com/tagtraum/jipes/audio/AudioSignalSource.java#L247
    private static int byteToIntBigEndian(final byte[] buf, final int offset, final int bytesPerSample) {
        int sample = 0;
        for (int byteIndex = 0; byteIndex < bytesPerSample; byteIndex++) {
            final int aByte = buf[offset + byteIndex] & 0xff;
            sample += aByte << (8 * (bytesPerSample - byteIndex - 1));
        }
        return (short)sample;
    }

    private static byte[] intToByteLittleEndian(final int sample, final int bytesPerSample) {
        byte[] buf = new byte[bytesPerSample];
        for (int byteIndex = 0; byteIndex < bytesPerSample; byteIndex++) {
            buf[byteIndex] = (byte)((sample >>> (8 * byteIndex)) & 0xFF);
        }
        return buf;
    }

    private static byte[] intToByteBigEndian(final int sample, final int bytesPerSample) {
        byte[] buf = new byte[bytesPerSample];
        for (int byteIndex = 0; byteIndex < bytesPerSample; byteIndex++) {
            buf[byteIndex] = (byte)((sample >>> (8 * (bytesPerSample - byteIndex - 1))) & 0xFF);
        }
        return buf;
    }

    public static void main(final String[] args) throws IOException {
        final int sampleRate = 44100;
        final boolean bigEndian = true;
        final int sampleSizeInBits = 16;
        final int channels = 1;
        final boolean signed = true;
        final AudioFormat targetAudioFormat = new AudioFormat(sampleRate, sampleSizeInBits, channels, signed, bigEndian);

        final byte[] a = new byte[sampleRate * 10];
        final byte[] b = new byte[sampleRate * 5];

        // create sine waves
        for (int i=0; i<a.length/2; i++) {
            System.arraycopy(intToByteBigEndian((int)(30000*Math.sin(i*0.5)),2), 0, a, i*2, 2);
        }
        for (int i=0; i<b.length/2; i++) {
            System.arraycopy(intToByteBigEndian((int)(30000*Math.sin(i*0.1)),2), 0, b, i*2, 2);
        }

        final File aFile = new File("a.wav");
        AudioSystem.write(new AudioInputStream(new ByteArrayInputStream(a), targetAudioFormat, a.length),
                AudioFileFormat.Type.WAVE, aFile);
        final File bFile = new File("b.wav");
        AudioSystem.write(new AudioInputStream(new ByteArrayInputStream(b), targetAudioFormat, b.length),
                AudioFileFormat.Type.WAVE, bFile);

        // mix a and b
        final byte[] mixed = mix(a, b, bigEndian);
        final File outFile = new File("out.wav");
        AudioSystem.write(new AudioInputStream(new ByteArrayInputStream(mixed), targetAudioFormat, mixed.length),
                AudioFileFormat.Type.WAVE, outFile);
    }
}
Hendrik
  • 5,085
  • 24
  • 56
  • I tried using this implementation for mixing two audio samples but I am getting a noise when trying to mix them. The two audio samples are stereo PCM 16 bit per sample. Not able to remove the noise from that. Please suggest anything over here. TIA. – umesh lohani Dec 30 '20 at 08:48
  • The code above is for mono sounds. You will have to modify it suitably to accommodate stereo. Typically the samples in a PCM stereo signal are interleaved, i.e. LRLRLR... You need to mix the Ls with the Ls and the Rs with the Rs. Good luck! – Hendrik Dec 30 '20 at 20:40
  • I don't have much understanding of Audio processing but as per what I read so far, for a 16 bit stereo sample It would be 2 bytes per channel. So the above algo should work for stereo sample. Please correct me if I am wrong. – umesh lohani Jan 07 '21 at 10:23
  • You still have to adjust the channel count, correctly read your data, etc. I suggest you ask a new question, showing exactly what you did and how ([mcve]). – Hendrik Jan 07 '21 at 13:20
  • I have added a new question here. https://stackoverflow.com/questions/65662025/mixing-two16-bit-encoded-stereo-pcm-samples-causing-noise-and-distortion-in-the – umesh lohani Jan 11 '21 at 06:02