Is there a consistent way to keep track the position in a Clip (or audio file) in Java?

Question

I'm trying to code a rhythm game in Java. I'm playing an audio file with a javax.sound.sampled.Clip and I want to know the exact position where I am.

Following my researches, a good practice for a rhythm game is to keep track of the time using the position in the music playing instead of the System time (to avoid a shift between the audio and the game).

I already coded a main loop which uses System.nanoTime() to keep a consistent frame rate. But when I try to get the time of the clip, I have the same time returned for several frames. Here is an extract from my code :

while (running) {
    // Get a consistent frame rate
    now = System.nanoTime();
    delta += (now - lastTime) / timePerTick;
    timer += now - lastTime;
    lastTime = now;
    if (delta >= 1) {
        if(!paused) {
            tick((long) Math.floor(delta * timePerTick / 1000000));
        }
        display.render();
        ticks++;
        delta--;
    }
    // Show FPS ...
}

// ...

private void tick(long frameTime) {
    // Doing some stuff ...
    System.out.println(clip.getMicrosecondPosition());
}

I was expecting a consistent output (with increments of approximately 8,333 µs per frame for 120 fps) but I get the same time for several frames.

Console return:

0
0
748
748
748
10748
10748
10748
20748
20748
20748
30748

How can I keep consistently and accurately keep track of the position where I am in the song ? Is there any alternative to the javax Clip ?

Answer 1

There are a number of difficulties to deal with. First off, Java doesn't provide real-time guarantees, so the moment when a portion of the sound data is being processed doesn't necessarily correspond to when it is heard. Secondly, it is easy to get messed up by the OS system clock's lack of granularity, which is what I'm guessing is contributing to your seeing timings appear multiple times. We had a lot of back-and-forth about that at java-gaming.org several years ago. [EDIT: now I'm thinking your check is looping faster than the granularity of the buffer size being used by the Clip.]

The only way I know to deal with getting really precise timing is to abandon the use of Clip altogether, and instead, count frames while outputting on a SourceDataLine. I've had good success with this. It is the very last point prior to output to the native sound rendering code, and thus has the best timing.

That said, it might be possible to implement some sort of pulse (for example, using a util.Timer) at a regular interval (corresponding to quarter notes or sixteenth notes) and have the TimerTask initiate the playback of Clip objects. I haven't tried that myself but it could be sufficient for the rhythm game. For this you would use Clip objects and not SourceDataLine. But make sure that each Clip is fully loaded and ready to go when played. Don't make the common mistake of loading them anew for each playback.

But when I last tried to do something like this, using a common game loop, the result wasn't that hot.

JavaFX is great for game writing! I find the graphics much easier to handle than AWT. I wrote a tutorial for getting started with JavaFX for game programming. It is out of date, as far as setting up with Eclipse, if you are using JavaFX 11. But if you are using Java 8 I think it still covers the basics.

At JGO, people are in a couple different camps. Many like to use Libgdx, a library, and some of us continue to use JavaFX or AWT/Swing. There are another couple libraries also championed by various members.

JavaFX has its own sound output, but I haven't played with it, and don't know if it is any better than a Clip for rhythmic precision. The cues all play great, but it is always going to be hard getting usable timing info from the cues given Java's system of continually switching between threads.

Was just reviewing contents on JGO. You might find this thread interesting, where the OP was trying to make a metronome. http://www.java-gaming.org/topics/java-audio-metronome-timing-and-speed-problems/33591/view.html

[EDIT: P.S., I've written a reliable metronome, and have an event playback system that is reliable enough to string together musical event in seamless rhythm, and Listeners that can follow things like the volume envelopes of the notes being played in real time. It should be doable to do what you want, ultimately.]

Answer 2

The issue with clip.getMicrosecondPosition() is that it can only tell you the position of audio that has been sent to the native audio system or corresponds to a buffer that is being rendered.

What does that mean?

When playing audio, you typically have a bunch of samples in memory (like in a Clip) that you send to the native audio system to play. Now, sending each sample individually would be super inefficient, which is why you always send samples in bulk. In javax.sound.sampled terms, you are writing them to a SourceDataLine using its write(buf) method. Now when you write the data, it is not sent directly to the speaker, but instead written to a buffer. The native system then takes samples from that buffer and sends them to the speaker. The system may choose again to take samples from that buffer in bulk.

So when you ask the clip for its microsecond position, it may not really give you the position of what has actually been rendered (that is played on the speaker), but what has been written from the in-memory clip to the line's buffer or what has been taken by the system from the line's internal buffer.

So the resolution of clip.getMicrosecondPosition() may depend on buffer sizes.

To influence the used buffer sizes, you might want to try something like this when creating your clip:

int bufferSize = 1024;
AudioFormat format = new AudioFormat(AudioFormat.Encoding.PCM_SIGNED,
    AudioSystem.NOT_SPECIFIED, 16, 2, 4, AudioSystem.NOT_SPECIFIED, true);
DataLine.Info info = new DataLine.Info(Clip.class, format, bufferSize);
Clip clip = (Clip) AudioSystem.getLine(info)

This asks the audio system to use a buffer of just 1024 bytes when playing the clip. Using a format with 16 bits per sample, 2 channels per frame, and a sample rate of 44100 Hz, this would correspond to 256 frames (4 bytes/frame), i.e., 256/44100=0.0058 seconds, a very short buffer.

For a high resolution (low latency audio), a short buffer is preferable. However, when making the buffer too short, you may experience buffer underflows, which means your audio will "hang" or "stutter". This is very annoying to the user. Keep in mind that Java uses garbage collection. So you may simply not be able to continuously write audio samples to the system, when Java is busy throwing out the garbage (lack of real time capabilities).

Hope this helps. Good luck!