Java Audio Metronome | Timing and Speed Problems

Hi all,

I’m starting to work on a music/metronome application in Java and I’m running into some problems with the timing and speed.

My test program plays two sine wave tones together at regular intervals and updates two visual displays (one display for each tone). At slower speeds the two tones will play in sync for a little while, then slightly out of sync for a few beats, then they will play back in sync again, and this cycle continues. At faster speeds, the synchronization is all over the place.

From researching good metronome programming, I found that Thread.sleep() is horrible for timing, so I completely avoided that and went with checking System.nanoTime() to determine when the sounds should play.

I’m using AudioSystem’s SourceDataLine for my audio player and I’m using a thread for each tone that constantly polls System.nanoTime() in order to determine when to play the tones and update the visual display. I create a new SourceDataLine and delete the previous one each time a sound plays, because the volume fluctuates if I leave the line open and keep playing sounds on the same line…I’ve read that you’re not supposed to leave the SourceDataLine open when not playing anything, but I wanted to see what happened if I did. Unfortunately the process of creating a new player each time is costing a lot of time, I timed the process for creating the player and it was upwards of about 270ms. However, I’m not sure what else to do with it.

In theory this seemed like a good method for getting each sound to play on time, but in reality it seems like the audio hardware isn’t able to keep up with the method that I’m using and is causing timing problems.

At the moment this is just a simple test in Java, but my goal is to create my app on mobile devices (Android, iOS, Windows Phone, etc)…however my current method isn’t even keeping perfect time on a PC, so I’m worried that certain mobile devices with limited resources will have even more timing problems. I will also be adding more sounds to it to create more complex rhythms, so it needs to be able to handle multiple sounds going simultaneously without lagging.

Another problem I’m having is that the max tempo is controlled by the length of the tone since the tones don’t overlap each other. I tried adding additional threads so that every tone that played would get its own thread…but that really screwed up the timing, so I took it out. I would like to have a way to overlap the previous sound to allow for much higher tempos.

One final point…my first attempt I used a larger buffer (loading multiple tones separated by silence) and played the buffer using SourceDataLine. However, the problem with this method was that I wasn’t able to know exactly when each tone played so that I could update the beat counter visual display. I do have a working example using this method, so if needed I could post my code for this as well.

Any help getting these timing and speed issues straightened out would be greatly appreciated. Thanks!

Here is my code…

SoundTest.java

import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
import javax.swing.event.*;

import java.io.*;
import javax.sound.sampled.*;

public class SoundTest implements ActionListener {
	static SoundTest soundTest;
	
	
	// ENABLE/DISABLE SOUNDS
	boolean playSound1	= true;
	boolean playSound2	= true;
	
	
	JFrame mainFrame;
	JPanel mainContent;
	JPanel center;
	JButton buttonPlay;
	JLabel[] beatDisplay;
	
	int sampleRate = 44100;
	long startTime;	
	SourceDataLine line = null;	
	int tickLength;
	boolean playing = false;
	
	SoundElement sound01;
	SoundElement sound02;
	
	public static void main (String[] args) {		
		soundTest = new SoundTest();
		
		SwingUtilities.invokeLater(new Runnable() { public void run() {
			soundTest.gui_CreateAndShow();
		}});
	}
	
	public void gui_CreateAndShow() {
		gui_FrameAndContentPanel();
		gui_AddContent();
	}
	
	public void gui_FrameAndContentPanel() {
		mainContent = new JPanel();
		mainContent.setLayout(new BorderLayout());
		mainContent.setPreferredSize(new Dimension(400,400));
		mainContent.setOpaque(true);
		
		mainFrame = new JFrame("Sound Test");				
		mainFrame.setContentPane(mainContent);				
		mainFrame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
		mainFrame.pack();
		mainFrame.setVisible(true);
	}
	
	public void gui_AddContent() {
		JPanel center = new JPanel();
		center.setOpaque(true);
		
		buttonPlay = new JButton("PLAY / STOP");
		buttonPlay.setActionCommand("play");
		buttonPlay.addActionListener(this);
		buttonPlay.setPreferredSize(new Dimension(200, 50));
		
		beatDisplay = new JLabel[2];
		
		beatDisplay[0] = new JLabel("~ SOUND 1 ~");
		beatDisplay[0].setPreferredSize(new Dimension(400, 50));
		beatDisplay[0].setHorizontalAlignment(SwingConstants.CENTER);
		beatDisplay[0].setOpaque(true);
		beatDisplay[0].setFont(new Font("Serif", Font.BOLD, 40));
		
		beatDisplay[1] = new JLabel("~ SOUND 2 ~");
		beatDisplay[1].setPreferredSize(new Dimension(400, 50));
		beatDisplay[1].setHorizontalAlignment(SwingConstants.CENTER);
		beatDisplay[1].setOpaque(true);
		beatDisplay[1].setFont(new Font("Serif", Font.BOLD, 40));
		
		center.add(buttonPlay);
		center.add(beatDisplay[0]);
		center.add(beatDisplay[1]);
		mainContent.add(center, BorderLayout.CENTER);
	}
	
	public void actionPerformed(ActionEvent e) {
		if (!playing) {
			playing = true;
			
			if (playSound1)
				sound01 = new SoundElement(this, "Sound1", 0, 800);
			if (playSound2)
				sound02 = new SoundElement(this, "Sound2", 1, 1200);
				
			startTime = System.nanoTime();
			
			if (playSound1)
				new Thread(sound01).start();
			if (playSound2)
				new Thread(sound02).start();
		}
		else {
			playing = false;
		}
	}
}

SoundElement.java

import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
import javax.swing.event.*;

import java.io.*;
import javax.sound.sampled.*;

public class SoundElement implements Runnable {
	SoundTest soundTest;
	
	
	// TEMPO CHANGE
	// 750000000=80bpm | 300000000=200bpm | 200000000=300bpm
	long nsDelay = 750000000;
	
	
	long before;
	long after;
	long diff;
	
	String name="";
	int soundNumber;
	int clickLength = 4100;	
	byte[] audioFile;
	double clickFrequency;
	SourceDataLine line = null;
	long audioFilePlay;
	
	// GUI
	int beatCount=1;
	boolean green=true;
	
	public SoundElement(SoundTest soundTestIn, String nameIn, int soundNumberIn, double clickFrequencyIn){
		soundTest = soundTestIn;
		name = nameIn;
		soundNumber = soundNumberIn;
		clickFrequency = clickFrequencyIn;
		generateAudioFile();
	}
	
	public void generateAudioFile(){
		audioFile = new byte[clickLength * 2];
		double temp;
		short maxSample;
		
		int p=0;
		for (int i = 0; i < audioFile.length;){
			temp = Math.sin(2 * Math.PI * p++ / (soundTest.sampleRate/clickFrequency));
			maxSample = (short) (temp * Short.MAX_VALUE);
			audioFile[i++] = (byte) (maxSample & 0x00ff);			
			audioFile[i++] = (byte) ((maxSample & 0xff00) >>> 8);
		}
	}
	
	public void run() {
		createPlayer();
		audioFilePlay = soundTest.startTime + nsDelay;
		
		while (soundTest.playing){
			if (System.nanoTime() >= audioFilePlay){
				play();
				gui_UpdateBeatCounter();				
				destroyPlayer();
				createPlayer();
				audioFilePlay += nsDelay;
			}
		}
		try { destroyPlayer(); } catch (Exception e) { }
	}
   
	public void createPlayer(){
		AudioFormat af = new AudioFormat(soundTest.sampleRate, 16, 1, true, false);
		try {
			line = AudioSystem.getSourceDataLine(af);
			line.open(af);
			line.start();
		}
		catch (Exception ex) { ex.printStackTrace(); }
	}
	
	public void play(){
		line.write(audioFile, 0, audioFile.length);
	}
	
	public void destroyPlayer(){
		line.drain();
		line.close();
	}
	
	public void gui_UpdateBeatCounter(){
		soundTest.beatDisplay[soundNumber].setText("" + beatCount);
		
		if (green){
			soundTest.beatDisplay[soundNumber].setBackground(Color.GREEN);
			green = false;
		}
		else{
			soundTest.beatDisplay[soundNumber].setBackground(Color.YELLOW);
			green = true;
		}
		
		if (beatCount == 4)
			beatCount = 1;
		else
			beatCount++;
	}
}

Check out this really useful article on Java timing for music and sound:

http://quod.lib.umich.edu/cgi/p/pod/dod-idx?c=icmc;idno=bbp2372.2007.131

[quote]REAL-TIME, LOW LATENCY AUDIO PROCESSING IN JAVA
Nicolas Juillerat, Stefan Muller Arisona, Simon Schubiger-Banz
ETH Zurich
Computer Systems Institute
[/quote]
Secondly, you will probably want to keep a SourceDataLine open and running constantly, for use as the output for a “mixer,” in the studio audio sense of the word mixer, not the Java sense. Have it play “silence” when there is nothing going else to play.

If you don’t want to write your own audio mixer, TinySound (on github, but also has posts here on JGO) has been written and already does this. With this library (pretty small, easy to read) you’ll be able to overlap sounds.

I wrote my own audio playback library also, but haven’t made it available, and added an “event system” for it a couple months ago. It checks a queue every frame for “trigger” commands. Something like this would be ideal for your use. To code this, you have to play your sounds one frame at a time, and check if on that frame count there’s another sound that is scheduled to be played. This is the only way I know to be really accurate, as Java does a good job of keeping sounds playing correctly (when you are at the stage of progressively loading the SourceDataLine), but is less accurate (doesn’t make real time guarantees) about issues that pertain to multiple threads or other issues mentioned in the article I cited.

There’s a trick for improving the accuracy of the sleep timer in windows. If you run it the command Thread.sleep(Long.MAX_VALUE) (on its own thread, of course!) the Windows OS will use its higher resolution clock (same one used for nanotime) for sleep and getmillis. But issues pertaining to the buffer used for playback and to thread management guarantees will still hinder accuracy.

Hi philfrei, thanks for the response! :slight_smile:

If at all possible I’d prefer not to use 3rd party libraries, because like I mentioned before my goal is to eventually create my app on mobile devices. I’m starting out and getting a working example in Java because I’m familiar with Java, then I’ll attempt to move over to mobile devices. I’d like to have my fingers on the inner workings of the low level audio stuff so I know how to translate it over onto different mobile platforms. I feel like I’d get totally stuck if I tried to use a 3rd party library because I’d have no idea how to make it work on other devices. I will certainly check out the code to see what I can understand and learn from it. I like the part about “small and easy to read”, so that’s encouraging. Some of the code I’ve looked at goes way over my head…I’m a musician long before a programmer. lol

I’ve thought about using “silence” in between playing the tones but I wasn’t sure exactly how to go about executing it. Theoretically I could do this in each of the threads I’m running by feeding SourceDataLine zeros if the play method isn’t called. But then I’m not sure how to have it respond quickly when data needed to be played while preventing it from emptying the buffer and then running into the same problem I had when I left the line open when there was no data to play.

But your suggestion is to use a single SourceDataLine instead and then create a mixer that would handle combing all of the sounds and silence? Is something like that feasible for a DSP newbie? This is my first time working with actual audio…I’ve done a few music related MIDI applets in Java but MIDI had its own sequencer, so timing MIDI wasn’t difficult like audio is turning out to be.

Thanks again!

I am also a musician first. I got into Java only four or five years ago, am self taught, and am also new to DSP. I’ve managed to teach myself a few things but also have had lots of help!

I also preferred to write things myself, in order to learn and understand, so I appreciate your wanting to do so as well. I’m in the process of working on the problem of converting my Java sound programs to iOS and Android, also. So maybe there is room to put heads together on this, down the road. (I think I’ve located the equivalent of the SourceDataLine output in both realms, so my framework code (all core Java) should run on these other system, but haven’t tried testing this yet.)

Do read the article I recommended! Many issues pertain to what Java can and can’t do. I prefer not to get into specifics about handling latency issues until you at least give it a go. It wasn’t the easiest read, but having it to refer to will be helpful.

A simple mixer is definitely feasible. The DSP being about as easy as DSP gets. The main task is the combining of PCM values, and that is just addition! Converting to and from bytes has to be done. There’s some tweaking to make sure you don’t overflow. Handling volume and panning can be kept very simple/stupid, at least initially.

Another excellent audio programmer will probably reply soon as well (he’s in Australia): nsigma. He has an excellent set of tools written and repeatedly given great advice as well as being generous in opening up his source code.

I just remembered, I started this thread when I started my second pass on a mixer:


The first post has a link with a jar with source code. The project has evolved a bit since then, and some of the decisions I made have not had unanimous support (!) but the code might still be helpful for you to look over.

It’s kind of funny when we say “self taught”, since we have the best teacher you can ask for at our finger tips…the Internet. :slight_smile: I did go to university for programming, but it was a “liberal arts” college that was mostly known for theater, so it was definitely NOT a tech school by any stretch of the imagination. LOL I feel like I’ve learned far more online than I learned in college as most of the teachers were “so-so”. I created a music related MIDI applet for my final project and my teachers couldn’t even help me with it since none of them worked with MIDI…after a couple months of trying to figure it out, got lucky and found an example that was similar enough to what I needed and I was able to make it work. I uploaded my finished app online and since you’re also a musician, I’ll go ahead and share the link here…it’s a random chord progression generator that creates chord progressions using music theory and then plays them via MIDI.
http://www.guitarforbeginners.com/tekker/Chord%20Progression%20Generator%20V3/ProgressionGenerator.html

What sound programs have you made? Have you made any kind of tools for musicians?

I’ve also been looking at finding the mobile equivalents to SourceDataLine as well, I believe AudioTrack is the equivalent for Android…iOS has so ridiculously complicated that I actually haven’t found the equivalent there yet. I’ve looked at several tutorials for iOS sound and it is insane the amount of overhead you have to do when working with lower level audio. Ugh!

When mixing the sounds, wouldn’t you also need to have some kind of DSP limiter to make sure you don’t clip when combining multiple sounds?

I will definitely take a look at the article you mentioned and the code you posted. I have downloaded the TinySound source code and quickly glanced through it, but I’ll definitely have to dedicate more time to really studying it to see what all I can get out of it.

Thanks again for your help, you’ve definitely given me some new directions to look into and creating my own audio mixer sounds intriguing for sure…got more studying to do there!

Good point!

[quote] I did go to university for programming, but it was a “liberal arts” college that was mostly known for theater, so it was definitely NOT a tech school by any stretch of the imagination. LOL I feel like I’ve learned far more online than I learned in college as most of the teachers were “so-so”.
[/quote]
I was at UC Berkeley as a Music Major, but took “breadth” courses in Comp Sci and in Psychology (emphasis Cognitive & Psychoacoustics–worked at a Hearing Lab for work-study). Did a bit of composing and sound design for the Dance Department, and the Drama Department and the Educational TV Office. The CompSci teachers were great, no complaints. But the material I chose: Pascal, Assembly (via PDP-11), Logic Design (e.g., TTL chips to make a Pong game!), are all pretty out of date now. I was there in the 1970-s & 1980’s. (Dating myself!)

[quote]I created a music related MIDI applet for my final project and my teachers couldn’t even help me with it since none of them worked with MIDI…after a couple months of trying to figure it out, got lucky and found an example that was similar enough to what I needed and I was able to make it work. I uploaded my finished app online and since you’re also a musician, I’ll go ahead and share the link here…it’s a random chord progression generator that creates chord progressions using music theory and then plays them via MIDI.
http://www.guitarforbeginners.com/tekker/Chord%20Progression%20Generator%20V3/ProgressionGenerator.html
[/quote]
Cool! I’m looking forward to checking it out. By the way, you might check out krasse’s related work, posted on JGO.
http://www.java-gaming.org/topics/some-reasonable-output-from-my-generative-music-system/27309/view.html

[quote]What sound programs have you made? Have you made any kind of tools for musicians?
[/quote]
I made a Theremin applet, with a real-time touchpad control for volume and pitch.
http://www.java-gaming.org/index.php/topic,24646.0.html
I made another touchpad that let’s you play sounds forwards or backwards at varying speeds. Another tool (based on a friend’s idea) makes semi-random sine waves that are harmonically related (whole-number-ratios), creating a kind of evolving cloud–sounds spacy–involved figuring out envelopes.

After getting the basic mixer working, I wrote my own version of the Java Clip that can play at varying speeds, but also can support multiple cursors, so you can do things like trigger multiple gunshots and not have the newest shot squelch the previous. I wrote an improved tool for looping these clips, in that one can specify that the ends overlap or not, and cross-fade or not.

I have yet another Clip variant that plays random slices of a sound. There’s an example posted somewhere here, where a 4-second recording of a brook is broken into something like 200msec random slices that are played with a bit of overlap, creating the illusion of a continual, non-repeating sound from a small source file.

Related, a “SoundField” construction that lets one select a number of Clips and play them semi-randomly, at a certain probability for given time period. This was also used in the above link, but the current version has a much more efficient way of generating the random bells than using Perlin’s algorithm.

There was a soundscape sort of piece involving wav files from an old analog synth that I own. There are five volume sliders that can be manipulated or left to drift randomly. I was attempting to use ogg compression, initially, but couldn’t get it to work without memory leakage. So the file is a bit bulky as the resources are raw wav files.
http://hexara.com/ffw/ffway.jar

I have been delving more deeply into frequency modulation, as I am a big fan of the Yamaha DX7 (vintage 1980’s synth, again dating myself). I’ve written an FM synthesizer, and have converted something like 8 of the DX7 patches to real-time Java synthesis, some bells, and pads and a keyboard so far.

I have an echo tool working, and a flanger/chorus algorithm working. Haven’t gotten to filters yet, nor reverb.

The “sound field”, FM and flanger can be heard in the puzzle game project I started but have yet to finish: Hexara.

Making a listing like this (first time in a long time!) I see that I’ve been at it for a while now, but also have a definite tendency to R&D over completing and bringing products to market. I am in awe of programmers like nsigma who have accomplished so much that is practical and commercially/publicly available.

[quote]I’ve also been looking at finding the mobile equivalents to SourceDataLine as well, I believe AudioTrack is the equivalent for Android…
[/quote]
Bingo! I agree. I just haven’t attempted a proof-of-concept example yet.

[quote]…iOS has so ridiculously complicated that I actually haven’t found the equivalent there yet. I’ve looked at several tutorials for iOS sound and it is insane the amount of overhead you have to do when working with lower level audio. Ugh!
[/quote]
A good friend of mine is a contract iOS programmer, and he recently said he found the hook. But this is a case where we have to successfully port Java to Object-C. There are a couple tools that seem plausible, one that goes directly, another path that uses an intermediate JavaScript conversion. Whether the code I wrote is basic enough Java to allow this conversion and runs efficiently, remains to be seen! I don’t know enough about this hook yet to pass on info to you. I only heard about it from him two weeks ago and am frustrated that I have to work on other things and can’t get to this immediately. The electronic tanpura project I’m working on (also stalled) has some potential to be a neat little iOS app.

[quote]When mixing the sounds, wouldn’t you also need to have some kind of DSP limiter to make sure you don’t clip when combining multiple sounds?
[/quote]
Yes. But simple can be effective for 90% of the situations: just use a Math.min and Math.max function. That prevents overflow. As to clipping, I use my ears for diagnosis, and just turn down the volume of the contributing sounds. Again, crude but simple. A limiter/compressor is not something I’ve dealt with yet. Like filtering, it involve understanding things like convolution (I have a tenuous grip on this–understand the basic concept but weak on appropriateness of where/how to implement), is a bit more advanced than I can handle.

[quote]I will definitely take a look at the article you mentioned and the code you posted. I have downloaded the TinySound source code and quickly glanced through it, but I’ll definitely have to dedicate more time to really studying it to see what all I can get out of it.

Thanks again for your help, you’ve definitely given me some new directions to look into and creating my own audio mixer sounds intriguing for sure…got more studying to do there!
[/quote]
It’s endless, and easy to get continually distracted! Good that you have a concrete project (the metronome) to help provide focus.

Ok, so I’ve been brainstorming about how to go about this and I have an idea in mind, but I’m not sure it’s 100% there yet. So I have just a few questions to make sure I understand this correctly…

Let’s say I did something like this pseudo-ish code:

buffer = new byte[1000];

while (run){
	// LOAD BUFFER
	for (int i=0; i<buffer.length; i++)
		buffer[i] = getNextAudioByte();

	// WRITE LINE
	line.write(buffer, 0, buffer.length);
}

Q1) Does writing to the SourceDataLine (“line”) operate on its own thread?
In this example, after “dropping off” the audio data on the line, would the program immediately go back to filling the next buffer while the audio is still being played on the line?

Q2) Does loading the buffer and writing to the line execute in exactly the same amount of time?
I don’t think it does. I think filling the buffer would execute way faster than writing the audio to the line because it seems like the speed of writing to the line would be limited by the sample rate. For example, with a sample rate of 44.1kHz each sample in the buffer would be played every 1 / 44100 = 22675ns, yet loading the buffer would operate significantly faster than this.

Q3) Would my example essentially crate a backlog of audio waiting to be written to the line?
If writing to the line operates on its own thread and loading the buffer executes faster than playing the buffer, it seems like the buffer would simply overrun the line with audio data and it would just keep getting backed up further and further. Thus making the program not very responsive if it tries to play something “immediately” yet there is still a lot of audio data in front of it to be played.

Q4) My idea is to have the program itself essentially operating “behind” the audio that is currently playing by the time that it takes to play one buffer (…I hope that makes sense), so would I need to slow down the process of loading the buffer in order to have it keep pace with playing the buffer?

I hope these questions make sense…its way too late and I’ve been frying my brain over this!

I’ll quickly reply to your latest response philfrei before my brain totally shuts off for tonight. lol

You certainly have quite an impressive list of audio projects there. It’s been a while since I’ve tried to load any java applets via my web browser, but is there currently some kind of problem with the java plugin? I’m seeing on the firefox page that they’ve blocked the current version because it has security vulnerabilities. I was trying to load your Theremin applet, but no luck. :frowning:

For Android’s AudioTrack, I found an Android metronome example that uses AudioTrack. It uses the identical write method that SourceDataLine uses, same parameters and everything.
http://masterex.github.io/archive/2012/05/28/android-audio-synthesis.html

I think I know what you’re referring to in terms of “the hook” for porting Java to mobile. I looked into a couple tools as well…the first one was Appcelerator Titanium, which uses JavaScript. The other one is Codename One, which uses Java (it has its own Java API that you use) and that’s the one I think I’m going with. Codename One is an impressive beast! It actually generates native code for most mobile platforms…Android, iOS, Windows Phone, and J2ME/RIM. However, one drawback is that it does not support low level audio, but it does allow you to write methods in native code so you can add additional low level functionality that Codename One doesn’t support. There are a bunch of videos by Shai Almog (the creator of Codename One) on Youtube.

Regarding the limiter, I’d prefer not to simply turn down the sounds as that might be strange to have the rhythms changing volume as more tones are added. I’d like them to stay at the same volume and simply prevent it from clipping, but as you said…that gets into the more complex DSP category there. But maybe there’s a way to plug in a simple-ish equation that would do the trick? It doesn’t have to be incredible and transparent sounding or anything since I’m not mixing complex music, basically just clicks and cowbells…I gotta have more cowbell in my metronome app! :smiley:

Thanks again for all your help!

I’m not entirely sure of the implementation, but I think SourceDataLine and TargetDataLine both rely on a BlockingQueue or something very similar. Basically, the audio loop proceeds in its own thread. The write method for SourceDataLine blocks until the underlying code has had a chance to process the previous buffer’s data and it is able to accept the new buffer array of data.

Yes, the loading of data into the buffer array should be significantly faster than the rate at which the sound data is played! The audio while loop basically should be spending most of its time in a blocked state (at the point where SourceDataLine is writing). If the loading of the buffer array ever takes longer than the write command blocks, you will probably hear drop-outs.

The api for SourceDataLine’s write method refers to the blocking:
http://docs.oracle.com/javase/7/docs/api/javax/sound/sampled/SourceDataLine.html

There are methods to check availability on the line, but there is usually no need to do this. Mostly we just keep shovelling buffer arrays into the line as fast as it is willing to accept them.

This should answer all 4 of your questions!


Yes, the whole deal with applets is messed up. I gave up trying to deal with it. I leave it to the user to figure out what they have to do with their browser & plugins to allow the applets to play. As far as I know, my programs are okay to trust. They’ve just been sitting on my website and protected from tampering by whatever security the provider (startlogic.com) has going.

Some of the links are downloads of jars. You might have more luck with them. Should I dig up the link to the theremin applet’s jar file?
Here it is:
http://Hexara/VSL/JThereminApplet.jar
But you will probably have to run it in your IDE as an Applet, not as an Application. Be sure to try the Echo effect. I’ve been meaning to add a control for vibrato, but haven’t managed this yet. Vibrato via a mouse is hard to make sound good.

Thanks again philfrei! Yes, that does answer my questions…and naturally created more questions. :persecutioncomplex:

Does the blocking occur during the first pass of writing audio to the line or does it block if it tries to write additional data to the line? In other words, in my pseudo code from before:

buffer = new byte[1000];

while (run){
   // LOAD BUFFER
   for (int i=0; i<buffer.length; i++)
      buffer[i] = getNextAudioByte();

   // WRITE LINE
   line.write(buffer, 0, buffer.length);
}

Would this code…
a) Load the buffer and wait while that data is being written to the line.
b) Load the buffer, drop the data off on the line, go back load the next buffer, and then wait until the previous buffer finished writing on the line.

I know it seems like splitting hairs since this operation would likely be very fast, and if I were only dealing with audio then it wouldn’t matter…but for syncing the beat counter visual display and the audio I need to know how much of a lag there is from when my program says to play a sound and when that sound actually ends up playing so I know precisely when to update the visual display.

Actually, I may be able to test when it blocks by using a really long buffer (like 10 seconds worth of audio) to slow the write process down and then printing out the system time just before it calls the load buffer method. That would hopefully tell me when it is blocking.

Also, I’ll try downloading your Theremin applet when I get home. Those things are certainly odd instruments. lol I got to try one in a music course I took in college.

You seem to be under the impression that the visual feedback is instant while the audio has significant latency. It’s actually the other way around. At 60Hz your visual update interval is 16.6ms, but the driver will have most likely 2 or 3 frames it is still working on, and then the data has to be transfered to the monitor, adding to the latency. The visual latency is therefor a multiple of 16.6ms and then some, while the audio is pretty damn low latency with sub-millisecond latency. The only issue is that the OS scheduler is not realtime, so we need some buffering, but anything over 20ms worth of samples seems excessive.

Luckily, your brain compensates for most latency, or we’d all be confused by realworld latency caused by the speed of sound. If visual cause matches only barely with the audio, your brain will match the two up and tell your conscience it was a single event. This calibration happens within a second, and can even be exploited to make it seem like the effect (audio) happened before the cause (visual) if at the time your brain was tuned for high latency. But I digress.

Long story short: you actually need a fair amount of artificial audio latency for the visual side of the event to be in sync. It’s fair to target 30-40ms delay between an ingame event and all feedback you get from your system.

Hi Riven, thanks for the reply!

I wasn’t actually assuming the visual latency would be “instant”…but at the moment my focus is on the audio end of the program such as determining how I’m going to be mixing and playing my audio as well as trying to determine the precise moment that the audio will be playing and how far behind my program is operating when it says “play sound” and it actually plays that sound. I will eventually be doing the same thing for the visual display, but I haven’t gotten to the point figuring out my visual method yet. Right now I’m just updating a JLabel with some text and color, but I’m sure that will change once I start actually working on the GUI.

Also, it seems like an audio buffer with even 1000 samples isn’t nothing…1/44100*1000 = 22.6ms. So in regards to my previous question if it ends up blocking when the second buffer tries to add data, then my program would respond almost two buffers behind which could be around 40+ms. Which may actually put it in nearly perfect sync if the visual latency is about the same as this…I totally meant to do that! ;D

It’s great to know what kind of range I should try to hit with the audio and visual synchronization. My ultimate goal is to develop this app for different mobile devices and I’m sure the range of latencies of visual display and audio playback will vary so it’ll likely be impossible to get all of them “perfect”.

Thanks again!

philfrei, I’ve tried running your Theremin applet within a couple IDE’s (jGrasp and Netbeans)…jGrasp wouldn’t even open it at all and while Netbeans did open it, the option to run it was grayed out. So still no luck on the Theremin. :frowning:

Thanks for elucidating this, Riven.

Tekkerue, as to the question about blocking: if the BlockingQueue is asked to do something that it can do, then the code executes. If it can’t do it, then it blocks. So, I pick your scenario (b). But the lack of real time guarantees in Java makes this moot if you want metronomic accuracy.

The article I cited earlier goes into this: “Real Time Low Latency Audio Processing”. It’s going to be vital that you understand the nature of Java’s lack of real time guarantees if you are going to use javax.sound.sampled for this.

For accuracy on par with Java’s MIDI sequencer, the only way I know of is to calculate the actual sound frame a note should play, count frames, and start the note on that frame (when the while loop is processing that frame). The moment in time at which sound data is loaded into the buffer is too variable relative to when it is actually heard to do otherwise.

Example: 44100 fps stereo, 16-bit (“CD Quality”). Let’s say you want the metronome to play 120 bpm. I would calculate the interval in frames ( = 22050 ) and have the while loop count frames as it processes the buffers of PCM bytes, and start the metronome click sound on that actual frame. So, in the middle of some buffer when you get to frame number N * 22050, you start the loading of the PCM data for the click sound.

I’m guessing you may have to hear for yourself before taking the trouble to use the frame counting method. I first ran into the ‘lack of real time guarantees’ issue with the Theremin, trying to get the sound to accurately track the mouse movements via a MouseMotionListener. My rude awakening (and considerable resistance to the truth) can be viewed on an old JGO thread. Eventually I got a reasonable solution.

Java’s MIDI package and sequencer can provide the needed accuracy–so maybe you should just use that, and schedule Midi ‘control messages’ to trigger your visuals.

Doesn’t NetBeans or your other IDE have a way to run Applets? I use Eclipse.
Doh! :clue: I forgot to give you the entry point:
Here’s the HTML code that is used for that jar:


<script src="http://www.java.com/js/deployJava.js"></script>
<script> 
    var attributes = {code:'jTheremin.ThereminApplet.class',
                      archive:'JThereminApplet.jar',
                      width:810, height:400} ;
    var parameters = {fontSize:10} ;
    var version = '1.6' ;
 
    deployJava.runApplet(attributes, parameters, version);
</script>

You know, there is the program “appletviewer.exe” that is in the bin file of your JDK, probably.
http://docs.oracle.com/javase/7/docs/technotes/tools/windows/appletviewer.html


I haven’t tested this, but I think if you put the above html in a valid html file, and put it in the same directory as the jar, you can run it via a command line.

Hi philfrei,

I read the article you provided, so I do understand what Java does to muck with the timing…but I also know that people have somehow been able to make it work. :slight_smile: If I recall correctly, Y-Metronome was made with Java and I use this one quite regularly. I also have a few different metronome apps on my Android phone that work great, and as you know Android is done with Java. So it is possible to get great results using Java.

In terms of “counting frames”, isn’t that essentially what I’m doing by determining the time in nanoseconds that the sound should start playing? I’m specifically calculating the point that the audio should play…I guess I’m not totally sure what you mean by “frame”…?

I did look into the MIDI route, but unfortunately it doesn’t appear that MIDI sequencers are supported on mobile OS’s. Not even on Android. It appears there is very basic MIDI support in that you can play a MIDI file, but doing the kind of low level on the fly MIDI synthesis is not supported for mobile. :frowning:

I do have an appletviewer, so I will look into that soon…hopefully tomorrow.

BTW, have you had any luck getting my Random Chord Progression Generator to load? I can’t load that either via the web browser.

…D’oh, never mind! I see what you mean by “frame” as you have 44100fps. I’m too used to calling them SAMPLES instead of frames. lol

Ok, so if I understand what you’re saying…

You keep track of every frame that plays by incrementing a frame counter by the size of the buffer every time you write to the line.
Then instead of doing math on an ever incrementing number of “nanoseconds” I would simply do that for an ever incrementing number of frames.

And since the GUI part wouldn’t be running on “frames” per say…I could convert the frames to nanoseconds with 1 frame = 22675ns at 44100fps.

Am I on the right track here?..

[quote]Am I on the right track here?..
[/quote]
Yes,

[quote]BTW, have you had any luck getting my Random Chord Progression Generator to load? I can’t load that either via the web browser.
[/quote]
I haven’t tried yet. Something about the way I’m wired, I will burn all sorts of time writing answers to questions and then get even further behind in everything else I am doing. :stuck_out_tongue: Next window of opportunity will be Sunday. Will be looking forward to it!

Woohoo! Looks like I have my plan forward…now off to write some code! Thanks again for your help and I’ll stop taking up all your time so you can get caught up on your own stuff now. :wink:

IT’S WORKING!!!


http://www.sherv.net/cm/emo/funny/2/banana.gif

http://www.sherv.net/cm/emo/funny/2/banana.gif

http://www.sherv.net/cm/emo/funny/2/banana.gif

http://www.sherv.net/cm/emo/funny/2/banana.gif

http://www.sherv.net/cm/emo/funny/2/banana.gif

Just wanted to let you know that you are a genius philfrei! :slight_smile: Using the method you described everything is working fantastically now. I even tried it at 2000bpm and it works!..no that’s not a typo! ;D It does glitch if I try to do other things while it’s running like minimize/maximize windows, etc but if I just leave the computer alone it plays perfectly at 2000bpm. Not that anyone would ever have a need for 2000bpm, but that is still way awesome it can do that! lol I could probably make it stop glitching completely if I increase the time between when it tries to load the sound and when it actually plays. I had to adjust it so that it played perfectly even at slower tempos as when I first got it playing it was taking to long to load the sounds so they weren’t playing when they were supposed to.

And just to let you know how long I have been trying to create this metronome application…I recently came across an old thread of mine over on JavaRanch from 4 years ago when I was trying to make this metronome app using MIDI instead of audio…but I never got anywhere with it. Now it’s working using audio and it should be able to work on mobile devices as well (in an ideal world of course! lol)…

I’ve had the idea to create the ultimate “one metronome to rule them all” for a while, but it seems like I’ve run into road blocks at every turn just trying to get well timed and consistent clicks that didn’t glitch or drop out or whatever…and thanks to you, it works great now! Seriously, you have no idea how ecstatic I am right now!!

Thank you again for all your help!!

More happy dance…


http://www.sherv.net/cm/emo/funny/2/banana.gif

http://www.sherv.net/cm/emo/funny/2/banana.gif

http://www.sherv.net/cm/emo/funny/2/banana.gif

http://www.sherv.net/cm/emo/funny/2/banana.gif

http://www.sherv.net/cm/emo/funny/2/banana.gif

Hello again!

So, I just tried running the metronome on a desktop, recording it on a laptop, then checking the timing using my recording software…and I found that my metronome is speeding up ever so gradually. I also tried going back to a much simpler and straight forward metronome version I made from before (see code below) and tried recording that, but it too speeds up gradually by nearly the exact same amount (they were only a few frames apart after 10 minutes).

After exactly 10 minutes the simple version was 323 frames ahead and my current version was 314 frames ahead, these are approximately 7ms ahead of where the click should be. This is strange because I figured that if the timing was going to be off at all it would be slightly behind the beat if the process of loading the buffer added a little bit more time…but instead it is ahead of the beat and I can’t figure that one out.

Unless I’m missing something, this metronome should be sample accurate and should not be drifting at all. I used 60bpm so a tone should be added exactly every 44100 samples thus there shouldn’t be any cascading rounding errors or anything. I’m not seeing anything that I’m doing wrong…so could this be a hardware problem or a JVM problem?

I would really like to be able to record for an hour (or longer) and have it match the tempo grid PERFECTLY in my recording software the entire time. Is there anything more I can do to guarantee perfect timing or is this as good as it can get using Java?

Thanks!

import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
import javax.swing.event.*;

import java.io.*;
import javax.sound.sampled.*;

public class TimingCheck implements ActionListener {
	static TimingCheck timingCheck;

	JFrame mainFrame;
	JPanel mainContent;
	JPanel center;
	JButton buttonPlay;
	
	int bpm = 60;
	double frequencyOfTick = 800;
	int sampleRate = 44100;
	int bufferSize = 2000;
	
	byte[] buffer;
	int tickBufferSize;
	double[] tickBuffer;
	SourceDataLine line = null;	
	int tickLength = 4100;
	
	boolean playing = false;
	
	public TimingCheck(){
		tickBufferSize = (int)((60 / (double)bpm) * (double)sampleRate);
	}
	
	public static void main (String[] args) {		
		timingCheck = new TimingCheck();
		
		SwingUtilities.invokeLater(new Runnable() {
			public void run() {
				timingCheck.guiCreateAndShow();
			}
		});
	}
	
	public void guiCreateAndShow() {
		guiFrameAndContentPanel();
		guiAddContent();
	}
	public void guiFrameAndContentPanel() {
		mainContent = new JPanel();
		mainContent.setLayout(new BorderLayout());
		mainContent.setPreferredSize(new Dimension(500,500));
		mainContent.setOpaque(true);
		
		mainFrame = new JFrame("Timing Check");				
		mainFrame.setContentPane(mainContent);				
		mainFrame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
		mainFrame.pack();
		mainFrame.setVisible(true);
	}
	public void guiAddContent() {
		JPanel center = new JPanel();
		center.setOpaque(true);
		
		buttonPlay = new JButton("PLAY / STOP");
		buttonPlay.setActionCommand("play");
		buttonPlay.addActionListener(this);
		buttonPlay.setPreferredSize(new Dimension(200, 50));
		
		center.add(buttonPlay);
		mainContent.add(center, BorderLayout.CENTER);
	}
	
	public void actionPerformed(ActionEvent e) {
		if (!playing) {
			playing = true;
			timingCheck.createPlayer();
			timingCheck.generateTick();
			new Thread(play).start();
		}
		else {
			playing = false;
		}
	}
	
	public void generateTick(){
		tickBuffer = new double[tickBufferSize];
		for (int i = 0; i < tickLength; i++)
			tickBuffer[i] = Math.sin(2 * Math.PI * i / (sampleRate/frequencyOfTick));
		for (int i = tickLength; i < tickBufferSize; i++)
			tickBuffer[i] = 0;
	}
	
	public void createPlayer(){
		AudioFormat af = new AudioFormat(sampleRate, 16, 1, true, false);
		try {
			line = AudioSystem.getSourceDataLine(af);
			line.open(af);
			line.start();
		}
		catch (Exception ex) { ex.printStackTrace(); }
	}

	public Runnable play = new Runnable() { public void run() {
		buffer = new byte[bufferSize];
		int b=0;
		int t=0;
				
		while (playing) {
			if (t >= tickBufferSize)
				t = 0;
			
			short maxSample = (short) ((tickBuffer[t++] * Short.MAX_VALUE));
			buffer[b++] = (byte) (maxSample & 0x00ff);			
			buffer[b++] = (byte) ((maxSample & 0xff00) >>> 8);
			
			if (b >= bufferSize) {
				line.write(buffer, 0, buffer.length);
				b=0;
			}
		}
		
		destroyPlayer();
	}};
	
	public void destroyPlayer(){
		line.drain();
		line.close();
	}
}

You are talking about less than 1/100th of a second after 10 minutes, when comparing the timing mechanisms of two entirely different pieces of software.

Maybe this is normal, I don’t know. Perhaps this is why we have things like SMPTE code, and other mechanisms for maintaining synchronization?

Is tickBufferSize calculating out to what you expect? Bouncing back and forth between doubles and ints is worrisome. You only need to have the denominator of the division be a double (I don’t know if making the sample rate a double adds to the error or not). Maybe test for a simple tempo like 60bpm or 120bpm, and eliminate this calculation – hard code in 44100 or 22050 as your tickBufferSize and see if the error still occurs.

It occurs to me the (int) cast at the end could be causing a slight truncation in the tickBufferSize, which would create a slightly faster than expected rendition.

How many seconds in 10 minutes? 600? How many tickBuffers in 10 minutes? (Depends on tempo.) Are you testing a tempo (e.g., slower than 60bpm, or some “awkward” number) that would create the shown error if the tickBufferSize was off by one due to the int cast?

Another thought: take a piece of music, a wav that you have in your DAW, and play it back with Java, and record that in the DAW and compare. Or compare with other software that can play back.