manipulating SourceDataLine

Newbie & Debugging Questions
#1

Hello all,

I have some songs I am playing using SourceDataLine and I was wondering if anyone knows if there is a way to start from a specific point in the song. I have a song that has a couple bars of intro before going into the main part, and I want to play the whole song once, then loop starting just after the intro, instead of playing the entire intro again. Here is the class I’m using for reference. Sorry for the wall of code! Also, any constructive criticism of my code is welcome, as I am new to Java’s sound API and coded this as a first effort, referencing members of this site, javadocs, and stackoverflow.com.

package com.noah.breakit.assets;

import java.net.URL;

import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.FloatControl;
import javax.sound.sampled.SourceDataLine;

import com.noah.breakit.util.Util;

public class Song {

	// music credit to sketchylogic
	public static final Song titlesong = new Song("songs/titlesong.wav");
	public static final Song playfieldsong = new Song("songs/playfieldsong.wav");
	public static final Song briefingsong = new Song("songs/briefingsong.wav");
	public static final Song gameoversong = new Song("songs/gameoversong.wav");

	private static boolean playing;
	private boolean looping;
	private boolean killThread;

	private URL url;

	private AudioInputStream ais;
	private AudioFormat baseFormat;
	private AudioFormat decodeFormat;
	private DataLine.Info info;
	private SourceDataLine sdl;
	private FloatControl gainControl;
	
	private String name;

	private Song(String filename) {
		name = filename;
		
		try {
			url = this.getClass().getClassLoader().getResource(filename);
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

	public synchronized void loopSong() {		
		SoundThreadPool.execute(new Runnable() {
			public void run() {
				while (playing){
					System.out.println(name + " waiting...");
				}//wait for any other song threads to finish executing...
				
				playing = true;
				looping = true;
				
				while (looping)
					play();
				playing = false;
			}
		});
	}

	public synchronized void playSong() {
		playing = true;
		SoundThreadPool.execute(new Runnable() {
			public void run() {
				play();
				playing = false;
			}
		});
	}

	private void play() {
		try {

			ais = AudioSystem.getAudioInputStream(url);

			baseFormat = ais.getFormat();
			decodeFormat = new AudioFormat(AudioFormat.Encoding.PCM_SIGNED, baseFormat.getSampleRate(), 16,
					baseFormat.getChannels(), baseFormat.getChannels() * 2, baseFormat.getSampleRate(), false);

			info = new DataLine.Info(SourceDataLine.class, decodeFormat);

			sdl = (SourceDataLine) AudioSystem.getLine(info);
			sdl.open();
			gainControl = (FloatControl) sdl.getControl(FloatControl.Type.MASTER_GAIN);

			sdl.start();
			int nBytesRead = 0;
			byte[] data = new byte[sdl.getBufferSize()];
			int offset;
			while ((nBytesRead = ais.read(data, 0, data.length)) >= 0) {
				offset = 0;
				System.out.println(name + " reading...");
				while (offset < nBytesRead){
					System.out.println(name + " writing...");
			    	offset += sdl.write(data, 0, nBytesRead);
			    }
				if(killThread){
					System.out.println(name + " killing...");
					break;
				}
				System.out.println(name + " reading...");
			}
			
			System.out.println(name + " draining, stopping, closing...");
			
			sdl.drain();
			sdl.stop();
			sdl.close();
			
			System.out.println(name + " drain, stop, close complete!");
			
			if(killThread){
				looping = false;
				killThread = false;
			}
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

	public void adjustGain(float gain) {
		if (gainControl == null) return;
		float value = Util.clamp((gainControl.getValue() + gain), gainControl.getMinimum(), gainControl.getMaximum());
		gainControl.setValue(value);
	}

	public void setGain(float gain) {
		gainControl.setValue(Util.clamp(gain, gainControl.getMinimum(), gainControl.getMaximum()));
	}

	public boolean atMin() {
		return gainControl.getValue() == gainControl.getMinimum();
	}

	public boolean atMax() {
		return gainControl.getValue() == gainControl.getMaximum();
	}

	public boolean isPlaying() {
		return playing;
	}

	public boolean fadeToBlack() {
		adjustGain(-0.4f);
		if (atMin()){
			killThread = true;
			System.out.println(name + " killThread set to true...");
		}
		return atMin();
	}
}
#2

When reading a file via AudioInputStream, I think one has to pretty much start at the beginning and go until the end, or until quitting, whichever comes first.

It is possible to read and throw away input data until you get to the desired starting point. You’d have to count elapsed sound frames in order to know when to switch over to actually streaming data to the SourceDataLine instead of throwing it away.

Another thing is to just take the cue itself and edit it down to exactly where you want to start it. I use Audacity for this sort of thing. If you don’t intend to use the first few seconds, clipping off the data will reduce the size of the file which is also a good thing.

Since you want to repeat the cue, you could either append the repeat, again using Audacity, or programmatically put in place a LineListener to determine when the cue ends and use that notification to start another iteration.

Simplest, though, if there is enough RAM to hold the entire cue, would be to go back to making the cue a Clip. Clips allow the programmer to set the “playback head” to any starting point as well as allowing looping.

#3

If this were my project I would just have two separate audio files, one with the intro and the other without. Then just start looping the second after the first has finished.

Either that or like philfrei suggested, use a clip

#4

“SourceDataLine” is works like Data buffer
https://docs.oracle.com/javase/7/docs/api/javax/sound/sampled/SourceDataLine.html
it have size depend on “AudioFormat”
FormatSize * channels == time (1 sec or ms not remember sure)

so you need load file “wav”
offset header data (header data needed for creating AudioFormat)
then you can send any data in any order as you want to SourceDataLine

simple remember: to not broke this loop - its important to be so (because of multy thread)


while (offset < nBytesRead){
	offset += sdl.write(data, 0, nBytesRead);
}

up:
i reread comments about SourceDataLine.write
and remembered that hi blocked thread until fill full data so “while” in theory can be skipped

#5

Well, I tried separating the intro from the body and playing the intro, then the body, but the problem I’m having with this approach is that they are separate threads, and syncing them correctly seems impossible, especially with that kind of precision. In fact, when I tried it, they just played on top of each other. I used the same class as posted previously, but with this extra method:

public synchronized void playIntroLoopBody(Song body) {
		playing = true;
		SoundThreadPool.execute(new Runnable() {
			public void run() {
				while (playing) {
					System.out.println(name + " waiting...");
				} // wait for any other song threads to finish executing...

				play();
				playing = false;
			}
		});
		body.loopSong();
	}

So, since boolean playing is static, the new thread is SUPPOSED to spinlock and wait for the other thread to finish, but it totally didn’t work! They just overlapped each other and it sounded really bad. Thread concurrency is really advanced, so if anybody knows a better way to make one thread wait for the other, and play seamlessly, I’m all ears. I’m doubtful, though, since you have to drain the line, stop it, close it, then wait for the thread to finish executing, then wait for the thread to be removed from the thread pool.

Other than that, I think using a Clip would be better for something like this, since at least it would all be on the same thread.

Also, i would be interested to know if anybody has experimented with MIDI playback, since really this is probably the best format for tight looping and controlling the flow of a song. Thanks always for your guys help!

#6

Just a quick fix: I placed a method call in the wrong scope. After fixing, the intro and the body no longer play on top of each other, but it is still seamy. I’ll play around with the Audacity edits and see if I can get it to transition more smoothly. Here is the updated code:

	public synchronized void playIntroLoopBody(Song body) {
		playing = true;
		SoundThreadPool.execute(new Runnable() {
			public void run() {
				while (playing) {
					System.out.println(name + " waiting...");
				} // wait for any other song threads to finish executing...

				play();
				playing = false;
				body.loopSong();//this was in the wrong scope
			}
		});
		//it was right here...
	}
#7

I think using a LineListener is going to be both more accurate and more efficient than polling. But if you are trying to make two files play perfectly contiguously and seamlessly, I don’t know if that is going to be possible without frame counting.

If you have the midi data I assume that approach can work. You then have to decide whether to provide you own samples or rely on those provided by sound cards. I’ve only just started working with Java midi myself, so I can’t offer much in the way of advice on that topic.

#8

Yes, I need them to play contiguously and seamlessly because the seam between the intro and the body is very obvious and jarring.

The same goes for the looping of the body: in Audacity, the body loops seamlessly and is completely transparent. However, looping the SourceDataLine requires draining and closing the old DataLine, shutting down the thread, starting the new thread, and creating, opening, and starting the new DataLine. The time it takes to do all that is enough to create a brief moment of silence, which creates a sound gap and throws the rhythm off just enough to be very noticeable.

I am gong to try a LineListener and see if it makes any difference. I’m not sure how to do frame counting but i’ll research it.

If all else fails, I’ll just ditch the intro portion completely, hard code a fade to black into the wav file using Audacity, and then just loop it normally. Not the ideal solution, but at least it will be listenable.

#9

It is possible to run two SourceDataLines at the same time from the same file, but each requires its own AudioInputStream instance.

Theoretically, if you put a LineListener on one and have it launch the other SDL, many of the intervening tasks you mention can occur independently, on their respective threads, and not contribute to a gap. But there will likely still be some sort of gap. I’ve not tried this myself except in very forgiving situations.

There are some notes about LineListeners here, and the tutorials touch on what I’m calling frame counting in the very last section (“Manipulating the Audio Data Directly”) of the tutorial Processing Audio with Controls. Actually, the best code example is in the tutorial Using Files and Format Converters, in the section “Reading Sound Files” – where the example code has the comment

      // Here, do something useful with the audio data that's 
      // now in the audioBytes array..."

I’m guessing you won’t want to get in that deep. Best will probably be just pre-process the sound files into the exact forms that you wish to have them play back as, in Audacity, and load them as Clips when you want to use seamless looping.

#10

Who say you this?)

as i said


//Thread 1
AudioInputStream ais_swap = null;
while(true) 
	synchronized(sdl){
		if(ais_swap != null){
			ais = ais_swap;
			ais_swap = null;
		}
	}
	nBytesRead = ais.read(data, 0, data.length));
	offset = 0;
	System.out.println(name + " reading...");
	while (offset < nBytesRead){
	   System.out.println(name + " writing...");
		offset += sdl.write(data, 0, nBytesRead);
	 }
	if(killThread){
	   System.out.println(name + " killing...");
	   break;
	}
	System.out.println(name + " reading...");
}
//Thread 2
	synchronized(sdl){
		ais_swap = new AudioInputStream
	}

*if AudioInputStream’s have same AudioFormat

Perfect will be after swap - drain previous data

  • but this step can be skipped because it’s coupl MS time of sound before swap new track

byte[] data
SourceDataLine depends only on AudioFormat it dont care what data you write in

p.s Auido in java is hard - MIDI even harder
Easiest way – try change exist “audio tutorials code” as small as possible :wink:

#11

Thanks for that clarification Icecore, I had a vague notion that was what you meant, I just wasn’t sure exactly how to implement it. This will definitely speed up the swapping! :slight_smile:

#12

Forgot add IO check)


while(true) 
   boolean swap = false;
   synchronized(sdl){
      if(ais_swap != null){
         ais = ais_swap;
         ais_swap = null;
         swap = true;
      }
   }
   nBytesRead = ais.read(data, 0, data.length));
   if(nBytesRead < 0){
      break;
   }
   if(swap){
      sdl.drain();
   }

Technical you can mix audio in Byte Array befor send it
but I have no idea how mix “byte Audio data” )
(I believe simple + 2 data’s is wrong)

#13

Thanks for that. Now I have a question about the synchronized block. It synchronizes on sdl, but there is nothing inside the block that actually references sdl. Can you please explain to me how this works? Forgive my ignorance!

#14

[quote]Technical you can mix audio in Byte Array befor send it
but I have no idea how mix “byte Audio data” )
(I believe simple + 2 data’s is wrong)
[/quote]

  1. Convert the byte data to PCM values (very likely to -32768 to 32767 range if 16-bit data).
  2. Add the values from each input (and check to prevent going out of range).
  3. Convert back to byte data and ship it out.

Icecore’s basic example with multiple AudioInputStream is a good one. And, actually, it is okay if the incoming audio formats differ, as long as you make the necessary conversions before writing the data.

You get to pick when you read from either AIS. Another way to code would be to test if the read from the AIS returns -1. If it does, flip a switch and read from the other AIS without dropping a beat. That would eliminate the need for using a LineListener.

Where I was talking about counting frames, I’m thinking you can also do that by using the skip(long n) method. Let’s say you want to start exactly 2 seconds in. If the frame rate is 44100 fps, that would be 88200 frames. If the format is stereo, 16-bit, then there would be 4 bytes per frame, so the number of bytes to read before starting would be 88200 * 4 or 352800 bytes.

Starting or stopping abruptly in the middle of a sound can create a click. To avoid that, do a fade in. Even as few as 32 or 64 frames can suffice. (In the 3-step chart above, the middle step would be to multiply the PCM data by a factor that ranges from 0 to 1 over 64 or however many steps.)

I think we are beyond “Newbie & Debugging…” and that this thread is a good candidate to move over to the Audio part of the Forum.

#15

It’s not my example, but I’m not seeing why synchronization is needed.

#16

For synchronization you need any non changeable(not null) object that you can access from 2 threads
https://docs.oracle.com/javase/tutorial/essential/concurrency/locksync.html

I use sdl like example – because in many cases you don’t want create new SourceDataLine object in Audio thread.
But to make code clean better create separate syn object


public static final Object Syn = new Object();

Using Direct synchronized from another Thread its rude, same as static Syn object,
but for raw example its ok, and it works just fine

Its prefers for using Sync, i show why:


//Thread 2 set ais_swap
if(ais_swap != null){//Thread 1
	//Thread 2 set ais_swap = null
	//but Thread 1 already pass null check
	ais = ais_swap;//Thread 1 ais = null;
	ais_swap = null;
	swap = true;
}

Yes its rare, very rare but you can simulate this in debug mode
(stop threads, step line by line for 1, 2 Thread as you want)
synchronized block Prevent this.

#17

I’m not sure about step 2 :slight_smile:
Yes it works and almost everyone use it, but is it right?
Its same as Add 2 Red colors bytes, when you must add luminance of colors…

I’m interesting if record audio from mic between 2 instruments that play one Note
How sound changes?
I doubt that simple multiply Hz of played note on 2.
At least it must be some exponential Curve for Raw adding,
But for more clear it must be something Like LAB space in Color

up find this:


http://www.voegler.eu/pub/audio/digital-audio-mixing-and-normalization.html

little offtop:)
Warcraft II - Tides of Darkness - Human 2 Midi


#18

That’s explains a lot, but can somebody pleased explain to me why the block is synchronized on sdl, and not ais or ais_swap? ???

#19

[quote]I’m not sure about step 2 Smiley
Yes it works and almost everyone use it, but is it right?
Its same as Add 2 Red colors bytes, when you must add luminance of colors…
[/quote]
That is a reasonable question to ask. But in fact, from what I have learned from working through this resource, audio signal are indeed linear and can be added. The math supports this.

[quote]I doubt that simple multiply Hz of played note on 2.
At least it must be some exponential Curve for Raw adding,
But for more clear it must be something Like LAB space in Color
[/quote]
You are correct in that the relationship between what we hear as a progression from silent to loud and the magnitude of the waves is not linear. However, in the specific application (goal is to avoid creating a click from the discontinuity in the data), linear progression works and executes at less of a cost than using a power curve. Here I am speculating, but I bet that one could shorten the number of frames needed for the transition from silent to full volume by using a power curve, maybe by as much as half or even more. Whether the benefit of using a sweep of 32 instead of 128 frames is worth it is debatable. 128 frames = 3 milliseconds, and at that point, sensory events are next to impossible to discriminate.

But the best test is to try it out and listen to the results.

The links that you provide are for the situation where the volumes of the contributing signals overflow. Yes, compensating for that on the fly requires significant complexity in that one wants to reduce the components in a way that preserves as much of the tonal content as possible.

But my point of view is that if you are getting signals that are too hot to mix, the sanest solution is to just turn them down! Then, all mixing can proceed linearly and all of those complexities (which can be a drag on a limited budget for on-the-fly audio processing) can be avoided. In my conception of how to run things, the person responsible for implementing the audio simply has to review “loudest case” scenarios and listen, checking for the distortion that arises from overflowing. If there is distortion, adjust volumes so that this doesn’t happen. If the low end of sounds get lost this way, send the cue back to the sound designer for compression or some other means of narrowing the dynamic range of the cue.

A good sound designer knows how to use a tool like Audacity to provide the desired amount of compression or whatever is needed to best make a sound with levels that “play well” with others. (I would make this a hiring point --> somewhere on the chain from musician or sf/x creator to audio implementer, the knowledge and ability to mix sounds without overflowing.)

There is also the safety mechanism of putting in a Max and Min (for example if the DSP range is -32768 to 32767) is a reasonable choice as well. A little bit of overshooting here can cause clipping, but in some contexts the sound is an interesting effect, especially if you like metal guitar playing.

#20

Given that the preparation of the cue should probably be on a different thread than the audio playback thread, guaranteeing that a concurrency conflict does not occur is needed. On this I agree with Icecore.

As with most things in programming, there is more than one way. :slight_smile:

My biases come from when I “got religion” via nsigma about making it a high priority to never block the audio thread. Thus, I avoid using synchronization in the audio thread if I can figure out an efficient non-blocking algorithm. If nothing else, maybe provide a boolean latch and have the audio thread check the latch and “fail” if the AIS is not ready rather than block and wait. An “IllegalStateException” is often thrown in this case.

Also, as the programmer and architect of the sound design, you have the ability to set things up so that the “open” and the “play” of this special sound object (employing multiple AIS and other code) never enter into a race condition. This sort of concurrency requirement would normally be prominently documented in the class, and it would be up to the programmer to implement safely.

But I can also see that if the only audio that is being blocked is the one cue, then using synchronization and waiting is reasonable. This sort of thing is more of a concern in a scenario where all the audio is being mixed down to a single audio thread, as I do with the mixing system I wrote, or with a system like TinySound that also funnels all sound through a single output. There, a single block can delay the entire sound mixing process and contribute to dropouts. (This assumes that the native code that plays back audio will continue to process other cues while the one cue blocks. I don’t know if that is how audio works on all implementations.)