Audio library demos

WIP games, tools & toy projects
#1

The jar for the first demo: “Circle Sounds”, is here.
The jar for the second demo: “Peaceful Brook” is here. The second demo is introduced at this link.
Jar for third demo: “Shepard Chord Builder” is here. This demo is introduced at this link.

518×540 45 KB

I’m working on an audio library for sound that can readily interact with games. Some of the most important pieces are in place, but as part of building this, I also want to make some jars that illustrate “use cases,” both to have demos to show and to personally get more experience with the usability of and issues around the tools I’m writing.

In this program, five circles (of random colors) are linked to five synth sequences each running at its own rhythm, but all locked onto a common Pulse. Each sequence is played by a different software PM synth. Volume and pan levels are continuously updated as the circles move around the screen. Each circle has a “NoteListener” that causes it to blink: fading up and down with the primary envelope (volume) of selected notes of the sequence, when those notes are played.

The background circle marks the “audible area.” Area can be dragged larger/smaller. Center is the loudest point, beyond the edge is silence. Pan is calculated using the angle from the center (0 = right, 180 = left, 90 & 270 = center).

The pitch and tempo can be altered, independently, via clicking inside rectangles on the bottom right. BOTH pitch and tempo are moved at the same time when clicking the cyan or orange square on either side of the rectangles.

Colored circles can be resized (drag edge) which only affects how much time is spent in the audible area. Circles can be dragged and released (while still moving) to push in a given direction, though they aren’t allowed to go very fast.

Mostly just trying to show some capabilities, and maybe tweak the imaginations of some artists or game designers who could easily go beyond these simple graphics.

99% of the audio code is core Java. There is a single “wrapper” class that links the core code to javax.audio.sampled. I also have a wrapper class for Android. My first tests on Android though were with a phone with a relatively weak cpu, and there were dropouts, even though sound code ran perfectly well in the AndroidStudio emulator on my Linux partition.

It would probably be simple enough to write an OpenAL wrapper as well. I don’t know enough about OpenAL to do so. But all that is needed is a streaming output, and calls to the library mixer for the PCM data for that stream, and a routine to convert the PCM to the appropriate byte format.

#2

Runs fine on my system. I could hear the music changing but I wasn’t sure how it related to the screen objects; maybe make the objects different shapes instead of all being circles? It ran smoothly after I remembered to shut down Vangard; it didn’t seem to like Vangard’s behavior or eating all CPU sores for 10-20ms every 33ms - with Vangard running in the background, there was a slight stutter every few seconds. I suppose the routine can’t build a buffer because it is reacting in real time to the game state…

#3

@ags1 Thanks for the very useful feedback.
I’m curious what Vanguard is. I did a search but had multiple hits.

Re timing–there might be more that can be done to help. Timing issues occur on multiple levels. At the lowest level is playback continuity, which relates to the audio frame rate. Here, the low level audio frame rate is 44100 frames per second. At a higher level is the “pulse”, which is happening about 8 times a second (plus or minus, depending on tempo selected). Many decisions or states don’t become manifested as audio until the next pulse. So, a buffer system that supported the first level but didn’t interfere with the second could be a good addition. (This is pushing a bit beyond the edges of my current abilities.)

Yes, it is hard to figure out what circle goes with which motif. I have to curb my enthusiasm for complexity if the goal is to make a “demo” where things are clear.

Coding various shapes could work. Another idea I’m getting is to give each circle an image that gets resized/redrawn. (Main prejudice against images, they require lots of bytes, and might weaken the point that the audio heard is done with very little. Having a jar with a complex system at 80KB is more impressive than 80KB + several hundred KB in images.) (OK then, draw something unique in each! Hmm. Spiral sprites!)

I could also start the thing off in a state where there is less going on initially. Couple ideas along those lines to ponder (I am open to others):

  1. have a slider that controls the number of circles, or a control can turn circles on/off (makes them invisible and inaudible)

  2. animate the “audible area” (gold background ring) so that it ranges from very small (where it is clearer what is passing through it, but will leave more silence or thinner textures) to very large (where everything is babbling at the same time). It is possible to manually change this area’s size, but most people won’t think to do so.

[EDIT – added] Another solution: make a piece where the parts don’t sound so similar! Instead, am playing around with composing and adding another track, more slow paced and deeper, compared to the others. It will probably be easier to spot compared to the others (aurally) but the others will still crowd a certain perceptual space.

#4

Vangard

#5

Doh!

#6

If you go for different shapes, exploit the innate synesthesiac nature of people: give higher-pitched, discordant audio to triangles and more harmonious, lower tones to large circles…

#7

I uploaded a slightly modified version. On the bottom left are little squares for turning individual circles on or off, color coded to the circles.

I am starting to switch over to JavaFX, so I think I’m going to stop investing in this GUI for a bit until I get a chance to rewrite it. Still going back and forth about adding a couple more concurrent themes/circles.

Meantime, am working on another demo that is more of an environmental soundscape.

#8

Now there are six sounds – added a lower, slower voice. And I changed the wrap-around to be a fixed amount. Before, I was making the wrap occur as soon as the circle went off screen, regardless of size, making the off screen area inconsistent.

There was one other behind-the-scenes change. One of the faster parts required 14-note polyphony to play, but it only played 3 different pitches. I did some changes so that the pitches are restarted, so now this part only requires 3-note polyphony. Getting this to work required adding a way also enable transposition of pitches, even with the pitch set predefined.

One thing I haven’t tried yet is setting the priority of the busy audio thread to high. This has been suggested by some on other posts on audio questions. Theoretically, it should work, because I think the audio thread is in a blocked most of the time.

But I also have to acknowledge, the approach I am taking will at first be limited in the number of notes polyphony that can be used. This demo has 44 synth notes being read every sound frame. There is still a fair bit of richness one can achieve with careful composing, but this does limit things, for now. More will eventually be possible as computing power continues to increase.

#9

Absolutely! The audio thread should have maximum priority. It’s not when it’s blocked that’s important, it’s when it isn’t! :wink: I can think of at least one Java audio project that goes as far as using native bindings to increase the thread priority above what can be achieved normally. My bindings to JACK also have that, although the thread in question is created externally to the VM.

Incidentally, you don’t have to block on the audio write either - you can use a larger buffer and write smaller amounts to it based on System.nanoTime() or line.getLongFramePosition() - see here.

#10

Does this library support the loading and playing of sound files like wav and ogg?

#11

Quick experiment #1, I set my audio thread to Thread.MAX_PRIORITY and nothing blew up. The graphics remained smooth. It’s getting late (dinner was served two hours ago and I’m still trying to get to it). Maybe I will send a copy with the MAX_PRIORITY to ags1 and see if she still gets the dropouts while running Vanguard…?

@nsigma
I’m not a very good code reader and am not sure about what I am looking for in the example you linked. Is your code counting frames or elapsed time and then putting itself to sleep? How should that help, theoretically? Or, is this needed because you are not using java sound and thus don’t have a blocking line of any sort between your processing and the native playback?

I read and process one frame at a time from the core audio mixer, returning the stereo audio values to a “wrapper class”. These output values are accumulated in an array in the JavaWrapper (or AndroidWrapper). For the JavaWrapper, I accumulate about 1/20th of a second’s worth of data and then write that via the javax.audio.sampled.SourceDataline.write() method. Android has a similar output method.

In both Java and Android, these write methods use blocking lines. I let them handle the rate at which their respective audio systems process the audio data.

I previously was having trouble with dropouts when using arrays holding less than 1/20th of a second, but perhaps by making the priority of the thread higher, a smaller buffer is both possible and helpful?

Quick test #2, with HIGH_PRIORITY, I can run the program with a write buffer of 4096 bytes instead of 8192, but I did get one dropout over the course of two minutes. However, running with a buffer of 8192 bytes and HIGH_PRIORITY for over five minutes, I have yet to hear a dropout.

#12

So far, I have wav input and playback, not ogg.

I plan to have, for both desktop and android:

  • ability to load data via “CD quality” wav files,
  • ability to load data via a 16-bit, 44100 fps stereo PCM stream.

With the latter, the programmer can read ogg, decompress it themselves, and present the result as a PCM stream for loading. I wasn’t planning to support playing back from compressed files as that chews up a lot of the audio thread’s capacity, and is inherently inflexible (compared to composing and manipulating sequences for live playback–my main interest). But it could still be used for loading resources (with the programmer handling the decompression step).

For direct ogg playback, something like TinySound makes more sense.

#13

I wrapped Headline Benchmark in an EXE and set the EXE process to High priority in the OS to make sure I was getting high priority in Windows. Java thread priorities are just hints to the VM, they don’t mean you will get high priority.

#14

Just had a first play with this, and really love it as a sound toy. Reminds me a bit of stuff I saw years ago by Tonne (http://www.studiotonne.com/work/music/). Take that as a big compliment from me! :slight_smile:

This is the JavaSound server, so yes it’s got the same blocking lines. :wink: The server supports either blocking mode (like you are doing) or two modes that don’t block on the output line. Instead it opens the line with a much larger buffer on the soundcard, and manages the timing itself using either nanoTime or the frame position. Frame position would be the most suitable, but doesn’t work on Windows because the reported value isn’t very accurate. The code does a mix of sleeping and busy looping until the next frame is due. It’s similar to some code on here for game loops.

This is not my technique originally, it’s been suggested by various Java audio people in the past, but I’ve hacked it around quite a bit. You’re bypassing certain unknowns in how the underlying JavaSound code / native driver is blocking and rescheduling your audio thread, which is sometimes not as fast as you might want it to be.

#15

A second demo has been added, pertaining to sound scapes, called “Peaceful Brook”:

[EDIT: link is now to a new revision 6/16/16]
peacefulbrook

All it has is a simple GUI with a bunch of sliders–nothing to look at. It would be cool though to have a small 3D area with a brook, frogs, crickets and wind chime. At best, I might be able to create a simple JavaFX 3D something myself, but I do have other priorities (like possibly coding a Shepard Tone builder).

The main thing that happens when you twiddle the sliders (on the right-hand side) is to affect the number of beings that are active (frogs, crickets) or how often they speak or sound (frogs, chimes).

The brook should sound continuous without looping. This is done by stringing together half-second slices from a 4-second sample of a local creek. The slicer also uses 1/10th second overlapping with interpolation for smoothness.

The frogs derive from a single croak, also recorded at the creek. The chimes and crickets are synthesized. There is more about the randomizing that controls playback that can be read via the “About” button.

file size: 841 K
brook sample: 709 K (after compression)
frog sample: 62 K (after compression)
total: 841 - 771 = 70 K for the supporting GUI, library and “About” content.

#16

I’m going to play with this tonight. Is there a way to run it without the GUI?

#17

@ags1 asked:

[quote]Is there a way to run it without the GUI?
[/quote]
There’s a catch-all class for the audio called “SoundHandler”. The methods exposed by this class run all the audio aspects. I could make a jar with just this class (and the audio library materials called) and also supply an API. At that point, it should be possible to use it as an “external jar”.

Is that what you had in mind?

EDIT: might even work if you link to the jar.

    new SoundHandler()
    public void start()  
    public void stop()
    public enum Track { BROOK, CRICKETS, BELLS, FROGS };
    public void updateVolume(Track track, float vol)  //where float vol is [0..1]
    public void upDateIntensity(Track track, float f)  // where float f is [0..1]
    public void setChimePitches(int transpose)  // where int transpose is number of half-steps, plus or minus

So you could something like this:

    SoundHandler sh = new SoundHandler();
    sh.start();
    sh.updateVolume(sh.BROOK, 0.5f);
    sh.updateVolume(sh.FROGS, 0.5f);
    sh.updateIntensity(sh.FROGS, 0.5f);

etc.

And tell me if it works!

#18

Looks like I won’t have a chance to try this for a couplke of days, but I will let you know what happens.

#19

Probably rushing too fast to show this.

Download is on first post of this thread.

The Shepard Tone is a sort of sonic barber pole illusion. I did a few things differently from the classic example: only used three octaves instead of 4, allowing chords. The speed lets you push past to the point where the illusion breaks down and the looping becomes more apparent.

I want to get a continuous/steady-state (glissando) going as well. Will work on that next week.

Am most happy to receive advice or ideas for improvements, as always!

#20

More than 60 days since last post! Dang, it took a long time to finish the modifications/improvements to the Shepard Chord builder.

New features:

  1. the glissando function works now
  2. real time volume changes are working
  3. two 3-octave ranges (high, low) and a 4-octave version (wide) to choose from
  4. ability to take a loop and save it to a wav file now in place

About the last feature: some improvement could be useful, as the “Moog52” patch has a distinct attack and the LERP that is done to smooth out the loop (for the gliss version) doesn’t eliminate the change in timbre.

But, maybe this is cool: if you load the wav as a Clip and play it via Loop, it will ascend or descend smoothly and continuously. No breaks.

Last thing: it should be possible to load the file as an externally linked jar to a project, and to generate tones procedurally. The nice thing is that you are not limited to the parameters I set. For example, you can set the ascending or descending going so fast that it turns into a sort of frequency modulation effect. Also you can set your own pitch range and starting pitches, rather than relying on the values hard coded into the gui.

I will test this shortly and post an API.

A spinning skin would be nice…am using JavaFX for the gui.