audio dsp (now on sourceforge)

WIP games, tools & toy projects
#1

Ok so this isn’t strictly gaming-related, but it could be useful as a tool in gaming projects for generating and processing audio.
What I’m working on is a software synthesizer/audio processor. Actually it could be used for any digital signal processing, but my focus is on sound here.
The idea is that you can drag-and-drop DSP objects to your workspace, connect them as you like, and as such create your own sound generators, effects and whatever you can come up with.

Here is a screenshot:

So what you’re seeing here is a network of digital signal processors (in this example, it’s an ‘auto-wah’ effect that actually sounds quite funky with a bass-guitar).
You can just drag-and-drop components from the left-side, and connect them together as you please to create your own sounds and save them as XML.
There are actually many more DSP components already there (delay, FM synthesis, reverb, phaser, chorus, wave-shaping etc), but I just need to add them to the left-hand menu to make them available in the editor.

Although I’m mostly focusing on musical applications for this at the moment, I think it could be useful for real-time audio processing in games as well. For example a reverb effect that adjusts itself to the area the player is in. Well, whatever you can think of. Anything that surpasses simply playing back samples I suppose.

If there is enough interest in something like this, I’m considering to make this an open source project (otherwise it’ll just be my own toy).

#2

I for one would love to have a tool like this to mess around with. Are you thinking of releasing a demo? I believe the interest would be there since to my knowledge there isn’t an open source tool like this out there and the commercial ones are quite expensive for casual hobbyists like most of us.

#3

I love this kind of stuff, unfortunately I haven’t been able to find a good (+ free) one that I liked yet.

Similar things I know of, some of which are audio specific: http://blog.interfacevision.com/design/design-visual-progarmming-languages-snapshots/

#4

I’ve just uploaded a very early demo to play around with.
It’s all still quite rough, but it sort of works.

Some pointers:

  • You can’t yet drag from the left-hand menu: You have to double click on it and it will appear on the top left of the workspace.
  • Asio IN/OUT obviously only works on windows if you have audio hw with Asio support (or have Asio4All installed). On other platforms you use Audio In/Out (i.e. JavaSound) instead.
  • Connections are always made by dragging from an output (i.e. right side of a DSP block) and dropping on an input (left side).
  • To disconnect, drag a connected output on the off it and on the workspace.
  • The Keyboard component simulates a piano keyboard on your computer keyboard, but you have to give it focus (i.e. click on the button in the center of the Keyboard block).
  • MIDI IN is untested at the moment (I will work on that today).
#5

Nice link, BurntPizza!
I love how Minecraft and Little Big Planet made the list :slight_smile:

#6

Well, funnily enough, I love these sort of environments. :wink:

Be interested if you do open this up - there may be some potential for code sharing here.

Also, if you’re interested, take a look at the JAudioLibs AudioServers project. I’ve just had a code contribution to review with a direct ASIO backend, but this would also give you access to JACK - low latency audio and inter-application routing on Windows, Linux and OSX. This would for example allow you to route any other ASIO program through your DSP environment.

#7

Wow that PraxisLIVE looks fantastic! I’ll have to try that out.

Thanks for the hint regarding JAudioLibs; it definitely looks very usable for this.

#8

Regarding opening it up, I think it’s a bit too early for that as I’m still tinkering with the general design of it so some things might change drastically in the near future.
The editor is sort of in the prototype phase.

One basic thing is that currently all signals are floats, but I’m considering making it all double. I think it shouldn’t negatively impact performance on a 64bit system, and in fact it might even improve since I already use doubles in various places where precision is important so there’s already quite some casting going on.

#9

This is awesome :o
Ok, it’s still in an early alpha state but there’s huge potential.
I could use something like this myself as I’m working on a 2D game engine with graphical Editor for nearly everything. We didn’t start on the sound system yet but this would defiantly make it a lot cooler to use :slight_smile:
I don’t know if it’s possible ATM but you could think of a random function to make a simple sound sound different everytime it’s played due to random pitch or other effects.
I’ll watch this and once it’s Open Source (I hope it will be one day) I’d try to contribute and (as I mentioned) use it myself.

Best regards
twinflyer

#10

LOL ;D I actually went the other way, and switched the pipeline from double to float, though I also use doubles in places where precision is important. Performance should be practically the same - AFAIK it doesn’t actually make a difference to the FPU(?) However, the thing that swung me was memory usage, particularly because of using things like in-memory sample buffers. Also the IO that offers floating-point is likely to be 32-bit I think (not sure on ASIO).

I remember reading somewhere (think CSound docs) that around 6 chained DSP operations can lead to audible differences between 32bit and 64bit signal paths, though depending on the algorithm the number of operations might be higher. To me it justifies double precision where necessary, but not necessarily double precision throughout.

RasmusDSP (an old project by the author of Gervill in the JRE), is the only Java audio project I’m aware of that uses doubles throughout.

#11

Hm good point about memory usage, I haven’t considered that yet. But what sort of memory usage are we talking about here that made you go back to float?
My consideration for using doubles is not necessarily for sound quality, but sometimes you need doubles in algorithms where high precision does make an audible difference (for example when there’s a lot of feedback involved). So I thought if performance will be the same anyway, why not use doubles everywhere and forget about precision concerns altogether? It’ll be simpler, and I like simple :slight_smile:
Asio is indeed 32bit float, but that would then be the only place where a down-cast is necessary, so I think that would still be cheaper than casting multiple times in your audio chain.

I should probably just test it out.

[quote=""]
There are noise generators, so that should be useful there probably. And it’s very easy to add new dsp blocks.

#12

Stop it! You’re making me dither again. :wink:

Seriously, there were not major arguments for me either way, and I ended up actually going for what the majority of people were doing as much as anything else.

Partly as I said around sample banks, etc. If you’re using a range of samples stored in memory, ideally you want them loaded up and ready to play instantly (ie. probably in the data format of your pipeline). Doubling the memory overhead here can be a problem.

Other than that, and something I’m less sure of the issues around, would be processor memory throughput / caching. I’m not sure what I read at the time, nothing I can remember being that conclusive an argument, but things like this (which I just found) http://dsp.stackexchange.com/questions/6079/when-to-consider-double-64-bit-floating-point-for-audio

I think I know what you’re getting at, though it sounds slightly like two opposing things. I found the CSound reference I referred to (somewhat incorrectly I realise) earlier - http://www.csounds.com/manual/html/MiscCsound64.html The audio accuracy arguments should be correct, though the performance assertions are probably irrelevant. Be interesting to benchmark a few of your algorithms at both precisions, though a full pipeline is probably more accurate (from a caching, etc. point of view).

#13

[quote]Partly as I said around sample banks, etc. If you’re using a range of samples stored in memory, ideally you want them loaded up and ready to play instantly (ie. probably in the data format of your pipeline). Doubling the memory overhead here can be a problem.
[/quote]
Ah, I see what you mean. I’m not using sample banks atm but I can see how that can impact things.
I suppose you can always have them in memory in a smaller data format ;D

The only places where I store signals in memory now is in buffers for delay lines and such, but even there I could imagine it could more quickly lead to cache misses.

As an aside, I got MIDI working now. I just patched myself a super-dirty dual-osc analog synth for giggles 8)
Things are quickly getting complex though. Just this simple monophonic synth has the screen filled with dsp blocks already.
I’ll have to start thinking about the option to create ‘meta-blocks’ or something (i.e. collapsing a whole network into a single DSP block)

#14

Wow! I just discovered this thread. Great work! This is something I am passionately interested in myself. I have also been making audio tools, but haven’t tackled filters or reverberation yet. I’m pleased to have found this thread and be able to follow along. I probably won’t have a lot to contribute, since my background is more as a musician and am couple steps behind on the java and audio engineering expertise.

I’m hot on a project that functions as a “tanpura” but also an intonation-training tool for classical musicians. When it is “done” (i.e., commercially released), I’m planning on continuing to expand the event-system tools developed for it to support ideas I have for algorithmic composition, generative/dynamic scores for java games.

I haven’t opened my audio library yet either, but various “toys” have been posted along the way.

A theremin: http://www.java-gaming.org/topics/java-theremin/24646/view.html – the biggest challenge was getting the unit to be responsive to the mouse/ui thread. I do plan to redo the wavetable synth sources to eliminate the aliasing (on a long queue of tasks to get to).

A “peaceful brook” demoing a wind-chime using an fm-bell synthesizer, and a continuous clip-slicing playback tool:
http://www.java-gaming.org/topics/what-music-do-you-listen-to-while-you-code/27824/msg/275109/view.html#msg275109 The use of Perlin noise for the randomization on the windchime was silly and inefficient. I have a better version of what I’m calling a “SoundField” tool now for managing background sounds with randomized timing elements.

Am looking forward to further exchange of ideas.

Am curious, what size buffer are you using in passing audio from one unit to the other? I’m simply using a single frame which is a significant inefficiency, but allows for considerably simpler coding. I do use buffering for actual file i/o (e.g., wav playback and SourceDataLine output). But the inner mixing loop and effects processing mostly occurs on a per-frame basis. It works.

Also curious, have you been able to get these tools to work on Android or iOS? I own neither and haven’t tried this yet on my own audio library, but the output is just a single, never-ending stream (SourceDataLine) which seems like it should be possible on both system.

Are you using any native code or any libraries other than javax.sound?

My FM synth can either create play in real time or be used to generate DSP data samples required for the upcoming score segments. I’m thinking it might be useful to be able to have the option, allowing one to dynamically optimize according to whether the bottleneck is processing or RAM. Just as much visual/graphic data can be generated at level-transitions, the same could be done with the sound design and musical score.

#15

That reminds me of a question I was going to ask erikd -

Are the oscillators you currently have just sin waves, or have you implemented square, saw, etc? If so, are they band-limited? I was having a look a while back at (fast) options for generating band-limited oscillators, either real-time or table lookup - lots of algorithms around though haven’t found anything in Java, or yet had the chance to try porting something. That might be some code that would be useful to share thoughts on around here?

#16

The idea I intended to try next was to build the 'band-limited" square/sawtooth tables from sine waves. The tables are small enough, and it is a one-time event, so the speed of the computation isn’t that critical, is it? Or am I off base, conceptually?

#17

Currently the oscillators are not band-limited so unfiltered you can get some aliasing, but it’s on my to-do list.
I’m not sure yet how I’m going to implement that. Perhaps pre-filtering the waveforms, or maybe simply oversampling and/or averaging is enough?
There are various waveforms though (sine, triangle, block, saw and more) and there is a pwm input.

[quote]Am curious, what size buffer are you using in passing audio from one unit to the other?
[/quote]
I’m not buffering anything.
My interface for a SignalInput is
void set(float signal);
and for a SignalOutput, it’s
void connectTo(SignalInput input);

So I’m basically simply passing single floats around.
In the units that handle external I/O such as javasound and Asio there is of course some internal buffering, but implementation details like SourceDataLine etc are hidden in such units to make the ‘core’ of it independent on J2SE-specific APIs like javax.sound.

[quote]Also curious, have you been able to get these tools to work on Android or iOS?
[/quote]
I have done some audio processing on Android (although not with this particular project), but I found that latency is an issue there so for this project I didn’t get into that yet. I don’t have any iOS devices, but it seems that iOS is more suitable for musical low-latency applications.

[quote]Are you using any native code or any libraries other than javax.sound?
[/quote]
I’m using JAsioHost on Windows because unfortunately javax.sound has way too much latency there. Asio uses a pull-mechanism rather than blocking I/O, so it was a little bit more tricky to properly synchronize things, but you can really have good latency numbers there (about 2.5ms in Asio instead of ~150ms in javasound). I found that using Asio, you can even use it for real time guitar effects with virtually unnoticeable latency.

As an aside, a good source of information is www.musicdsp.org for this sort of thing.

#18

One of my go-to resources is where musicdsp relocated to: the KVR audio forums: http://www.kvraudio.com/forum/viewforum.php?f=33&sid=ebf901e536d1ce2bfbfde2755c5e6afc
I remember seeing at least a few threads about efficient ways of band-limiting your oscillators, BLEPs, etc, e.g. a few pages in: Tutorial: BLEPs (using PolyBLEPs but extensible)

#19

Thanks for the link :slight_smile:

One of my main issues in the editor now is to keep things scalable from a usability p.o.v.

To give you an example, this is a fairly simple emulation of a funky clavinet sound with a resonant filter (think Stevie Wonder 8)) using Karplus-Strong for basic sound generation:

http://www.gaga-play.com/test/karplus-strong-test.png

Not only is this mess a fairly simple patch, it’s also monophonic. As you can see, things can quickly get quite unmanageable :persecutioncomplex:

At least I’ll have to create some way to collapse a selected network of units into 1 ‘meta-unit’ that can be reused.
And then to make synths polyphonic, I’ll probably need to create some specific interface for such a ‘meta-unit’ so that I can simply duplicate them into multiple voices that can be driven by a polyphonic set-up.

#20

A little update:
I have experimented with band-limiting the oscillators, but I’ve obviously been on the wrong track there.
The oscillators use pre-calculated wave-forms, and what I did was simply pre-filtering them. Sure it helps with aliasing at higher frequencies, but of course it also made it all sound quite dull.
I think the best solution there is to do real-time oversampling of the oscillators instead.
It might be a bit more expensive, but then again I haven’t gone beyond ~2% CPU load of a single core yet (and that is while the CPU was running at about half speed), so I probably shouldn’t worry too much about performance yet.

I have also worked a bit on making the editor a bit more manageable. Now you don’t really have to add ‘controller-knob’ units anymore. Instead it will simply show knobs for all editable settings for the selected units at the bottom. Much less clutter!
The following screen shot is of the same patch as the last one (but now including soft clipping):

http://www.gaga-play.com/test/karplus-strong-test2.png