PFTheremin

WIP games, tools & toy projects
#1

[list]jar downloads:
[list]- most current version PFTheremin
[/list]

1011×532 75.8 KB

Continuing on the theme of making scalable audio assets…a theremin in the works. The JavaFX GUI allows it to be played in real time.

I started writing about this and went overboard, so deleted everything. Minimum needed to know to try it out:

mouse down in the orange area to get playback
X-axis is pitch, Y-axis is volume (and optionally mapped “Expression Axis” features)

Next post will be an api for use as an external library, if you want to try incorporating this audio functionality in a game you are writing.[/list]

#2

PFTheremin, SoundHandler API

SoundHandler sh = new SoundHandler();

Class is located in the package: com.adonax.jtheremin

// Following are methods of SoundHandler, e.g., sh.start();
void start() // turns on audio thread
void stop()  // turns off audio thread

void startSync()  // save a reference sound frame and nanotime for internal event
			// used to schedule the pitch updates, can be called any time the 
		// pitch updates develop lag beyond a built-in 1800 frames

void setSynth(String name) // {"FM Sawtooth", "FM Square"} 

void startNote()  // theremin note starts playing, called by mouse-down
void releaseNote()  // theremin note released, called by mouse-up
void updateVolume(float normalizedFloat)  // Y-axis position scaled to float [0..1]
void updatePitch(float pitch, long timeStamp) // pitch = actual Hz to be played
		// timeStamp (nanotime) = when that pitch occurs
		// the frame where the pitch will be scheduled is calculated from the 
		// values stored in startSync()
void updateVibDepth(float normalizeFloat) // vibrato affects both pitch and volume
void updateVibRate(float normalizedFloat) // [0..1] corresponds to 2Hz..18Hz
		// distributed via a power function x^2 
void updateTimbre(float normalizedFloat)  // [0..1], corresponds to a range of values
		// preset in synth, sometimes referred to as FM Modulator Index
void setModulatorFeedback(float normalizedFloat) // more FM stuff, affects timbre

// control of Echo via the SoundHandler, e.g., sh.setEchoRunning(true);
void setEchoRunning(boolean echoRunning)  // make the echo effect audible/quiet
void setEchoMillis(int millis) // time until first echo
void setEchoVolume(float normalizedFloat)  // volume of echo
void setEchoFeedback(float d) //

The following function may be useful for generating pitch values procedurally. Midi note values are linear, one per “piano key”, with “middle C” = 36. Fractional notes are possible.

double PitchFunctions.getHertzFromMidi(float midiNoteValue)

The package of this static class: com.adonax.pfaudio.midi

(Please message me if you want to use this as an external library in any game or application you are making.)

#3

Very cool.

Do you happen to have the source for this on github or a similar site?

#4

Nice app, worked fine for me on windows 10 java 8.
The default settings make a spooky wail. Will be interesting to see how you and others tune and use it.
I have no experience in how to make or use sound but intend to learn one day.

#5

Thank you for the compliment!

The only code that is posted has been posted here on JGO over the years. I’ve been working on getting better at using Java Sound (javax.sound.sampled) for several years now, and the posting reflect progress made and issues dealt with. Some of the later code is better than the earlier stuff I posted.

I am happy to answer questions on it or on how to make things using Java Sound. I have a notification attached to the jgo audio forum and always check on anything posted there.

#6

Good to hear!

Re learning audio: no time like the present!

You comment brings up a point though which is that I haven’t made a place where people can share patches. Maybe, since they are text files (xml, that is), the simplest thing is to post patches on this thread if anyone wants to share? I don’t think I can support a forum on my website. Maybe it is possible but I haven’t figured out how.

I’ve been meaning to post more myself, but I’m letting “perfect pictures” get in the way. I’ve been intending to do things like rent “The Day the Earth Stood Still” or “The Red Planet” and try and replicate the effects used there, or maybe try and match the theremin in “Good Vibrations” or “Dark Shadows”. Too ambitious, should just post some that illustrate basic capabilities.

I did make one upgrade to the program a few weeks ago. There’s a trade off with screen size, pitch range and pitch precision. The wider the range, the more difficult it is to hit specific notes. What I came up with is this: when hitting the shift key, the mouse pitch stays the same, playing or not, where ever you move. Thus either in silences or on held notes, use of the shift key and maneuvering can shift the screen pitch range to the area you wish to play in.

#7

Unlocked :point:

#8

Progress report, and an attempt to figure out what to do next. It is hard to write about the design issues without cranking out a full article.

Updated project file: PFTheremin.jar.

I’ve started collecting YouTubes of famouse examples, and then try to duplicate the sounds. It’s been an interesting exercise.

Question:

Can I publish a video with short clips from famouse uses: clip of Theremin himself playing, clip from “Day the Earth Stood Still”, “My Favorit Martian”, “Dark Shadows”, wav clip from Beach-Boys “Good Vibrations” and keep the shots under something like 5-10 seconds and have it be considered a “Fair Use”?

It would be nice to show this theremin recreating those sounds, to the best of its ability. Seems like a video like this could be made inexpensively (assuming the only additional footage is just deskcam and screenshots). I have a decent microphone, so it seems like it should be possible to make something like this to support sales without spending multiple thousands on video production. As an indy programmer, we get some latitude on video “polish”, yes?

Mulling (need to prioritize):

  1. Maybe make a finer discrimination on the pitch width: to the nearest 0.5 or 0.25 octave rather than just a number of octaves. [EDIT: Just changed it to 0.5. Haven’t uploaded yet.]

PURPOSE: can more finely tune the screen to spread an intended note range (e.g., you have a specific song in mind) over as much screen real estate as possible.

The function also addresses the pitch-range vs. accuracy problem, by allowing one to move the screen location of a pitch.

  1. Make the window resizable. This was partially addressed by making the settings panal have two sizes. To do this, I made the horizontal width a variable rather than a constant. That gets us partially there.

PURPOSE: again, addressing the pitch range vs accuracy issue.

  1. Add a couple more synths? I came across an algo for making an FM synth have a “formant”. If it works, it would be possible to make the Y-axis cover a range of vowel sounds. Could be going oooh-aaah-eeee-aaah with the synth. IF it works as a synth. Algo was from Bill Schottstaedt, mentioned in Charles Dodge’s “Computer Music” book.

Other synths might be an actual sawtooth or square via wave-table synthesis + filter, or some nice string synth options.

PURPOSE: more tonal qualities possible.

  1. Add another “effect”, e.g., a flanger with standard modulator controls. Flangers are relatively easy effects to code. Or add some other effect? Chorus and distortion come to mind. If so, and a question in general: should the effects settings be part of the save/load? Or separate save/loads?

  2. Record mouse/keyboard events as macros. Allow playback of macros. Allow export to wav of macros. (I am reluctant to simply try to put in a direct save-to-wav for various reasons.) I did some refactoring so that the data going to the audio player comes exclusively via the AnimationTimer (JavaFX) and not from the asynchronous inputs. This makes the input data 60FPS, and it should be straightforward to read from a “macro” file rather than from the “control” level variables.

  3. Mouse tails? Design mouse cursors? What to do about skins on the design. Keep this color-scheme, keep it simple, or do something to give it a cooler look?

  4. Ability to move or stretch the keyboard guide vertically?

8 ) Add an alternative “vibrato” key? (V? I am reserving for macro recording.) This would be used as an alternate way to intensify the vibrato for given pitches. The configuration could get complex: as we have both the amount it would add and the rate at which this additional vibrato would come on and drop off.

PURPOSE: theremin players often reserve vibrato for given notes, like a musician or singer would. On a real theremin, this is done with a separate hand, not with the mouse doing everying. Clear example: listen to “Good Vibrations” where the vibrato is mostly on the third note of the pattern that is played.

  1. Come up with a better name for the program?

  2. Free or $1 version and “deluxe” version (only a couple more $)? If so, what features would go on each?

I’m thinking of marketing on itch.io, for starters.

Definite TO-DO list:

  • Clean up the use of .properties file. [I’ve been avoiding finshing/testing the details of save/load.]
  • Add JVM so it can be stand-alone.
  • Add “Help/instructions” (how much? post online link instead?) and “About” info.
  • Make some patches to go along with the instrument.

Current GUI:

1002×582 78.4 KB

#9

nice one :slight_smile:
reminds me on https://play.google.com/store/apps/details?id=com.zebproj.etherpad

i’d throw it in into my mix if it’d support …

  • asio
  • midi-in
  • midi-mapping

o/

#10

@basil_

I had closed my mind to making plugins for DAW’s because there seems to be so little in terms of Java support for this. Most audio is written in C++ afaik. But maybe if one is including a JVM in the jar, there would be a way to use Java as a VST.

Do you know of resources to learn how to do this? Did you have another plan in mind besides VST?

As far as the theremin, it seems to me there might already be good ones available. Any digital implementation of a ribbon controller, for example, could suffice.

It might be nice to have the 2D screen pop up within the DAW for live recording. In that case, I’m wondering about saving as 60fps stream data rather than converting to midi control events. But maybe interfacing the MIDI system is more practical.

One thing, though, there would be no need to worry about the built in echo or adding more effects, as it is easy to add them in the mix on a DAW.

#11

I can’t exactly point you to any 3rd party resources I might have used in the distant past (circa '08 last time I recall), but it’s possible to create VST plugins with Java w/ a GUI and all. Instead of audio processing I had several Java / VST plugins which didn’t process audio at all, but took VST control data input w/ a Java2D GUI outputting OSC to SuperCollider allowing these plugins to be automated in any VST host; AudioMulch was the test bed back then.

Just search for “java VST” or “java VST tutorial”. It’s definitely possible and things worked fine and I suppose audio processing w/ Java would be as well though I was mainly focused on the GUI / control data to OSC aspect in my own efforts.

#12

Search uncovered this:
http://jvaptools.sourceforge.net/index.html
Will check it out.

I don’t remember this being here a couple years ago, last time I looked. (Date on bottom says last update 2006!)

I had a couple ideas for VST’s. Wanted to do a fancy volume (key velocity) overlay (kind of like a transpose amount for key selection) to a given midi part, with tools geared to metrical patterns.

#13

The nice thing is that the VST host handles all the MIDI / control data input and forwards it onto the plugin. As mentioned I can’t recall which 3rd party Java integration tooling I used… This also comes up http://jvstwrapper.sourceforge.net/. Hah, being SourceForge that is about right circa aughts. It was pretty straightforward and performance was fine and this was a while ago.

It’d be great to see some of your efforts integrate w/ VST as that will provide a continuity w/ general music production environments that is better than standalone apps.

#14

Resurrecting this project. Most of the week has been spent disentangling the three original files into more specialized classes and improving the model-control-display aspects. Amazing sometimes, looking back at old code and wondering how one got it to work when it was such a tightly-coupled and difficult-to-read mass.

Here’s a programming problem I’m pondering. I’m going to do a little thinking aloud.

I’d like to be able to “record” a gesture or motion on the play-pad, as a “macro”, and be able to play it back. As it stands, the play-pad area updates “control” variables for the X and Y axis. But I only read those variables via a JavaFX AnimationTimer (60 fps, functions as the equivalent of a game-loop) and use those values to run the playback synth.

It would be pretty easy to toss the X & Y values into an array, and to later pull from that array. The hitch with this is that there is a key event (a vibrato booster) that bypasses the Animation Timer and the mouse up and down sends synth start and release commands to the synth (in this case, the animation timer reads a “mousePressed” boolean and reacts when it changes state).

Capturing these rare events, I’m wondering if I need to include the mostly static values as additional variables as part of the continuous stream, of if there would be a way to include some sort of time-stamped feed for the rare events. Also, there is a matter of whether to include all the settings as they existed at the start of the playback file, or to allow the streamed X & Y data to be used with different playback settings.

If I do the former, I’ll probably have to do some more refactoring, aggragating the state variables into one or more data-only classes.

Am not going to attempt to integrate with a DAW or use VST at this point. (Though, with jlink now available, it seems more practical these days.) Instead, there will be a way to “export” the recorded macro as a wav. People using DAW’s can import the wav’s if they want to use them. MAYBE if successful with macro recording and playback (a necessary step along the way) then could reconsider, as I will have an interface defined via the macro “api”.

A few things were done since the original posting, might be worth mentioning:
I did add a vocal/vowel patch. I think I already posted about it on “what I did today” somewhere. Last year?

I also added a lot more customization to the Y-axis controls.

I’m looking into adding distortion as an effect. Lots to learn about that! (Pondering saturation vs distortion among other things.) Also, to figure out how to code (want to make the Y-axis optionally be able to pertain to a degree of distortion). Also, looking at incorporating over-sampling as a way to get more highs without aliasing.

It is not easy seeing a project through.

#15

Decision made to simply record the entire state. A StateChipbnow has 18 fields (but does not include echo settings or synth choice).

First record and playback successful, just happened!

Now, need to work out the control logic.
How many macros allowed?
Assign to number keys?
How “arm”, e.g., in order to record to slot “1”, have keystroke “1” followed by keystroke “R”?, or to playback, “1” followed by “P”? or reverse this, putting us in playback or record mode, and number key triggers this action for that macro?

Later:
have the display show the cursor movement recorded in the macro?
export a macro as a .wav?
ability to load/save macros?

Maybe make an automatic default record of last [fill in amount] that a person can listen back to?

Add some sort of primitive editing? e.g., set some sort of split point on the macro and allow all in front or back to be discarded? concatenate two macros? Don’t want to go overboard with additional coding.

Am wondering how other apps handle macros, but am drawing a blank trying to think of a similar app to this.
Always happy to hear ideas, encouragement.