UselessGameDev logo UselessGameDev title

A Weird VR Musical Instrument

Sun 20 November 2022 #UselessGameDev
This text is a written version of the video above, with future plans to incorporate graphics and figures for illustration.
As I'm working my way through the backlog of videos that were published before this website, I suggest reading the article alongside the video for a comprehensive experience.

VR Instruments cannot work

VR Bongos, VR Guitar, VR... Bagpipe?

If you want to make a VR musical instrument, you’re going to hit a wall: Virtual Reality doesn’t give the player the feeling of touching any object. And from there there are really two ways of circumventing this issue: either you release a tangible controller shaped like your instrument, which is a whole other logistics issue in itself, Or you wait for VR gloves to become a real thing.
Which actually looks like it’s coming pretty soon looking at the amazing stuff Lucas VRTech does.

But in the meanwhile, what if there was a musical instrument that doesn’t require physical contact, and could be played remotely?

Meet the Theremin

Meet the Theremin. Invented in 1920 by Lev Sergueïevitch Termen, the Theremin works by leveraging magnetic fields.

A metallic vertical pole emits an electro-magnetic field. When your hand enters the field and disrupts it, it will alter the pitch of the sound. The closer the hand is to the antenna, the higher the frequency.

There is also a second antenna, which usually loops horizontally, and uses the same principle to control the volume of the sound.

The Theremin is very popular in films for its eerie timbre, just listen to this music played by the man himself.

This contactless instrument looks like it’s perfect for VR. I’m Leonard, welcome to Useless Game Dev.

Making a theremin

Pitch Detection

I have a problem: the problem is, this instrument is dreadfully easy to make in Unity. Calculating the distance from the hand of the player to the nearest point on the antenna is essentially one line of code. I can break it down into multiple lines to feel better about myself but I’m basically done.

Add a few tricks to remap the distance to a range of frequencies, copy-paste to make the volume antenna, and this project is basically completed in under 30 minutes.

The real question is how we’re going to synthesize sound.

MIDI

The first thing I did was to make the program behave like a MIDI controller. Simply put, MIDI is a protocol allowing electronic musical instruments to communicate, usually with a computer that will play sound based on the inputs provided by the instrument.

So by tricking my computer into thinking the program was an actual external peripheral trying to communicate via MIDI protocol, I could drive a MIDI software and play sound with it. Any sound actually, once you have software that receives input, you can play a theremin sound, or any other instrument, and add filters and stuff.

But this MIDI protocol part isn’t super interesting, it would make for a very different kind of video that I’m not really into.

Synthesizing

If I could synthesize the sound from Unity however, the player would be able to adjust the sound of the theremin directly from the VR environment.

I’m using a sound generation library called NAudio.

With it I’m able to generate simple sounds, like sine waves, or sawtooth waves.

So if I take two waves, with frequencies that are multiples of each other, and then add them together, then wabam you get a more complex sound with harmonics.

You can then chain other effects such as a low-pass filter, that smoothes out the higher frequencies.

Trying it out

Separating these into different brick, we can make a nice modular sound generator.
Let’s slap together a couple VR interfaces, struggle with the physics of making cables, and don the headset.

Let’s see what we have here. I made these modules that allow the player to configure the timber they want.
First thing is a simple wave generator. When linked to the audio output module, it makes a simple sine wave, which can be switched to a sawtooth wave.

I can plug many of these into a mixer module to combine them.
Each wave module has a frequency multiplicator for harmonics.
Then I can mix the volume of each wave in the mixer module.
Finally, the low-pass filter smoothes things out.

Conclusion

Let’s wrap up. Well, super interesting project to make. There were unexpected complications like cable physics, and the sound generation was really tricky so I didn’t go as far as writing custom effects like distortion and such. Which is fine because plugging the theremin into a MIDI software opens the door to all sorts of cool sound effects. By the way the software I used is called VCV Rack 2, it is free, open source, and absolutely amazing, so check it out if you’re interested.

If you want cool Theremin music, check out Carolina Eyck and Grégoire Blanc

Have a good one everybody.

Music Credits