Add procedural and parametrizable audio support

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

Sound designers in the industry end up having to use tools such as FMOD or Wise, with not really any FOSS alternative. Godot users are forced to use this bit of proprietary software in order to have more advanced audio in their games.

Interactive music is being handled by a separate PR/proposal being worked on, so this focuses entirely on procedural audio.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

This would be a special AudioStream resource in Godot that would make it possible to have procedural/parametrizable audio playback in Godot.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

A new AudioStream class will be added: AudioStreamGraph. When edited, it will look like this:

The general idea is that the editor will be divided in two.

• Audio Graph
• Stream Positions

In the audio graph, several AudioGraphNodes will exist. Those have input and output Audio connections, as well as Controller connections (as a different type of data port).

The following audio graph nodes will be present:

Basic

• Output (comes by default)
• Stream: Allows setting any AudioStream in Godot. When one of these audio graph nodes is present, user will be able to adjust the offset on where it starts playing.
• Effect: Allows setting any AudioEffect from Godot (from the existing ones), which can change the sound accordingly. Audio effects export parameters, so these parameters will appear as input controller connections.
• Parameter: A named parameter, this parameter exports a controller output and can be named. This will appear as editable property in the AudioStreamPlayer[2D/3D] nodes. If a parameter is not connected to a controller, a default value can be set in the input port itself, like with Visual Shader and Visual Script.

Generators

These generate audio output:

• Oscillator: Simple oscillator (sine/saw/etc)
• Noise: Noise generator (pink/white/blue/etc).

Operators

These combine audio outputs:

• Mixer: Adds two audio streams
• Modulator: Multiplies two audio streams
• Frequency Modulator: Does frequency modulation to two audio streams

And the idea is to have what is common in this type of software for controlling parameters, such as LFO, Envelopes, etc. This is just the general proposal.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Audio is low level.

Is there a reason why this should be core and not an add-on in the asset library?

While this could be an add-on, games often require a mature and maintained solution for this. Given the popularity of tools such as Wise and FMOD, it sounds like it should be better this is core.

[MarioLiebisch]

I'm totally for this. Once planned to try something similar as an addon for chiptunes specifically, but I've never found the time to actually do so (so far).

First thought about the proposal above: I don't think this would need a timeline (nor should it have one). Instead there should probably be a Delay modulator node or something like that (along with others to cut parts of streams, too).

But just to put this in for discussion – random idea – might be worth maybe even making this fully scriptable (think VisualShader vs. Shader). The big thing would be the framework either way, considering most processing can be broken down to rather simple math operations.

Quick example for a sinus generator:

audio_node generator; // has no input sample

input float frequency = 10000.0f;
input float amplitude = 1.0f;

// Basically called once per requested sample
void modulate() {
  SAMPLE.lr = amplitude * sin(TIME * 2 * PI * frequency);
}

Or some stereo balance modulator:

audio_node modulator; // gets a sample as input

input float balance = 0.5f; // 0.0 = full left, 1.0 = full right

void modulate() {
  sample s = mono(SAMPLE); // reduce to mono
  SAMPLE.l = (1.0f - balance) * s;
  SAMPLE.r = balance * s;
}

So one script would define a single node (input or modulator), but full scripts could also be merged to single "complex" nodes again. Maybe overthinking, but I'd consider this very neat (and also very useful in education/audio processing) and maybe something for GPU processing, too (as long as no samples are based on previous ones)?

[theraot]

I would like to point out that we have AudioStreamGenerator/AudioStreamGeneratorPlayback. Sadly, getting the data from a stream into that is… ern… well, there was an issue about that, but never a proposal: AudioStreamSample.data compatibility with AudioStreamGeneratorPlayback - Without that, it is not easy to implement the addon, or to do this from code for a game.

[alexfreyre]

Maybe PureData (foss) and MaxMSP can give some ideas. They both are used to create interactive projects with procedural sounds/music/synthesizers.

I have seen another software that target procedural music creation for live events either via nodes or via code. It's called PraxisLive (foss). Also, you can make visual effects or shaders on it but it is made by a musician-programmer.

[starry-abyss]

Also what do you think about this FOSS project: https://github.com/soul-lang/SOUL? Perhaps, it could be leveraged as the "audio shader" language

[michaelhartung]

This is awesome :)!

Some thoughts from my perspective as Technical Game Sound Designer who's worked on a number of Unity/UE4 using both their native audio and FMOD/Wwise (and one Godot project so far :)):

One of the key features that makes FMOD / Wwise so attractive, apart from from all the convenient mixing and helpful profiling tools is that they're event driven. An event can do a plethora of things ranging from simple audio playback of a simple random wave container to setting global parameters that affect many currently playing sounds, stop other sounds etc.. The designers are given a lot of freedom while the code-side requirements are fairly non-complex most of the time.

Where "procedural" or parameter driven audio gets very interesting in a game audio context, is sounds affecting other sounds, when the system can talk to itself, update state, and talk back to the game loop so gameplay can react to sound instead of the other way around.

The one new native audio implementation that I'm most excited about currently is UE5's new meta sounds:

https://docs.unrealengine.com/5.0/en-US/AudioFeatures/MetaSounds/MetasoundsReferenceGuide/

Maybe there's some inspiration / ideas to be found there that could fit the Godot way of doing things.

[reduz]

@MichaelKlier I have seen some of the work that Epic is doing on MetaSound, but I have the feeling it is largely too complex for a musican or sound designer to use. For something so complex, I think it would be easier to just use PureData or similar as an add-on. I wanted to focus on having a simpler workflow where you can still do most of the most common audio manipulation tasks.

[gfazzito]

Strongly agree. Implementing an interface and a workflow that musicians and sound designers are used to (for example: a DAW like Reaper and a middleware like Wwise), would be an excellent improvement.

[bughandler]

MetaSound is too complex to use, it would be good enough for Godot to have a solution that is capable to replace FMOD/Wwise

[michaelhartung]

@reduz totally agree that Metasound is certainly complex. I didn't mean to imply to do sth. similar, merely suggesting to take inspiration/ideas by some of the functionality it implements :). Metasound is the only native audio implementation that I would personally consider to use over 3rd party Middleware (native UE4 is capable as well but still limited). Again, always considering the actual requirements of the project, not every project needs middleware.

I guess it really depends on what the end goal is. If it's about getting up to par with audio middleware out of the box to make it an attractive solution for sound designers to use, then it probably needs to offer at least the core set of features these tools have.

On that note, they only have very basic implementations of actual sound generators (sinewaves etc.), with Wwise having some more advanced ones for certain types of ambient sounds like wind and I don't know many people who are using them. And procedurally generating sounds like. wind/fire etc. using just sound generators paired with filter nodes, envelopes, lfo's etc. is a complex task on it's own.

By planning to offer this kind of functionality, when I read the outline correctly, you're kind of entering the PD/MaxMsp/Metasounds realm anyway.

I believe this discussion could benefit from actively bringing in more voices of audio people who are working with these tools on a daily basis.

[MarioLiebisch]

Regarding sound generators: I wouldn't expect Godot to have any complex generators, but certainly think it would be nice to have some very simple out of the box solution to generate basic effects or sounds similar to what bfxr and similar tools can do.

[SoyoTamo]

Without a doubt this would be something that I would use in all my projects. BFXR is ideal for small or rapid development projects, seeing something like this integrated into the engine together with the initial proposal would leave all the basics served with respect to audio, getting closer and closer to having to leave GODOT to do something, so less something simple.

[GeorgeS2019]

Topic: AudioStreamSample.data compatibility with AudioStreamGeneratorPlayback

Just curious, has the issue proposed here being addressed?

do some bit-shifting as well as a signed conversion for 16-bit samples

for interoperability between AudioStreamSample.data and AudioStreamGeneratorPlayback?