deno compatibilty [feature request]

[mash-graz]

it would be really useful, if your audiocontext pollyfill wouldn't only realize a reliable common ground for audio apps in different browsers, but also support elementary webaudio capbilities in deno to test and utilize browser targeted java script multimedia code also on this plattform resp. in headless mode.

deno tries to mimic the browser APIs as much as possible, but webaudio isn't supported up to now. nevertheless, i think, it could be realized in some very minimalist kind, just as recently demonstrated by peter salomonsens impressive work on assemblyscript/wasm based audioworklet emulation, which also use the webaudio APIs and just simply write the signal to stdout.

it would really appreciate, if a more complete and standard compliant webaudio polyfill, like yours, would provide similar capabilities usable even in deno.

[chrisguttandin]

Hi @mash-graz,

that's an interesting idea. I don't know much about Deno yet. I only watched some talks by Ryan Dahl introducing the basic idea.

If you need a library to replace or mimic the Web Audio API for tests I would recommend standardized-audio-context-mock. I created it exactly for that use case. I'm not sure if it works with Deno, though.

If you want to run Web Audio code without a user interface it might be an option to run Firefox or Chrome in headless mode.

There are also some existing projects which try to bring the Web Audio API to Node.js: https://www.npmjs.com/package/web-audio-engine, https://github.com/audiojs/web-audio-api. But they seem to be unmaintained and I also don't know if they can be used with Deno.

I'm definitely not against adding Deno support but I guess it's not an easy endeavor. I currently think it would be better to provide that functionality as a standalone library.

[chrisguttandin]

Hi @petersalomonsen, I hope it's okay to include you in this issue. Since your work was mentioned above, do you have any input on how this could be solved best?

[mash-graz]

@chrisguttandin
thanks for your inspiring response.

There are also some existing projects which try to bring the Web Audio API to Node.js: https://www.npmjs.com/package/web-audio-engine, https://github.com/audiojs/web-audio-api. But they seem to be unmaintained and I also don't know if they can be used with Deno.

unfortunately they look indeed rather outdated and unmaintained. :(

nevertheless i could modify the StreamAudioContext implementation of web-audio-engine by a few quick and dirty hacks to run in deno resp. successful write raw audio data to stdout and pipe it to aplay. and at least some very simple webaudio tests worked with good grace in this minimalist environment. :)

it's definitely not a very promissing and exemplary solution, but it's at least an interesting starting point to experiment and work on a better solution...

I currently think it would be better to provide that functionality as a standalone library.

yes -- most likely this would be a better solution, because the required modifications are hardly compatible to the official webaudio API, and the strict standard compliance of your implementatione is one of its most impressive advantages.

nevertheless it would be really desirable, if you could sometimes provide something similar to this StreamAudioContext in your implementation as well, because the other software options are by no means a realistic alternative to your much nicer and more modern development approach.

sure -- it could also make sense, to realize this kind of extension by help of servo-media or something similar as a proper deno-plugin to get closer to the real performance and feature set of nowadays browser implementations, but this more minimalist and flexible javascript and wasm based approaches also have their advantages.

[petersalomonsen]

Great projects (Deno and this) :-)

It would be super cool if possible to use JACK, which I hope to look into myself for the WebAssembly synth when time permits.

I haven't looked into what this would mean for Web Audio, but the AudioWorkletProcessor interface is very similar to the rendering callback of JACK. So for my WebAssembly synth I would just have to call the same method as the AudioWorkletProcessor does. For WebAudio I guess we would need another type of Offline/Headless AudioContext where we could "pull" a buffer of 128 frames like theprocess method on the AudioWorkletProcessor does ( instead of WebAudio writing to a device, let JACK pull from WebAudio ).

The "pulling" approach is better than pushing in order to achieve low latency with the best stability.

[mash-graz]

@petersalomonsen
thanks peter!

It would be super cool if possible to use JACK, which I hope to look into myself for the WebAssembly synth when time permits.
...
The "pulling" approach is better than pushing in order to achieve low latency with the best stability.

yes -- although your stdout hack was indeed a very encouraging first step to overcome the requirements of huge base infrastructure requirements and APIs which are only available while running inside the browser, it's obviously a little bit to simple to be seen as a satisfying final solution.

we definitely should try to write the output straight to one of the more suitable audio APIs on our machines. especially in the context of deno, where we do not have more advanced direct debugging capabilities until now, stdout is usually polluted by lots of verbose debugging reports, which are hardly compatible with piping the output to audio players ;) -- nevertheless your very radical approach looked exceptionally inspiring to me and was a first motivation to think about more suitable ways to brake out of the given webaudio base infrastructure requirements.

but at the end we are very likely discussing something, which is already at least partially solved by the Audio Device Client proposal and it's fist prototype implementations. this API proposal looks like a rather interesting solution to our needs, because it's a relative small standard proposal, which should be reasonable implementable even outside of the huge browser development companies and all the manpower and expertise of highly specialized development teams. its also providing an answer to WASM and WASI related interface access requirements, that you mentioned recently in your presentation at webassembly summit.

concerning your preference for JACK and pulling based audio interface APIs, i'm unfortunately still not able to express a clear personal opinion.

JACK is definitely a very well solution for the high end users and dedicated utilization in music production, but more common and platform independent approaches also have their obvious advantages.

in case of of deno, which is written in rust, cpal would be most likely the preferred intermediate library for audio device access by most developers, although it doesn't support JACK until now, but portaudio and even more linux focused solutions, like pipewire, could be attractive solutions as well.

last year there were a lot of discussions and radical changes going on in the cpal and rust audio developer circles concerning the most optimal/compatible audio exchange API and main audio processing loop handling (see: RustAudio/cpal#278 and RustAudio/cpal#301), and this somehow also corresponds to the preferred realization of async interfaces in deno which represent one of the most significant differences in regard to node resp. callback hell and also effects the compatibility to older webaudio emulation solutions, which were written long ago. i personally simply have to think a little bit more about this specific aspect before i'm hopefully sometimes able to see more clear, how to solve this challenge in my own code in a consequent and acceptable manner.

well -- we'll see, if i can figure out at least another small step to get closer to this desires, which we have in common. ;)

[petersalomonsen]

yeah I was really just suggesting JACK because of the pulling approach which is also the same for CoreAudio and ASIO. But the Audio Device Client proposal seems to already have this covered, and then it would be no problem to support JACK / CoreAudio / ASIO.

And for stdout it would just be to write an Audio Device Client that pulls from the context and writes to stdout.

Looks like Audio Device Client is what we want then? (It sure does answer my WASI requirement concerns that I mentioned at the summit). Deno wouldn't need to implement the actual client, just the possibility to attach a client (and some example clients).

[chrisguttandin]

I thought I already posted it to this thread but apparently I forgot about it. 🤷‍♂️ There is also a project called LabSound which describes itself as a fork of Blink's Web Audio API. It's of course written in C++ but maybe it's possible to create a native deno/node module that provides JavaScript bindings for it.

[chrisguttandin]

It's also worth mentioning that @nick-thompson (I hope you don't mind me linking you here.) announced a new project which is meant to allow audio programming with Node.js. I guess it could be used with deno, too.

https://www.youtube.com/watch?v=AvCdrflFHu8

[nick-thompson]

Thanks @chrisguttandin! Indeed Elementary could be fit into deno as well, and I've already looked at it a bit myself. In the near term I will be keeping my focus on Node, but as Deno grows I could very well see supporting both

[chrisguttandin]

There is currently an open issue over on the audiojs/web-audio-api repo discussing the project's future. audiojs/web-audio-api#82

I think it's somehow related to what we talked about here.

[orottier]

In case anyone is still looking for a Web Audio API implementation outside of the web browser: we have a complete NodeJs implementation over at https://www.npmjs.com/package/node-web-audio-api
It uses a Rust audio backend so it does not compete on your JS main thread.
Support for the AudioWorkletNode was added very recently so it may still have some rough edges, we would love to hear about those over at our repo https://github.com/ircam-ismm/node-web-audio-api