Support WebAssembly Component bindings, aka wasip2, in addition to javascript bindings

[brooksmtownsend]

The current wasm implementation supports compiling to wasm with wasm_bindgen, which is meant to provide bindings for usage from JavaScript in the browser. This aligns with many of the concepts from wasi_preview_1 and uses the browser's fetch() API to actually make the request. This works great for the browser, now there's tons of applications that want to use a higher level API for making requests from Wasm and run them on the server. The next iteration of WASI, wasi_preview_2, is the perfect basis for this.

WASI Preview 2 released earlier this year WebAssembly/WASI#577 with a common set of standard interfaces, including wasi-http which provides an interface both for incoming and outgoing HTTP requests. What I would love is for reqwest to support binding to wasi:http/outgoing-handler which would let Rust devs compile reqwest directly to a Wasm component targeting p2. I am a maintainer of wasmCloud which uses wasi-http for it's requests and, as you can see in this sample, using a higher level library in Rust greatly simplifies the code.

I opened #2290 to start implementing this but I think there's a couple of questions to answer:

1. How do we want to gate compiling for the browser v.s. compiling to a component? Is it feature flags, and if so, which one is the default? (It's worth noting there is a specific target for components, wasm32-wasip2, however this is a new target and doesn't properly denote the difference between "browser" wasm and a component)
2. Does feat(wasm): add support for wasip2, aka webassembly components, using wasi-http #2290 look reasonable or should we approach the problem differently? Most of the logic happens in the client's fetch function, and you can see the translation from the reqwest::Request and the wasi::http::OutgoingRequest.
3. Do we like the file organization in feat(wasm): add support for wasip2, aka webassembly components, using wasi-http #2290 with the refactor or prefer for components to be a new folder? WASI Http module #1956 contains a reference implementation with slightly old bindings with a different file structure

This PR is essentially a duplicate of #1956, I just promised to open another issue so we can dedup as necessary.

I think this PR is related to #1977 but it might not entirely supercede it since there's a deeper discussion on abstraction.

Thanks @seanmonstar for being receptive to the issue + change. Looking forward to the discussion!

[brooksmtownsend]

Added example in #2290, and for additional clarity here's what's upcoming in terms of wasm32 targets in Rust. Eventually I think we can remove the feature flag and instead use the target as a differentiator of when to use the p1 bindings v.s. p2 bindings.

[brooksmtownsend]

@seanmonstar would you like to see any additional commentary/fixes/thoughts here? Do you have any questions?

One question I have for you is if you think we could include this in the synchronous version of the crate, since wasip2 components don't technically native async we could avoid the use of the futures::block_on function.

[smndtrl]

One of the major issues I experienced with wasi-http experiments in reqwest is that the (non)overlap in API surface leads to issues with third party crates relying on reqwests non wasm target APIs. Especially cookies, proxies and tls are not available on wasi-http. What expectation should a user have in terms of using APIs and/or third party crates?

While I would be happy to have a 100% shim with warnings when features are not available when using the wasi-http, it might not be a good way to go.

[brooksmtownsend]

@smndtrl I think your comment makes a lot of sense. With the current level of standard HTTP support for WASI, there's not really an expectation for a user to be able to use cookies/proxies/tls right now. We could always model that in terms of a different interface, but it would be up to the WebAssembly host to implement that and that doesn't belong in a general purpose library.

The reality right now is that this type of feature would make Wasm applications in Rust that use HTTP look a lot nicer, and it would only be a small step for existing Rust applications that hope to seamlessly swap over to the wasi-p2 target. The shim with warnings (or failure to compile) is essentially what I was thinking for the first pass here.

I'm going to spend a bit of effort making the PR smaller in scope to, hopefully, make the surface area smaller for testing + questions

[pimeys]

Could the recent development of having wasm32-wasip2 as a tier2 target, and the work getting wasip2-network code in std be an alternative approach?

rust-lang/rust#129638

I know TLS might still be an issue, or not. If std::net works on wasi, and mio works on wasi, what would prevent reqwest working on wasi?

[brooksmtownsend]

I think that wasm32-wasip2 as a tier2 target and work getting done in std is certainly an alternate approach, but one that may take a bit more time. TLS is definitely an issue there, but that's a bit out of my realm of expertise to fix upstream.

In the interest in preventing this from going stale, I've done some refactors to remove some of the bits that were blocking this merging, like the async executor that needed an internal nanosecond sleep to continue to measure progress. Essentially, this makes the requests blocking again, but it at least pushes this forward. Another definite benefit of not having this be async is removing the need for a futures executor within the Wasm component, which just added friction.

In short, I'm going to close #2290 in favor of #2453 and prepare it for review. This does depend on servo/rust-url#983 to work on stable, or the main branch of rust-url to make this work on nightly. I was hoping that perhaps we could get an upstream release of this, but in the meantime I've introduced a patch for the url & form_urlencoded crates. Removing this patch should be considered a requirement to make this code ready for review.

[dakom]

@brooksmtownsend - I'm not sure if this is the right issue for me to track, but it certainly reads like it is :)

I'm currently using wstd 0.5 and it has native support for http. See the HTTP Client under examples: https://docs.rs/wstd/latest/wstd/#examples

However, the API is not really ideal and I'd rather use reqwest with its more battle-tested ergonomic API. The bump to wstd is fairly new (it used to have this whole "reactor" thing, no more), but I'm using it in wasmtime with wasi p2 targets and so far it seems to be great

Would that simplify / speed up getting native WASI p2 support in reqwest?

[brooksmtownsend]

@dakom I think it definitely would, I'd be happy to help integrate this as long as the dependency is fine to bring in for wasm32-wasip2 targets 😄