Support tokio

[jasta]

General tracking issue to work on support for tokio in the esp-rs ecosystem. As I mentioned in #9 (comment), I'm looking to get this off the ground by first introducing a solid poll implementation in mio, then fixing things along the path in tokio, socket2, etc.

[jasta]

Looks like your socket2 changes weren't merged either: rust-lang/socket2@master...esp-rs-compat:socket2:master . Just bumped into this one as I'm working to get a demo of tokio + esp32 off the ground. Any gotchas I should be aware of if I try to upstream this work?

[jasta]

I've almost got it working with my forks but it's crashing at runtime. I'll post my sample repo and keep iterating tomorrow...

[jasta]

Looks like it's failing because you need to call esp_vfs_eventfd_register before eventfd will work. That's news to me, that feels like a pretty surprising little gotcha folks will run into. Hmm, I wonder if smol or polling are doing that for you somehow???

[ivmarkov]

Looks like it's failing because you need to call esp_vfs_eventfd_register before eventfd will work. That's news to me, that feels like a pretty surprising little gotcha folks will run into. Hmm, I wonder if smol or polling are doing that for you somehow???

They are not. I'm calling that manually as well.

[ivmarkov]

Looks like your socket2 changes weren't merged either: rust-lang/socket2@master...esp-rs-compat:socket2:master . Just bumped into this one as I'm working to get a demo of tokio + esp32 off the ground. Any gotchas I should be aware of if I try to upstream this work?

Yes: my changes are against an older version of socket2 (0.3 if I'm not mistaken). You'll probably need these changes either into their 0.4 or their 0.5 (master) branch. Not sure what tokio depends on, and I wasn't aware that tokio needs socket2 in the first place.

[ivmarkov]

While support for tokio is certainly very nice as some smaller, network-related crates can be utilized on the ESP IDF with little to no changes then, I assume you realize that you'll still need a variation of async-task as the main executor, and tokio running via the smol "compat" layer?

I mean, is tokio's executor customizable to that level? The first wave of async drivers for esp-idf-hal will have their wakers awoken directly from an ISR. So unless you can customize tokio's task scheduler in a similar fashion, you still need async-task (in an incarnation identical or similar to edge-executor) at the helm, or we need to think of a way where the async drivers' ISRs don't awake the wakers directly, but do some trick like awaking a high priority FreeRtos thread/task (possibly via the lightweight FreeRtos task notification mechanism or via an event group), which task - in turn - calls waker.wake() on the driver and thus does the executor awakening and task scheduling in a normal thread execution context.

[jasta]

While support for tokio is certainly very nice as some smaller, network-related crates can be utilized on the ESP IDF with little to no changes then, I assume you realize that you'll still need a variation of async-task as the main executor, and tokio running via the smol "compat" layer?

The main reason for my work is just to shave down the sharp corners in the async story for esp32 specifically and embedded platforms more generally. rust-esp32-std-demo is a good place to organize the discussion around the async story in general I think (as opposed to the work in esp-idf-hal which is great but doesn't speak much to async as a whole). I'm sure you're aware that right now in Rust it's nigh impossible to build an ergonomic and high quality networking library that is also agnostic to a particular async framework. I started my journey looking for a good re-usable CoAP library that I can port around with me to various embedded projects and I quickly realized this just isn't realistic for so many reasons -- most of them bad ones. Then I started looking at matter-rs and ofc that meets the same fate:

https://github.com/project-chip/matter-rs/blob/sequential/matter/Cargo.toml#L53
https://github.com/project-chip/matter-rs/blob/sequential/matter/src/transport/runner.rs#L77

So what I'm really hoping for here is to just make this story a little neater overall by not making it impossible to even compile code that might have happened to use tokio net, time, etc as if we're somehow asserting that it's bad to choose tokio's facilities for these things but good to choose smol or vice versa. Doesn't really make sense to me why it should even matter which one you use so long as you abstract the actual executor part away and design with good test-driven practices so you can run the core of your code on the host machine without a fancy embedded executor.

[jasta]

Not sure if this helps or hurts my point but I just realized you committed to the matter-rs implementation that is getting itself so tightly coupled with smol and esp-idf. Hopefully we can at least agree that it would be much better if Rust's async story was a little bit more consistent across platforms. Anyway, happy hacking :P

[ivmarkov]

The main reason for my work is just to shave down the sharp corners in the async story for esp32 specifically and embedded platforms more generally. rust-esp32-std-demo is a good place to organize the discussion around the async story in general I think (as opposed to the work in esp-idf-hal which is great but doesn't speak much to async as a whole).

I don't have any preference where the discussion is handled, although in a way rust-esp32-std-demo is in my space, while the rest of the stuff is at least in the esp-rs echosystem. Which rust-esp32-std-demo is demoing, after all.

With that said, I'm a bit perplexed by your statement that esp-idf-hal doesn't speak much to async. As I just tried to explain, the async story of esp-idf-hal WILL require (in fact already DOES - as the gpio driver is already released) the executor to have certain capabilities - which I'm raising the point that tokio might not have; and contributes to edge-executor this. tokio + ISR possibly not playing well is of course not the end of the world, as you can combine tokio with async-task, but still - FYI of sorts.

The thing is, my view on what is async on embedded has certainly evolved to include other stuff besides networking. What happens if you want to schedule networking async workloads together with embedded driver async workloads?

I'm sure you're aware that right now in Rust it's nigh impossible to build an ergonomic and high quality networking library that is also agnostic to a particular async framework. I started my journey looking for a good re-usable CoAP library that I can port around with me to various embedded projects and I quickly realized this just isn't realistic for so many reasons -- most of them bad ones.

I don't necessarily agree with that. It is much more work, involves a lot of generic metaprogramming, but IS possible. It is just that folks are lazy and choose the easy path. Not having a single, "standard" async API (ideally no_std compatible) for networking and FS IO is IMO the culprit, not the executor. Where the former is probably is blocked on AFIT and friends. Including dyn async traits (to lessen the generics pressure) - which are not even on the roadmap indeed.

Then I started looking at matter-rs and ofc that meets the same fate:

https://github.com/project-chip/matter-rs/blob/sequential/matter/Cargo.toml#L53 https://github.com/project-chip/matter-rs/blob/sequential/matter/src/transport/runner.rs#L77

Yeah. Thanks for citing my own code. :-)

Did you notice this though? It gives you a lower level API, which you can arrange with whatever async executor / task spawner you want. Which of course brings my earlier topic that we have a certain laziness (or maybe unwillingness to take too much complexity) on both sides - library creators and library consumers.

By the way - w.r.t emulating task spawning with select and ending up with a "one giant future": that might be an OK compromise in the embedded space - specifically for networking - because:

• MCU resources are so scarce, that you probably want all your networking code (and more!) scheduled on a single thread, in an executor which is local to that thread. Which by the way relieves you from futures needing to be Send + 'static which pays well with no_std
• The extra latency you'll get this way (as the giant future has a big internal state and polling it is more expensive) is probably OK, at least for a networking code in embedded, where high latencies are tolerated

So what I'm really hoping for here is to just make this story a little neater overall by not making it impossible to even compile code that might have happened to use tokio net, time, etc as if we're somehow asserting that it's bad to choose tokio's facilities for these things but good to choose smol or vice versa. Doesn't really make sense to me why it should even matter which one you use so long as you abstract the actual executor part away and design with good test-driven practices so you can run the core of your code on the host machine without a fancy embedded executor.

So first of all - absolutely! I'm not saying bringing tokio to ESP IDF (or embedded in general) does not have value - quite the opposite!

I'm just stating that the road is longer than that, and more complex than that. One reason is - as I mentioned multiple times - non-networking, driver-based code. The other is - by the way - that STD-only, tokio (or async-std) based crates are often not optimized for embedded. Meaning, they box and arc like mad pushing too much pressure on the alocator and bringing unpredictability at runtime w.r.t. OOMs due to heap fragmentation and general incapability to estimate your heap memory consumption statically. And folks are complaining, you know. But then I'm also complaining, as Rust's placement-new story sucks too.

[ivmarkov]

Not sure if this helps or hurts my point but I just realized you committed to the matter-rs implementation that is getting itself so tightly coupled with smol and esp-idf. Hopefully we can at least agree that it would be much better if Rust's async story was a little bit more consistent across platforms. Anyway, happy hacking :P

Sorry if I'm a bit blunt here, but apparently you do not understand the matter-rs code base (yet). There is nothing in matter-rs that tightly couples it with either ESP IDF or smol. You are looking at optional dependencies. Quite the opposite: everything there is executor-agnostic, reactor-agnostic and networking-stack agnostic (I mean, in the no_std and sequential branches, which is my work). Hell, the baremetal embedded-esp folks are running these with embassy-net and (CORRECTED - used to say smol) smoltcp!

Maybe you should start by looking at the README file in the main branch that explains the minimal requirements of matter-rs w.r.t. the underlying platform. :-)

I also don't have any vested interest to make matter-rs bound to the ESP IDF. For one, I'm not an Espressif emplyee. It just so happens that I often hack on the ESP32, so the first embedded demo of matter-rs happens to compile for the ESP IDF. Hopefully other folks will contribute for other MCUs.

If I wanted to bind matter-rs to Espressif, I would've been working in the exact opposite direction, don't you think? I would've kept matter-rs STD-only and allocating like mad (as ESP IDF is the only MCU framework which is STD compatible - which is by the way also my work).

I did exactly the opposite - no_std, no allocations, and async. I.e. compatibility with Embassy and whatnot echo-systems. :-)

Hope that clarifies my position and philosophy w.r.t. async Rust on embedded.

[ivmarkov]

@Dirbaio @lulf Sorry guys for pulling you here out of nowhere, but really - nobody from the Embassy echosystem having any interest of assembling an onoff_light demo on top of the sequential branch of matter-rs ^^^? The pre-reqs for running matter-rs on any MCU platform are listed here.

It seems that the existing, ESP IDF-only demo is giving a very wrong impression w.r.t. matter-rs philosophy and dependencies to folks in the embedded space.

I was looking lately that you have the W5500 ethernet driver running with embassy-net. And you've got network stack offloading to an ESP32 MCU via an embassy-net driver as well.

I would have implemented an embassy-net demo (ideally - for a non-ESP32 host MCU!) myself, but I only have so much free time. :-(

But I can support of course. :-)

[ivmarkov]

One roadblock of sorts is that the current matter-rs codebase needs ~ 140K (!) of memory. Reducing the number of sessions from 16 to 8 and the number of exchanges - say - from 8 to 4 should reduce this twice or so, though.

[jasta]

Thanks for all the info @ivmarkov, it's gonna take me a bit to digest all of this. I am somewhat new to this space and am responding mostly to how difficult it is to be productive as a beginner. I come from an Android background where early on (10+ years ago) the story was very similar: developers had a lot of cognitive overhead to build anything and reusing components across desktop /server use cases was effectively impossible because of memory constraints and incompatible APIs.

So what I'm trying to improve is that it's a lot of work to build quality reusable libraries. Just like I saw with Android is that if you make it too nuanced and heady to do the right thing you'll just end up with a lot of junk in the ecosystem and the platform stagnates. I understand things like no_std and reducing the use of alloc is always going to be a bit of extra design work, but the async story IMO is adding even more headache without a very good justification (why for example are there seemingly so many implementations of the exact same I/O code across mio, polling, socket2, etc that each have their own quirks and gotchas???)

[ivmarkov]

No prob, and if you could keep up the good work on the tokio-to-poll port, that would be really appreciated! :)