IDEs and proc-macros

[Veykril]

Edit: I've written my current thoughts on the matter down in a blog post: https://veykril.github.io/posts/ide-proc-macros/#conclusion

This issue is meant as a place to collect information and ideas about the current completion(IDE support rather) dilemma we have with proc-macros.

Current State of Things

When typing inside an attributed item(or proc-macro) the user will inevitably create some form of syntax error at some point which currently causes those proc-macros to just bail out, emitting a compile_error! invocation that r-a will happily replace the entire item with. This has the downside that when typing in such an item, all ide features will momentarily stop working resulting in incorrect or straight up missing completions and syntax highlighting flickers(when semantic highlighting is being used).

First Attempt at Solving This

Our first idea to solve this was to nudge proc-macro authors towards doing more fallible expansions, meaning instead of only emitting a compile_error! on unexpected input, they should instead try to produce output as close as possible to what the macro would expand to in the happy case together with the compile_error!.
This actually seems to be not as good of an idea as we have first imagined.
It goes against syn's ways as the crate is designed to return early on errors, and in fact, as it turns out(I wasn't aware of this fact prior) attribute and derive macros have the invariant/contract¹ that they should always receive TokenStreams that parse to valid rust syntax.

Quoting @dtolnay's comment on the matter¹ which raises a lot of good points on this matter:

Passing syntactically invalid input to attribute macros, whether in the context of rustc or rust-analyzer, is not a good plan because doing the recovery, emitting diagnostics about the recovery, and then running macros on the original unrecovered input means that in general you're forcing macros to reimplement their own recovery independent and inconsistent with rustc/rust-analyzer — so rustc/rust-analyzer will diagnose what it thinks you meant, and the macro will diagnose what it thinks, and probably they won't align, resulting in dissonant user-facing messages. The correct behavior in my opinion is for rustc/rust-analyzer to perform its normal high-effort syntax recovery the same as for nodes that are not macro input, report diagnostics on that recovery as normal, then pass the recovered input for the attribute to proceed on.

Recovery in this context would involve rust-analyzer snipping out the nearest syntactically invalid syntax tree node and swapping in whatever syntactically valid placeholder/sentinel it wants in its place. It can then run the macro which will expand successfully, then provide autocompletion and other functionality to the programmer based on the position where it finds its sentinel in the expanded output, and the original snipped out syntax.

While the user is typing inside of attribute macro input this approach will give high quality results for the IDE in vastly more cases than rust-analyzer's current behavior, without trying to force changes for invalid macro input into all the attribute macros in the ecosystem.

Possible Solutions

So with this, we have a few options at hand currently:

• Nudge proc-macro authors to "fix" their macros.
  For attributes and derives(both of which expect rust syntax) this does seem like a bad idea after all after considering the points dtolnay has raised.
  For function-like proc macros on the other hand, as there is no rust syntax being passed as well as for macros that error out on semantic problems like expecting certain identifiers, in these cases we should still nudge proc-macro authors to make their macros recoverable.
  As there is nothing r-a can do in those cases.
  This also has problem that we would define an implicit interface² for r-a relying on exactly this behaviour for proc-macros, basically running the danger that if we would ever want to change our mind on this we would cause a eco system churn.
• Analyze the macro expansion output, and if it merely expands to a compile_error! keep the original item. Uncertain whether this is feasable in the first place, but it sounds like a bad trade off in general as a lot of macros that change their input item will not reflect their change in this case properly due to them still failing expansion.
  Still violates the proc-macro contract.
• Fix up the input nodes in a heuristic manner with what we expect they should be for completions/snip them out. This would ideally give the best results if properly implemented, but is also the most difficult to get right.
• Do nothing. Obviously a choice, and obviously a not so wise one.
  Still violates the proc-macro contract.

Fixing up the input nodes seems like the best approach to me personally, to reiterate that would mean:

• For attribute and derive macros, we will fix up the input items syntax errors as best as we can
• We ask proc-macro authors to still implement recovery strategies for when their macros receive unexpected but syntactic valid input, as IDEs fundamentally need help here from macro.

Now we aren't the only IDE for rust, IntelliJ seems to currently hardcode a bunch of popular attributes to special case(ignore) them³. So this obviously also weighs into the decision and we should agree on what to do here so that people do not have to write specific adjustments for each IDE(that would be a truly nightmare-ish scenario). cc @vlad20012

Assuming we go with the approach of fixing up/snipping out syntax invalid syntax nodes. The question remains how to do this reliably in a way that enables completion to work as good, if not better than with proc-macros trying to recover from everything as our initial plan was. This is the main question we would have to resolve here next.

On a side-note not relevant to fixing up syntax nodes, regarding completions with proc-macros, there is potentially interesting trick we could try to make use of to get proc-macros to help out IDEs with guiding completions outlined here⁴. This is not relevant to fixing up syntax nodes though.

Footnotes

1. https://github.com/rust-analyzer/rust-analyzer/issues/10468#issuecomment-975480478 ↩ ↩²

2. https://github.com/rust-analyzer/rust-analyzer/issues/10468#issuecomment-975315065 ↩

3. https://github.com/rust-analyzer/rust-analyzer/issues/10468#issuecomment-975436728 ↩

4. https://github.com/rust-analyzer/rust-analyzer/issues/7402#issuecomment-770196608 ↩

[matklad]

The question remains how to do this reliably in a way that enables completion to work as good

An interesting bit here is that, today, rust-analyzer doesn't actually know how a valid syntax tree looks like. We never encode in a reliable manner which parts of a tree are mandatory, and which are optional. This is something ungrammar potentially can help with, as it is intended to encode valid tree structure. Though, it doesn't encode valid input on the token level.

[flodiebold]

I'm not convinced that there's a 'contract' that attribute/derive proc macros can't receive invalid syntax. On the contrary, the fact that rustc passed invalid syntax to these macros for several versions, even if that was unintentional, means that there isn't such a contract. This plus the fact that letting proc macros handle the invalid syntax is the only way to get 100% correct behavior for some proc macros means to me that we should at least have a way to let proc macros opt in to receiving invalid syntax and handling it themselves.

(Also, I still think that having a full Rust parser in every proc macro is a bad idea. Whenever Rust introduces some new syntax, these proc macros will break until their parser is updated. This could be fixed by having the compiler provide parsing functions, at the expense of hugely increasing the proc macro API interface. In that case, we could also provide a proper error-resilient parser to the proc macros.)

[jgarvin]

If there were a way for macros to express "I am not going edit the token stream, only add around it, and I expect the stream to be normal rust code (as opposed to some arbitrary DSL)", would that case be much easier for rust-analyzer to do good completion for? As I understand it a lot of the tricky cases come from the possibility that the macro may do arbitrary edits, but a lot (most?) macros (both declarative and proc in my experience) just generate extra code to surround existing valid rust code. Either generating extra items after a struct or inserting extra control flow around a block, without actually changing the starting struct or block contents, or only changing it in a very limited way (e.g. it removes #[blah] markers on struct fields because those are just the indicator to tell it to generate some extra field specific code). This combined with macro hygiene seems like it would make it so rust-analyzer would know that it's valid to offer any completion option that would make sense if the struct/block were present without being a macro input. But I'm no expert, maybe there are still trivial examples where completion would be ambiguous?

[flodiebold]

If there were a way for macros to express "I am not going edit the token stream, only add around it, and I expect the stream to be normal rust code (as opposed to some arbitrary DSL)", would that case be much easier for rust-analyzer to do good completion for?

Macros that do just that already work very well, so such an annotation wouldn't really give us anything.

[Hades32]

As an FYI for everyone reaching this after having their while code (e.g. of a Cloudflare worker) marked as red with unexpected token: rust-analyzer:
The easiest way to workaround this is to comment out the attribute (e.g #[event(fetch)]) while editing and uncomment it, when you're done.

[twitchax]

Want to make sure that the request to Rocket (rwf2/Rocket#1967) is tracked here.

I don't think Rocket, for example, substantially edits the code inside of the endpoint body, so it makes me wonder if there could be a rust-analyzer option to opt-in to ignoring certain macros. It's not a great north star, but it might be a good stop gap.

As an example, when I am working on one of these endpoints, I go comment out the macro, finish my work, and then uncomment the macro. Might be nice if I could just blanket ignore those macros so that working on the code of each would is painless. Even though other parts of the code may throw errors in r-a, those can be out of sight, out of mind.

[memoryruins]

@twitchax

it makes me wonder if there could be a rust-analyzer option to opt-in to ignoring certain macros

With #11193, it is now possible to specify macros to be replaced by dummy expanders https://rust-analyzer.github.io/thisweek/2022/01/10/changelog-111.html.

---

If desired, can disable different kinds of proc-macros altogether for now as well https://rust-analyzer.github.io/manual.html#configuration as well as specific diagnostics (not for everyone, but this can be a reasonable option for some).