[Parser] Do not eagerly lex keywords #6541
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
|
|
||
| std::optional<std::string_view> Lexer::takeKeyword() { | ||
| if (curr) { | ||
| return std::nullopt; |
There was a problem hiding this comment.
Can you explain this part? It reads as "if there is a current token, ignore it and return nullopt". Why is that correct?
There was a problem hiding this comment.
This goes away at the end of the patch sequence, but it is required for correctness at these intermediate steps. Let's say the lexer reaches an integer and eagerly lexes it into a token stored in curr. That integer may be followed by a keyword, but takeKeyword() should fail until takeU32() (or similar) has been called first. Without this check, takeKeyword() would succeed even though the preceeding integer was never consumed.
There was a problem hiding this comment.
I see, thanks. Which is the PR where I can see the final step where it goes away, to get an idea for the direction?
|
Looks like this causes the spec tests to hang forever for some reason. Will investigate. |
tlively
force-pushed
the
parser-no-tok-parens
branch
from
45f9c5a to
d1df91c
Compare
tlively
force-pushed
the
parser-no-tok-keyword
branch
2 times, most recently
from
e47dc44 to
b08c1d7
Compare
Lex them on demand instead to avoid wasted work.
tlively
force-pushed
the
parser-no-tok-keyword
branch
from
b08c1d7 to
5fa865a
Compare
Lex them on demand instead to avoid wasted work.