Skip to content

Conversation

@jeschkies
Copy link
Owner

@jeschkies jeschkies commented Oct 2, 2019

This implements the token scanning.
I tried to stay as close to the original Java implementation as possible while still
being "rusty".

Differences to the Java implementation:

  • There is no null check and a few if-else blocks became unwrap_or'('\0').
  • Literals are not of type Object but use String and f64.
  • is_digit and is_alpha are much simpler in Rust.
  • The KEYWORDS map is not only static but create at compile time thanks
    to phf.

Possible improvements in Rust:

  • One could use a PeekableIterator for Chars. However, this would diverged from
    the original scanner quite a bit.
  • One could strike for zero copy for lexemes. To keep it simple I stuck with String.
  • The strangest part is creating a substring with
    self.source .get(self.start..self.current).expect("Source token is empty.");
    
    I was afraid that changing it would make it harder to relate to the original chapter.
  • The scanner could be less stateful and avoid self.current and self.start.

@jeschkies jeschkies merged commit 9fef15e into master Oct 10, 2019
@jeschkies jeschkies deleted the karsten/chapter-4 branch October 10, 2019 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants