Improved error recovery for unclosed strings (including f- and t-strings) #20848

MichaReiser · 2025-10-13T17:15:16Z

Summary

This PR improves our lexer to preserve STRING tokens instead of converting them to Unknown if a string literal misses its closing quotes.
Instead of converting to UNKNOWN, it sets a flag on the string literal that allows upstream tools to check if it's an unclosed string literal.

The benefit of preserving string literals is that it gives us much better error recovery because the parser now recognizes those literals.
That means, ty will correctly infer the literal type for a = "unclosed to be Literal["unclosed"].

Unfortunately, preserving the kind for unclosed string literals regressed the f-string's and t-string's recovery mechanism. So, I went ahead and improved that too.

There are a few improvements:

Preserve the F-STRING middle even if it's unclosed (e.g. f"unclosed) instead of parsing this as f""
Better recovery for missing }. E.g., the parser now matches the quotes for f"{ab" instead of assuming that the closing quotes start a new string
Better recovery for r format specifiers if the } is missing: f"{ab:r" now parses the r as the raw conversion flag rather than r" the start of a raw string literal

Fixes #19751
Fixes #20849

Review

You probably want to skip the first commit :) It updates all snapshots to now include the unclosed: <UNCLOSED> flag.

Test Plan

Reviewed and updated the snapshot tests. I also reviewed all usages of TokenKind::String to find cases where the missing closing quote could now cause issues.

This change should have no impact on AST-based lint rules or the formatter because they both only run when there are no parse errors.

MichaReiser · 2025-10-13T17:15:43Z

crates/ruff_linter/resources/test/fixtures/flake8_implicit_str_concat/ISC_syntax_error.py

+# This is also true for
 # unterminated f-strings.


I know this looks silly, but keeping the comment over two lines reduces the snapshot changes.

MichaReiser · 2025-10-13T17:18:38Z

crates/ruff_python_parser/src/token.rs

 bitflags! {
    #[derive(Clone, Copy, Debug, PartialEq, Eq)]
-    pub(crate) struct TokenFlags: u8 {
+    pub(crate) struct TokenFlags: u16 {


While this increases the size of TokenFlags, it doesn't increase the size of Token. Which is why I didn't bother with any fancy encoding (e.g. it's unclosed if RAW_STRING_UPPERCASE and RAW_STRING_LOWERCASE are set)

github-actions · 2025-10-13T17:26:36Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

crates/ruff_python_parser/src/lexer.rs

MichaReiser · 2025-10-14T09:16:18Z

crates/ruff_python_parser/src/lexer.rs

                    self.current_flags |= TokenFlags::UNCLOSED_STRING;

                    self.push_error(LexicalError::new(
-                        LexicalErrorType::StringError,


This error was just wrong. a = "string<EOF reported an unexpected string error rather than unclosed string literal

ntBre

Nice! This looks great to me!

…rable * origin/main: [ty] Add (unused) `inferable` parameter to type property methods (#20865) Run macos tests on macos (#20889) Remove `release` CI job (#20887) [ty] CI: Faster ecosystem analysis (#20886) Remove `strip` from release profile (#20885) [ty] Sync vendored typeshed stubs (#20876) [ty] Add some completion ranking improvements (#20807) Improved error recovery for unclosed strings (including f- and t-strings) (#20848) Enable lto=fat (#20863) [`pyupgrade`] Extend `UP019` to detect `typing_extensions.Text` (`UP019`) (#20825) [`flake8-bugbear`] Omit annotation in preview fix for `B006` (#20877) fix(docs): Fix typo in `RUF015` description (#20873) [ty] Improve and extend tests for instance attributes redeclared in subclasses (#20866) [ty] Ignore slow seeds as a temporary measure (#20870) Remove parentheses around multiple exception types on Python 3.14+ (#20768) Update Black tests (#20794)

…nt-sets * dcreager/non-non-inferable: (174 commits) [ty] Add (unused) `inferable` parameter to type property methods (#20865) Run macos tests on macos (#20889) Remove `release` CI job (#20887) [ty] CI: Faster ecosystem analysis (#20886) Remove `strip` from release profile (#20885) [ty] Sync vendored typeshed stubs (#20876) [ty] Add some completion ranking improvements (#20807) Improved error recovery for unclosed strings (including f- and t-strings) (#20848) Enable lto=fat (#20863) [`pyupgrade`] Extend `UP019` to detect `typing_extensions.Text` (`UP019`) (#20825) [`flake8-bugbear`] Omit annotation in preview fix for `B006` (#20877) fix(docs): Fix typo in `RUF015` description (#20873) [ty] Improve and extend tests for instance attributes redeclared in subclasses (#20866) [ty] Ignore slow seeds as a temporary measure (#20870) use existing method Remove parentheses around multiple exception types on Python 3.14+ (#20768) Update Black tests (#20794) just the api parts [ty] Fix further issues in `super()` inference logic (#20843) [ty] Document when a rule was added (#20859) ...

MichaReiser added the parser Related to the parser label Oct 13, 2025

MichaReiser commented Oct 13, 2025

View reviewed changes

AlexWaygood reviewed Oct 13, 2025

View reviewed changes

crates/ruff_python_parser/src/lexer.rs Outdated Show resolved Hide resolved

MichaReiser added 4 commits October 14, 2025 11:06

Add unclosed flag but don't set it anywhere

5cc3121

Set UNCLOSED_STRING flag for string literals with missing closing quotes

c0737de

Improve f-string recovery

f8d5130

Use same heuristic for triple quoted strings

a012e2b

MichaReiser force-pushed the micha/unclosed-string branch from 698f74c to a012e2b Compare October 14, 2025 09:06

MichaReiser commented Oct 14, 2025

View reviewed changes

MichaReiser added 2 commits October 14, 2025 11:17

Clippy & review TokenKind string usages

38a65f5

Update comments

f7d5c56

MichaReiser requested a review from dylwil3 October 14, 2025 09:31

MichaReiser marked this pull request as ready for review October 14, 2025 09:31

MichaReiser requested a review from dhruvmanila as a code owner October 14, 2025 09:31

MichaReiser mentioned this pull request Oct 14, 2025

[ty] Add some completion ranking improvements #20807

Merged

MichaReiser changed the title ~~Better error recovery for unclosed strings (including f- and t-strings)~~ Improved error recovery for unclosed strings (including f- and t-strings) Oct 14, 2025

MichaReiser requested a review from ntBre October 14, 2025 16:54

ntBre approved these changes Oct 15, 2025

View reviewed changes

MichaReiser merged commit 4fc7dd3 into main Oct 15, 2025
38 checks passed

MichaReiser deleted the micha/unclosed-string branch October 15, 2025 07:50

AlexWaygood mentioned this pull request Oct 15, 2025

Update parser snapshots #20893

Merged

BrewTestBot mentioned this pull request Oct 16, 2025

ruff 0.14.1 Homebrew/homebrew-core#249923

Merged

MichaReiser mentioned this pull request Oct 21, 2025

Improve error recovery for strings and f-strings #11916

Closed

ntBre mentioned this pull request Oct 21, 2025

ISC001 panic: begin <= end (1 <= 0) when slicing ' #21023

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improved error recovery for unclosed strings (including f- and t-strings) #20848

Improved error recovery for unclosed strings (including f- and t-strings) #20848

MichaReiser commented Oct 13, 2025 •

edited

Loading

Uh oh!

MichaReiser Oct 13, 2025

Uh oh!

MichaReiser Oct 13, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

MichaReiser Oct 14, 2025

Uh oh!

ntBre left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Improved error recovery for unclosed strings (including f- and t-strings) #20848

Improved error recovery for unclosed strings (including f- and t-strings) #20848

Conversation

MichaReiser commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review

Test Plan

Uh oh!

MichaReiser Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

MichaReiser Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

Uh oh!

MichaReiser Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

ntBre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MichaReiser commented Oct 13, 2025 •

edited

Loading

MichaReiser Oct 13, 2025 •

edited

Loading

github-actions bot commented Oct 13, 2025 •

edited

Loading

`ruff-ecosystem` results