Skip to content

[syntax-errors] Parenthesized context managers before Python 3.9 #16523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Mar 17, 2025
Merged

[syntax-errors] Parenthesized context managers before Python 3.9 #16523

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
1d2c4fa
[syntax-errors] Parenthesized context managers before Python 3.9
ntBre Mar 5, 2025
359686d
simplify tests, improve error message, catch error
ntBre Mar 5, 2025
557142b
clean up unused snapshot
ntBre Mar 5, 2025
73f1047
move check into parse_with_items
ntBre Mar 6, 2025
f524953
document variant
ntBre Mar 6, 2025
256279b
expand test_ok cases
ntBre Mar 6, 2025
4d74013
replace numbered link
ntBre Mar 6, 2025
ba288ba
fix missing block end
ntBre Mar 6, 2025
20c8b0d
Merge branch 'main' into brent/syn-parenthesized-context-managers
ntBre Mar 6, 2025
b8de4ac
add tests with only some `as` bindings
ntBre Mar 7, 2025
18fb900
always flag parenthesized context managers on Python 3.8
ntBre Mar 14, 2025
a31aa22
Merge branch 'main' into brent/syn-parenthesized-context-managers
ntBre Mar 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions crates/ruff_python_parser/resources/inline/err/parenthesized_context_manager_py38.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# parse_options: {"target-version": "3.8"}
with (foo as x, bar as y): ...
with (foo, bar as y): ...
with (foo as x, bar): ...
13 changes: 13 additions & 0 deletions crates/ruff_python_parser/resources/inline/err/tuple_context_manager_py38.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# parse_options: {"target-version": "3.8"}
# these cases are _syntactically_ valid before Python 3.9 because the `with` item
# is parsed as a tuple, but this will always cause a runtime error, so we flag it
# anyway
with (foo, bar): ...
with (
open('foo.txt')) as foo: ...
with (
foo,
bar,
baz,
): ...
with (foo,): ...
4 changes: 4 additions & 0 deletions crates/ruff_python_parser/resources/inline/ok/parenthesized_context_manager_py39.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# parse_options: {"target-version": "3.9"}
with (foo as x, bar as y): ...
with (foo, bar as y): ...
with (foo as x, bar): ...
6 changes: 6 additions & 0 deletions crates/ruff_python_parser/resources/inline/ok/tuple_context_manager_py38.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# parse_options: {"target-version": "3.8"}
with (
foo,
bar,
baz,
) as tup: ...
46 changes: 46 additions & 0 deletions crates/ruff_python_parser/src/error.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -661,6 +661,46 @@ pub enum UnsupportedSyntaxErrorKind {
TypeAliasStatement,
TypeParamDefault,

/// Represents the use of a parenthesized `with` item before Python 3.9.
///
/// ## Examples
///
/// As described in [BPO 12782], `with` uses like this were not allowed on Python 3.8:
///
/// ```python
/// with (open("a_really_long_foo") as foo,
/// open("a_really_long_bar") as bar):
/// pass
/// ```
///
/// because parentheses were not allowed within the `with` statement itself (see [this comment]
/// in particular). However, parenthesized expressions were still allowed, including the cases
/// below, so the issue can be pretty subtle and relates specifically to parenthesized items
/// with `as` bindings.
///
/// ```python
/// with (foo, bar): ... # okay
/// with (
/// open('foo.txt')) as foo: ... # also okay
/// with (
/// foo,
/// bar,
/// baz,
/// ): ... # also okay, just a tuple
/// with (
/// foo,
/// bar,
/// baz,
/// ) as tup: ... # also okay, binding the tuple
/// ```
///
/// This restriction was lifted in 3.9 but formally included in the [release notes] for 3.10.
///
/// [BPO 12782]: https://github.com/python/cpython/issues/56991
/// [this comment]: https://github.com/python/cpython/issues/56991#issuecomment-1093555141
/// [release notes]: https://docs.python.org/3/whatsnew/3.10.html#summary-release-highlights
ParenthesizedContextManager,

/// Represents the use of a [PEP 646] star expression in an index.
///
/// ## Examples
Expand Down Expand Up @@ -777,6 +817,9 @@ impl Display for UnsupportedSyntaxError {
UnsupportedSyntaxErrorKind::TypeParamDefault => {
"Cannot set default type for a type parameter"
}
UnsupportedSyntaxErrorKind::ParenthesizedContextManager => {
"Cannot use parentheses within a `with` statement"
}
UnsupportedSyntaxErrorKind::StarExpressionInIndex => {
"Cannot use star expression in index"
}
Expand Down Expand Up @@ -832,6 +875,9 @@ impl UnsupportedSyntaxErrorKind {
UnsupportedSyntaxErrorKind::TypeParameterList => Change::Added(PythonVersion::PY312),
UnsupportedSyntaxErrorKind::TypeAliasStatement => Change::Added(PythonVersion::PY312),
UnsupportedSyntaxErrorKind::TypeParamDefault => Change::Added(PythonVersion::PY313),
UnsupportedSyntaxErrorKind::ParenthesizedContextManager => {
Change::Added(PythonVersion::PY39)
}
UnsupportedSyntaxErrorKind::StarExpressionInIndex => {
Change::Added(PythonVersion::PY311)
}
Expand Down
41 changes: 41 additions & 0 deletions crates/ruff_python_parser/src/parser/statement.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2066,8 +2066,49 @@ impl<'src> Parser<'src> {
return vec![];
}

let open_paren_range = self.current_token_range();

if self.at(TokenKind::Lpar) {
if let Some(items) = self.try_parse_parenthesized_with_items() {
// test_ok tuple_context_manager_py38
// # parse_options: {"target-version": "3.8"}
// with (
// foo,
// bar,
// baz,
// ) as tup: ...

// test_err tuple_context_manager_py38
// # parse_options: {"target-version": "3.8"}
// # these cases are _syntactically_ valid before Python 3.9 because the `with` item
// # is parsed as a tuple, but this will always cause a runtime error, so we flag it
// # anyway
// with (foo, bar): ...
// with (
// open('foo.txt')) as foo: ...
// with (
// foo,
// bar,
// baz,
// ): ...
// with (foo,): ...

// test_ok parenthesized_context_manager_py39
// # parse_options: {"target-version": "3.9"}
// with (foo as x, bar as y): ...
// with (foo, bar as y): ...
// with (foo as x, bar): ...

// test_err parenthesized_context_manager_py38
// # parse_options: {"target-version": "3.8"}
// with (foo as x, bar as y): ...
// with (foo, bar as y): ...
// with (foo as x, bar): ...
self.add_unsupported_syntax_error(
UnsupportedSyntaxErrorKind::ParenthesizedContextManager,
open_paren_range,
);

self.expect(TokenKind::Rpar);
items
} else {
Expand Down
199 changes: 199 additions & 0 deletions ...f_python_parser/tests/snapshots/invalid_syntax@parenthesized_context_manager_py38.py.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
---
source: crates/ruff_python_parser/tests/fixtures.rs
input_file: crates/ruff_python_parser/resources/inline/err/parenthesized_context_manager_py38.py
---
## AST

```
Module(
ModModule {
range: 0..126,
body: [
With(
StmtWith {
range: 43..73,
is_async: false,
items: [
WithItem {
range: 49..57,
context_expr: Name(
ExprName {
range: 49..52,
id: Name("foo"),
ctx: Load,
},
),
optional_vars: Some(
Name(
ExprName {
range: 56..57,
id: Name("x"),
ctx: Store,
},
),
),
},
WithItem {
range: 59..67,
context_expr: Name(
ExprName {
range: 59..62,
id: Name("bar"),
ctx: Load,
},
),
optional_vars: Some(
Name(
ExprName {
range: 66..67,
id: Name("y"),
ctx: Store,
},
),
),
},
],
body: [
Expr(
StmtExpr {
range: 70..73,
value: EllipsisLiteral(
ExprEllipsisLiteral {
range: 70..73,
},
),
},
),
],
},
),
With(
StmtWith {
range: 74..99,
is_async: false,
items: [
WithItem {
range: 80..83,
context_expr: Name(
ExprName {
range: 80..83,
id: Name("foo"),
ctx: Load,
},
),
optional_vars: None,
},
WithItem {
range: 85..93,
context_expr: Name(
ExprName {
range: 85..88,
id: Name("bar"),
ctx: Load,
},
),
optional_vars: Some(
Name(
ExprName {
range: 92..93,
id: Name("y"),
ctx: Store,
},
),
),
},
],
body: [
Expr(
StmtExpr {
range: 96..99,
value: EllipsisLiteral(
ExprEllipsisLiteral {
range: 96..99,
},
),
},
),
],
},
),
With(
StmtWith {
range: 100..125,
is_async: false,
items: [
WithItem {
range: 106..114,
context_expr: Name(
ExprName {
range: 106..109,
id: Name("foo"),
ctx: Load,
},
),
optional_vars: Some(
Name(
ExprName {
range: 113..114,
id: Name("x"),
ctx: Store,
},
),
),
},
WithItem {
range: 116..119,
context_expr: Name(
ExprName {
range: 116..119,
id: Name("bar"),
ctx: Load,
},
),
optional_vars: None,
},
],
body: [
Expr(
StmtExpr {
range: 122..125,
value: EllipsisLiteral(
ExprEllipsisLiteral {
range: 122..125,
},
),
},
),
],
},
),
],
},
)
```
## Unsupported Syntax Errors

|
1 | # parse_options: {"target-version": "3.8"}
2 | with (foo as x, bar as y): ...
| ^ Syntax Error: Cannot use parentheses within a `with` statement on Python 3.8 (syntax was added in Python 3.9)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BurntSushi's new diagnostic system will allow us to mark both parentheses.

3 | with (foo, bar as y): ...
4 | with (foo as x, bar): ...
|


|
1 | # parse_options: {"target-version": "3.8"}
2 | with (foo as x, bar as y): ...
3 | with (foo, bar as y): ...
| ^ Syntax Error: Cannot use parentheses within a `with` statement on Python 3.8 (syntax was added in Python 3.9)
4 | with (foo as x, bar): ...
|


|
2 | with (foo as x, bar as y): ...
3 | with (foo, bar as y): ...
4 | with (foo as x, bar): ...
| ^ Syntax Error: Cannot use parentheses within a `with` statement on Python 3.8 (syntax was added in Python 3.9)
|
Loading
Loading