Parse triple quoted string annotations as if parenthesized

[Glyphack]

Summary

Resolves #9467

Parse quoted annotations as if the string content is inside parenthesis.
With this logic x and y in this example are equal:

y: """
   int |
   str
"""

z: """(
    int |
    str
)
"""

Also this rule only applies to triple quotes(link).

This PR is based on the comments on the issue.

I did one extra change, since we don't want any indentation tokens I am setting the State::Other as the initial state of the Lexer.

Remaining work:

• Add a test case for red-knot.
• Add more tests.

Test Plan

Added a test which previously failed because quoted annotation contained indentation.
Added an mdtest for red-knot.
Updated previous test.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

ℹ️ ecosystem check encountered format errors. (no format changes; 1 project error)

openai/openai-cookbook (error)

warning: Detected debug build without --no-cache.
error: Failed to read examples/Assistants_API_overview_python.ipynb: Expected a Jupyter Notebook, which must be internally stored as JSON, but this file isn't valid JSON: expected `,` or `]` at line 197 column 8

Formatter (preview)

ℹ️ ecosystem check encountered format errors. (no format changes; 1 project error)

openai/openai-cookbook (error)

ruff format --preview

warning: Detected debug build without --no-cache.
error: Failed to read examples/Assistants_API_overview_python.ipynb: Expected a Jupyter Notebook, which must be internally stored as JSON, but this file isn't valid JSON: expected `,` or `]` at line 197 column 8

[carljm]

Thank you!! The red-knot test and changes look fine to me, but that's a small part of this PR. Would rather have @dhruvmanila take a look at the parser/ruff side.

[MichaReiser]

Thank you. Would you mind adding a test where an expression is incorrectly parenthesized?

def f(
	a: """
		int | 
str)
"""
)

or

def f(
	a: """
		int) | 
str
"""
)

I'm curious how it recovers in those cases and if we need to do something more than just setting nesting = 1

[MichaReiser]

I'd move it next to Expression and let's add a comment explaining what this mode is.

[Glyphack]

@MichaReiser I added the test cases you suggested and the tests pass(see this commit)
But I think that is because the parser is recovering.

So I fixed the nesting check to check for nesting value of 1 when mode is ParenthesizedExpression. This way the lexer will detect the problem that nesting is one level lower than started as well.

[dhruvmanila]

What do you think about parse_string_annotation or parse_string_annotation_content? Additionally, we should provide documentation similar to what you've done in parse_parenthesized_expression_range as this is a public function and the crate is being used by multiple projects.

[Glyphack]

I agree, renamed to parse_string_annotation and added documentation.

[dhruvmanila]

Can we add the same test case using triple-quoted string which will be useful as a documentation?

# but, using the same annotation inside a triple-quoted string is valid
valid: """\n int"""

[Glyphack]

Actually I made a mistake here, invalid: "\n int" is in a python file. So this string will not actually have a newline character but \ and n as characters. So also """"\n int""" fails.
Is there a way to actually create a type annotation that contains newline inside a python file without using triple quotes?

https://pyright-play.net/?code=JYOwbghgNsAmBcACARKgOiRoAurkCh9RIYEVUiRcKg

[Glyphack]

I removed this right now. Since I could not figure out how to do it correctly.

[dhruvmanila]

Can we add some test cases for unclosed parenthesis similar to how these? It would also be useful to provide test cases with multiple parentheses.

[MichaReiser]

Thanks. This looks great.