[pydoclint] Implement docstring-extraneous-parameter (DOC102)

[augustelalande]

Summary

Implement docstring-extraneous-parameter (DOC102). This rule checks that all parameters present in a functions docstring are also present in its signature.

Split from #13280, per this comment.

Part of #12434.

Test Plan

Test cases added.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

ℹ️ ecosystem check detected linter changes. (+66 -0 violations, +0 -0 fixes in 4 projects; 51 projects unchanged)

DisnakeDev/disnake (+3 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --no-fix --output-format concise --preview

+ disnake/client.py:814:9: DOC102 Documented parameter `Example` is not in the function's signature
+ disnake/ext/commands/base_core.py:855:5: DOC102 Documented parameter `params` is not in the function's signature
+ disnake/ext/commands/base_core.py:895:5: DOC102 Documented parameter `params` is not in the function's signature

apache/superset (+24 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --no-fix --output-format concise --preview --select ALL

+ superset-extensions-cli/tests/utils.py:186:9: DOC102 Documented parameter `name` is not in the function's signature
+ superset/async_events/api.py:57:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/charts/data/api.py:105:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/charts/data/api.py:287:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/dashboards/api.py:1711:11: DOC102 Documented parameter `requestBody` is not in the function's signature
+ superset/dashboards/api.py:1717:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/dashboards/api.py:408:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/dashboards/api.py:459:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/dashboards/api.py:518:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/dashboards/api.py:716:11: DOC102 Documented parameter `requestBody` is not in the function's signature
+ superset/dashboards/api.py:723:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/dashboards/api.py:800:11: DOC102 Documented parameter `requestBody` is not in the function's signature
+ superset/dashboards/api.py:807:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/databases/api.py:1058:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/databases/api.py:1139:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/databases/api.py:1373:11: DOC102 Documented parameter `requestBody` is not in the function's signature
+ superset/databases/api.py:1380:11: DOC102 Documented parameter `responses` is not in the function's signature
+ superset/databases/api.py:989:11: DOC102 Documented parameter `responses` is not in the function's signature
... 6 additional changes omitted for project

bokeh/bokeh (+26 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --no-fix --output-format concise --preview --select ALL

+ src/bokeh/core/has_props.py:427:13: DOC102 Documented parameter `json` is not in the function's signature
+ src/bokeh/core/has_props.py:429:13: DOC102 Documented parameter `models` is not in the function's signature
+ src/bokeh/core/property/dataspec.py:450:13: DOC102 Documented parameter `name` is not in the function's signature
+ src/bokeh/core/property/descriptors.py:402:13: DOC102 Documented parameter `json` is not in the function's signature
+ src/bokeh/core/property/descriptors.py:404:13: DOC102 Documented parameter `models` is not in the function's signature
+ src/bokeh/core/property/descriptors.py:671:13: DOC102 Documented parameter `new` is not in the function's signature
+ src/bokeh/core/property/descriptors.py:790:13: DOC102 Documented parameter `json` is not in the function's signature
+ src/bokeh/core/query.py:69:9: DOC102 Documented parameter `obj` is not in the function's signature
+ src/bokeh/core/validation/decorators.py:60:9: DOC102 Documented parameter `code` is not in the function's signature
+ src/bokeh/document/document.py:353:13: DOC102 Documented parameter `patch` is not in the function's signature
+ src/bokeh/document/events.py:395:13: DOC102 Documented parameter `column_source` is not in the function's signature
+ src/bokeh/document/events.py:487:13: DOC102 Documented parameter `column_source` is not in the function's signature
+ src/bokeh/document/events.py:585:13: DOC102 Documented parameter `column_source` is not in the function's signature
+ src/bokeh/driving.py:149:9: DOC102 Documented parameter `x` is not in the function's signature
+ src/bokeh/embed/bundle.py:367:9: DOC102 Documented parameter `objs` is not in the function's signature
+ src/bokeh/embed/bundle.py:380:9: DOC102 Documented parameter `objs` is not in the function's signature
+ src/bokeh/embed/bundle.py:442:9: DOC102 Documented parameter `objs` is not in the function's signature
+ src/bokeh/plotting/_tools.py:86:9: DOC102 Documented parameter `tools_map` is not in the function's signature
+ src/bokeh/protocol/message.py:210:13: DOC102 Documented parameter `buf_header` is not in the function's signature
... 7 additional changes omitted for project

langchain-ai/langchain (+13 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --no-fix --output-format concise --preview

+ libs/core/langchain_core/callbacks/file.py:136:13: DOC102 Documented parameter `file` is not in the function's signature
+ libs/core/langchain_core/outputs/chat_generation.py:47:13: DOC102 Documented parameter `values` is not in the function's signature
+ libs/core/langchain_core/utils/mustache.py:451:9: DOC102 Documented parameter `partials_path` is not in the function's signature
+ libs/core/langchain_core/utils/mustache.py:454:9: DOC102 Documented parameter `partials_ext` is not in the function's signature
+ libs/core/tests/unit_tests/test_tools.py:1350:13: DOC102 Documented parameter `banana` is not in the function's signature
+ libs/core/tests/unit_tests/test_tools.py:1351:13: DOC102 Documented parameter `monkey` is not in the function's signature
+ libs/langchain/langchain_classic/agents/agent.py:1098:13: DOC102 Documented parameter `values` is not in the function's signature
+ libs/langchain/langchain_classic/agents/agent.py:833:13: DOC102 Documented parameter `values` is not in the function's signature
+ libs/langchain/langchain_classic/agents/openai_functions_agent/base.py:69:13: DOC102 Documented parameter `values` is not in the function's signature
+ libs/langchain/langchain_classic/evaluation/scoring/eval_chain.py:306:13: DOC102 Documented parameter `prediction_b` is not in the function's signature
... 3 additional changes omitted for project

Changes by rule (1 rules affected)

code	total	+ violation	- violation	+ fix	- fix
DOC102	66	66	0	0	0

[ntBre]

Thanks for your work here and for splitting off the PR! This looks great to me overall, just a few minor ideas/suggestions. Did you look into reusing any code from D417? That seemed like the only comment from the other PR that wasn't fully resolved.

I'd also like @MichaReiser to have a chance to look since he reviewed #13280 originally.

[ntBre]

Thank you! You went above and beyond my diagnostic suggestion. I think we could use that for the main diagnostic now. The code looks good to me otherwise.

However, I took a quick peek at the ecosystem results, and this one in particular looks pretty suspicious:

disnake/client.py:805:9: DOC102 These documented parameters are not in the function's signature: Example, --------, .. code-block

I know we have some open issues around sphinx docstrings, but ideally we'd eliminate these false positives at least.

This one also looks interesting, I wonder if we should consider skipping *args as well as **kwargs:

disnake/ext/commands/context.py:310:9: DOC102 Documented parameter entity is not in the function's signature

This one looks like it should have been straightforward, but we have a false positive on cls:

disnake/ext/commands/flag_converter.py:556:9: DOC102 Documented parameter cls is not in the function's signature

We'll need to go through all of the ecosystem results, but those are a few I pulled out from the disnake run.

[augustelalande]

I fixed:

disnake/ext/commands/context.py:310:9: DOC102 Documented parameter entity is not in the function's signature

and

disnake/ext/commands/flag_converter.py:556:9: DOC102 Documented parameter cls is not in the function's signature

however the problem with

disnake/client.py:805:9: DOC102 These documented parameters are not in the function's signature: Example, --------, .. code-block

is that Example is not an official numpy docstring section, where Examples is prefered. As defined here and here

[ntBre]

Thanks! Have you looked through the other ecosystem results? It looks like airflow has some YAML blocks in docstrings that we're trying to parse as parameter names (lots of - diagnostics), and there are a couple of splitting issues in bokeh like this:

src/bokeh/core/has_props.py:435:13: DOC102 Documented parameter setter(ClientSession is not in the function's signature

I wonder if we should be a little more permissive with the numpy example(s) section. I don't really like the look of emitting diagnostics like this:

• disnake/client.py:812:9: DOC102 Documented parameter Example is not in the function's signature
• disnake/client.py:813:9: DOC102 Documented parameter -------- is not in the function's signature
• disnake/client.py:815:9: DOC102 Documented parameter .. code-block is not in the function's signature

It may help in several of these cases to filter out candidates that aren't valid Python identifiers, if we can't avoid checking them in the first place.

[augustelalande]

It may help in several of these cases to filter out candidates that aren't valid Python identifiers, if we can't avoid checking them in the first place.

I have limited the detection of parameters to those beginning with valid characters.

I wonder if we should be a little more permissive with the numpy example(s) section. I don't really like the look of emitting diagnostics like this:
disnake/client.py:812:9: DOC102 Documented parameter Example is not in the function's signature
disnake/client.py:813:9: DOC102 Documented parameter -------- is not in the function's signature
disnake/client.py:815:9: DOC102 Documented parameter .. code-block is not in the function's signature

We can, and it is easy to change, but it seems out of scope for this PR

src/bokeh/core/has_props.py:435:13: DOC102 Documented parameter setter(ClientSession is not in the function's signature

As far as I'm concerned this one is a problem with their formatting, they should have a space between the parameter and the type.

lots of - diagnostics

This is also a problem with their docstrings, they aren't using google or numpy style, so we can't parse them properly.

[ntBre]

Thanks! I've been thinking about how to handle the false positives. I think I agree with you that these are technically issues in the ecosystem docs, but they'll still look like false positives in Ruff, so I'd like to err on the side of filtering them out.

I had a couple of ideas about identifier handling in an inline comment that I think could help with some of them.

I'm somewhat less concerned about the Example case and the inline YAML docstrings, but one idea that came to mind is if we can bail out if we're not in a known Args or Parameters section? That seems like it would filter out at least the YAML cases I looked at, which don't fall under any known section heading as far as I can tell.

[augustelalande]

Thanks! I've been thinking about how to handle the false positives. I think I agree with you that these are technically issues in the ecosystem docs, but they'll still look like false positives in Ruff, so I'd like to err on the side of filtering them out.

I had a couple of ideas about identifier handling in an inline comment that I think could help with some of them.

I'm somewhat less concerned about the Example case and the inline YAML docstrings, but one idea that came to mind is if we can bail out if we're not in a known Args or Parameters section? That seems like it would filter out at least the YAML cases I looked at, which don't fall under any known section heading as far as I can tell.

If by YAML docstring you are referring to something like this https://github.com/apache/superset/blob/4b71adaa9c89925676e7fed6af0c56c5b0ece863/superset/dashboards/api.py#L1711, the issue is that one of the keys is parameters which causes our parser to think it is in a parameters section. We already split the docstring into sections, so we are not processing anything outside of identified sections.

[augustelalande]

@ntBre Any further comments?

[ntBre]

No, I think you've convinced me. I might do one more pass over the code, but I think we should land this.

There are still a few false positives for unusual docstring formats, but the ecosystem check is also showing quite a few true positives. I think it's a helpful rule.

[augustelalande]

@ntBre Sorry to be pushy, but it's been a year since I made the initial pull request, it's been reviewed 3 or 4 times, and every time the conversation just goes silent with no more comments and no merge. I'm fine with making more changes, and fine with it not being merged, but I would really appreciate some transparency about what is holding things up.

Best,

Auguste

[ntBre]

Sorry, just a bit behind on my notifications. This is still in my inbox to do another once-over on the code, but I didn't quite get to it today. I'll make this my first review tomorrow.

[augustelalande]

@ntBre Thanks for the quick reply, and appreciate everything you guys do

[ntBre]

Awesome, thanks for all of your work on this and for your patience with all the reviews!

I pushed one commit using TextSize a bit more heavily in the parameter parsing functions. The +1 to the length wasn't quite right for files with \r\n line endings, but we can use the full_line_end method from the LineRanges trait to help with that. Then it became a bit easier to use TextSize elsewhere and avoid some unwraps to boot.

I also wrote up a follow-up issue for the last two types of false positives in the ecosystem check: #20923. I think we should go ahead and land this, but I wanted to track those somewhere.

I'll make sure my change doesn't regress the ecosystem check, and then I'll hit merge!