Skip to content

Conversation

sharkdp
Copy link
Contributor

@sharkdp sharkdp commented Jun 17, 2025

Summary

Adds a new ecosystem-analyzer workflow with a similar purpose to the mypy-primer workflow. It creates a richer ecosystem diff report using ecosystem-analyzer (example report). This is still experimental and also quite a bit slower than mypy_primer, so I chose to make this opt-in for now via a ecosystem-analyzer label. This would give us a way to play with this while still evaluating if we should further invest in this or not.

Advantages over the mypy_primer diff output:

  • Interactive filtering of diagnostics
  • Statistics overview which breaks down added/removed/changed diagnostics across lint rules
  • Has the concept of "changed" diagnostics, which makes it easier to review changes where diagnostic messages have changed (along with other changes).
  • Compute diff based on old and new project-lists (good.txt). This allows us to diff changes to the project list itself. This has caused confusion in the past where we tried to add new projects to good.txt, but then ran the main-branch version of ty on that new list (where the bug was not yet fixed)

Disadvantages:

  • The report currently needs to be downloaded from the workflow run, as I don't know if we have a way of deploying HTML files like this temporarily to some hosted infrastructure.

@sharkdp sharkdp added ci Related to internal CI tooling ty Multi-file analysis & type inference labels Jun 17, 2025
@sharkdp sharkdp force-pushed the david/ecosystem-analyzer branch 5 times, most recently from 2ac9b7e to eacb704 Compare June 17, 2025 15:29
@sharkdp sharkdp force-pushed the david/ecosystem-analyzer branch from eacb704 to 9b4cd51 Compare June 17, 2025 19:19
@sharkdp sharkdp marked this pull request as ready for review June 20, 2025 09:17
@AlexWaygood
Copy link
Member

This would give us a way to play with this while still evaluating if we should further invest in this or not.

I honestly think it's "useful enough" just for the top-level summary it gives us, which is a fantastic improvement for tackling huge primer reports that sprawl for many, many lines!

Disadvantages:

* The report currently needs to be downloaded from the workflow run, as I don't know if we have a way of deploying HTML files like this temporarily to some hosted infrastructure.

Could we have a bot post a comment with a link that you could click on to download the diff.html file...? It's currently quite a few clicks to get to the link to download the HTML file, and you don't get a notification when the workflow has successfully completed either

@sharkdp
Copy link
Contributor Author

sharkdp commented Jun 20, 2025

Could we have a bot post a comment with a link that you could click on to download the diff.html file...? It's currently quite a few clicks to get to the link to download the HTML file, and you don't get a notification when the workflow has successfully completed either

I thought about it, but it's still a horrible experience because it's a zip file. So if we want to move forward with this, I'd really love to have a way to host these reports for a limited period of time (e.g. two months).

@sharkdp
Copy link
Contributor Author

sharkdp commented Jun 20, 2025

Merging this for now, so we can play with it.

@sharkdp sharkdp merged commit 0ef324b into main Jun 20, 2025
36 checks passed
@sharkdp sharkdp deleted the david/ecosystem-analyzer branch June 20, 2025 13:23
dcreager added a commit that referenced this pull request Jun 20, 2025
* main: (21 commits)
  [`flake8-logging`] Avoid false positive for `exc_info=True` outside `logger.exception` (`LOG014`) (#18737)
  [`flake8-pie`] Small docs fix to `PIE794` (#18829)
  [`pylint`] Ignore __init__.py files in (PLC0414) (#18400)
  Avoid generating diagnostics with per-file ignores (#18801)
  [`flake8-simplify`] Fix false negatives for shadowed bindings  (`SIM910`, `SIM911`) (#18794)
  [ty] Fix panics when pulling types for `ClassVar` or `Final` parameterized with >1 argument (#18824)
  [`pylint`] add fix safety section (`PLR1714`) (#18415)
  [Perflint] Small docs improvement to `PERF401` (#18786)
  [`pylint`] Avoid flattening nested `min`/`max` when outer call has single argument (`PLW3301`) (#16885)
  [`ruff`] Added `cls.__dict__.get('__annotations__')` check (`RUF063`) (#18233)
  [ty] Use `HashTable` in `PlaceTable` (#18819)
  docs: Correct collections-named-tuple example to use PascalCase assignment (#16884)
  [ty] ecosystem-analyzer workflow (#18719)
  [ty] Add support for `@staticmethod`s (#18809)
  unnecessary_dict_kwargs doc - a note on type checking benefits (#18666)
  [`flake8-pytest-style`] Mark autofix for `PT001` and `PT023` as unsafe if there's comments in the decorator (#18792)
  [ty] Surface matched overload diagnostic directly (#18452)
  [ty] Report when a dataclass contains more than one `KW_ONLY` field (#18731)
  [`flake8-pie`] Add fix safety section to `PIE794` (#18802)
  [`pycodestyle`] Add fix safety section to `W291` and `W293`  (#18800)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Related to internal CI tooling ecosystem-analyzer ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants