Automatically exclude `.gitignore` content by adopting OmniParser

[aheritier]

What problem are you trying to solve?

Our project use Git as SCM (classical) and has a .gitignore file to exclude some files and folders which shouldn't be store in the SCM (often large folders like node_modules etc). OpenRewrite proposes a manual option to exclude some folders from the update. By default, or at least optionally it should allow to follow the .gitignore content because it is 99.99999% sure that you never want to update them (whatever are the update rules in the recipes).

Describe the solution you'd like

Either by default and in any case with an option to turn on or off the feature to respect the .gitignore content

Have you considered any alternatives or workarounds?

For now I manually add exclusion entries to avoid to scan the whole project.

[timtebeek]

hi @aheritier ; thanks for the suggestion! I would have sworn we had something like that already, but I can't find it right now so perhaps I'm mistaken, or confused with the Moderne CLI.

There's perhaps two parts to this: one is whether we want to parse files in .gitignore, and another if we want to modify files in .gitignore. While I agree with you that it's often the case that we could skip any git ignored files, there might also be cases where folks for instance have a code generator that creates files in a git ignored path. Those files might then have to be parsed to get complete type information, or even modified if they want recipes applied before they are packaged up. As such it's hard to say if we can blanket skip any git ignored files from both parsing and modifications.

For some common cases like node_modules we already have those ignored by default. Are there any other common patterns that you think should be excluded perhaps?

Welcome to help think this through; either way it sounds like this should be optional at least, as to not rule out the above cases.

[aheritier]

Hi tim,

Good point. We should have different rules for parsing vs modifying (at least we could configure them differently).

In fact I realized this "issue" when I saw warnings in "Parsing source files" step about large files which are all in folders I excluded in .gitignore like .angular, .pnpm-store, ...

[INFO] Project [CFP] Parsing source files
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/9bb8b7fe7df79eb7b019c2fdabea22f6156d0275/9.pack as its size 348 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/9bb8b7fe7df79eb7b019c2fdabea22f6156d0275/8.pack as its size 10 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/781a704791961d6000a5127fd8e1b8b8802e6433/0.pack as its size 224 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/73029b88bc2d11c0627e034bcc21a59adf9a93f4/4.pack as its size 260 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/73029b88bc2d11c0627e034bcc21a59adf9a93f4/0.pack as its size 263 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/73029b88bc2d11c0627e034bcc21a59adf9a93f4/8.pack_ as its size 493 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/73029b88bc2d11c0627e034bcc21a59adf9a93f4/6.pack as its size 260 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/62f2f0a354b2d0e196407d99a5533938ca92f3ee/18.pack as its size 213 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/62f2f0a354b2d0e196407d99a5533938ca92f3ee/12.pack as its size 260 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/62f2f0a354b2d0e196407d99a5533938ca92f3ee/27.pack as its size 22 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/62f2f0a354b2d0e196407d99a5533938ca92f3ee/10.pack as its size 15 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/62f2f0a354b2d0e196407d99a5533938ca92f3ee/6.pack as its size 271 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/b5bb6f680b2d0b3e76ad2e60564713f09656e471/0.pack as its size 349 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/0ef42559b2078554fb336b862f11b82e8251a17e/0.pack as its size 356 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/7c5984a669326519916f3bab96d68dc5a3cb598a/0.pack as its size 221 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/526f3479332bc8c0096ef35eadebe4599685e88e/9.pack as its size 286 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/526f3479332bc8c0096ef35eadebe4599685e88e/0.pack as its size 242 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/e3ef4d935ab76f887ac0f4b27317c85d67fced9d/0.pack as its size 358 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/51e362905afc69a6d76a31d7a05769f978ce6879/8.pack as its size 275 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/51e362905afc69a6d76a31d7a05769f978ce6879/1.pack as its size 12 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/51e362905afc69a6d76a31d7a05769f978ce6879/11.pack as its size 260 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.angular/cache/18.2.12/angular-webpack/51e362905afc69a6d76a31d7a05769f978ce6879/0.pack as its size 215 MB exceeds size threshold 10 MB
[INFO] Not parsing quark /Users/donaldduck/some/path/to/my/project/.pnpm-store/v3/files/b1/f26a06bc1301f1a9a1da99c9cdfe466a454b2f4c4db239723bf6bd22645d9432d0da98f99bd143c027eddfa6bc785ece76878ef3a7fe3190adcd4fcdbd84f9-exec as its size 11 MB exceeds size threshold 10 MB

I don't feel it has a performance impact thus maybe it's ok to keep it like this.

[shanman190]

@timtebeek, here's the link:
https://github.com/openrewrite/rewrite-polyglot/blob/main/src%2Fmain%2Fjava%2Forg%2Fopenrewrite%2Fpolyglot%2FOmniParser.java#L72-L84

To the point though, there are often a lot of other directories that may be ignored beyond the default set.

On the flip side, as Tim knows already, sometimes files that are Git ignored have to be parsed to produce enough information for type attribution. It's definitely a balance.

[timtebeek]

Thanks for the hint @shanman190 ! Indeed the OmniParser has two different ways of walking a file tree, depending on whether it's a git repositry or not; if it's a Git repository then we already call out to jgit to see if the entry is ignored.

From the looks of it (and knowing) @aheritier he'll be using the rewrite Maven plugin, which does not use the OmniParser, and lacks a similar mechanism to skip git ignored files (for now). For the .java/.kt sources that mechanism would need to be added here, whereas for other resources we can add that here, or swap in the OmniParser there for consistency.

With that I think this issue uniquely affects our Maven plugin, so I'll move the issue there. Thanks all!

[aheritier]

It makes sense to me @timtebeek and thanks for your feedbacks @shanman190
I effectively use the maven plugin (not sure why ;-D ) and based on the exchange here I fill that what is the most "annoying" for me is that the plugin doesn't to configure easily what should be excluded from parsing or from updating steps. There is only the exclusions parameter.

[timtebeek]

Yes I think we can do better; for comparison, the Gradle project parser does use the OmniParser, and skips over already parsed files, as well as anything that is Git ignored. From the looks of it, most of what's in the Maven plugin's ResourceParser can move to use the OmniParser to gain that git ignore most easily. Something to look into, if I find the time.

[aheritier]

Thanks Tim. It's not blocker/urgent. I might try to look at it but it won't be before 6 months.

[github-actions]

This issue is stale because it has not had any activity for 60 days. Remove question label or comment or this will be closed in two weeks. Issues may be reopened when there is renewed interest.