black skips subdirectories due to incorrect interpretation of .gitignore rules

[A-CGray]

Describe the bug

I have a repo in which I use the following rule to ignore all files without extensions (e.g all compiled executables)

# To ignore compiled binaries that have no extensions we need to first ignore everything, 
# then unignore files with extensions and directories
*
!*.*
!*/

When I run black in the root of this repo, none of the python files in any of the subdirectories are checked because black thinks those subdirectories are ignored:

$ black -v .     

Identified `/home/ali/repos/tacs` as project root containing a .git directory.
Sources to be formatted: "."
.
.
.
/home/ali/repos/tacs/tests ignored: matches a .gitignore file content
/home/ali/repos/tacs/tacs ignored: matches a .gitignore file content
.
.
.
All done! ✨ 🍰 ✨
1 file left unchanged.

Presumably this is happening because black is checking whether /home/ali/repos/tacs/tests would be ignored, which it would, while the correct path for that directory is actually /home/ali/repos/tacs/tests/, which would not be ignored.

To Reproduce

clone the repo:

git clone git@github.com:smdogroup/tacs.git

Run black in verbose mode in the root directory

cd tacs
black -v .

The resulting error is:

/home/ali/repos/tacs/reports ignored: matches a .gitignore file content
/home/ali/repos/tacs/examples ignored: matches a .gitignore file content
/home/ali/repos/tacs/testflo_report.out ignored: matches a .gitignore file content
/home/ali/repos/tacs/Makefile ignored: matches a .gitignore file content
/home/ali/repos/tacs/Makefile.in ignored: matches a .gitignore file content
/home/ali/repos/tacs/build ignored: matches a .gitignore file content
/home/ali/repos/tacs/.git ignored: matches the --exclude regular expression
/home/ali/repos/tacs/.vscode ignored: matches a .gitignore file content
/home/ali/repos/tacs/src ignored: matches a .gitignore file content
/home/ali/repos/tacs/docs ignored: matches a .gitignore file content
/home/ali/repos/tacs/extern ignored: matches a .gitignore file content
/home/ali/repos/tacs/tests ignored: matches a .gitignore file content
/home/ali/repos/tacs/tacs.egg-info ignored: matches a .gitignore file content
/home/ali/repos/tacs/conda ignored: matches a .gitignore file content
/home/ali/repos/tacs/tacs ignored: matches a .gitignore file content
/home/ali/repos/tacs/roadmaps ignored: matches a .gitignore file content
/home/ali/repos/tacs/lib ignored: matches a .gitignore file content
/home/ali/repos/tacs/.pre-commit-config.yaml ignored: matches a .gitignore file content
/home/ali/repos/tacs/setup.py wasn't modified on disk since last run.

All done! ✨ 🍰 ✨
1 file left unchanged.

Expected behavior

black should run on all python files in the repo:

All done! ✨ 🍰 ✨
214 files left unchanged.

Environment

• Black's version: 23.3.0
• OS and Python version: Linux/Python 3.10.9

[A-CGray]

Looks like this is the same issue as #3694

[A-CGray]

@JelleZijlstra I'd be happy to try fixing this myself if you can give me some pointers on which section(s) of the code need to be changed

[ortk95]

Hi @A-CGray - from when I had a very quick explore of the code for #3694, I think gen_python_files and normalise_path_maybe_ignore are probably responsible for this bug

black/src/black/files.py

Lines 309 to 394 in 3dcacdd

	def gen_python_files(
	paths: Iterable[Path],
	root: Path,
	include: Pattern[str],
	exclude: Pattern[str],
	extend_exclude: Optional[Pattern[str]],
	force_exclude: Optional[Pattern[str]],
	report: Report,
	gitignore_dict: Optional[Dict[Path, PathSpec]],
	*,
	verbose: bool,
	quiet: bool,
	) -> Iterator[Path]:
	"""Generate all files under `path` whose paths are not excluded by the
	`exclude_regex`, `extend_exclude`, or `force_exclude` regexes,
	but are included by the `include` regex.
	Symbolic links pointing outside of the `root` directory are ignored.
	`report` is where output about exclusions goes.
	"""
	assert root.is_absolute(), f"INTERNAL ERROR: `root` must be absolute but is {root}"
	for child in paths:
	root_relative_path = child.absolute().relative_to(root).as_posix()
	# First ignore files matching .gitignore, if passed
	if gitignore_dict and _path_is_ignored(
	root_relative_path, root, gitignore_dict, report
	):
	continue
	# Then ignore with `--exclude` `--extend-exclude` and `--force-exclude` options.
	root_relative_path = "/" + root_relative_path
	if child.is_dir():
	root_relative_path += "/"
	if path_is_excluded(root_relative_path, exclude):
	report.path_ignored(child, "matches the --exclude regular expression")
	continue
	if path_is_excluded(root_relative_path, extend_exclude):
	report.path_ignored(
	child, "matches the --extend-exclude regular expression"
	)
	continue
	if path_is_excluded(root_relative_path, force_exclude):
	report.path_ignored(child, "matches the --force-exclude regular expression")
	continue
	normalized_path = normalize_path_maybe_ignore(child, root, report)
	if normalized_path is None:
	continue
	if child.is_dir():
	# If gitignore is None, gitignore usage is disabled, while a Falsey
	# gitignore is when the directory doesn't have a .gitignore file.
	if gitignore_dict is not None:
	new_gitignore_dict = {
	**gitignore_dict,
	root / child: get_gitignore(child),
	}
	else:
	new_gitignore_dict = None
	yield from gen_python_files(
	child.iterdir(),
	root,
	include,
	exclude,
	extend_exclude,
	force_exclude,
	report,
	new_gitignore_dict,
	verbose=verbose,
	quiet=quiet,
	)
	elif child.is_file():
	if child.suffix == ".ipynb" and not jupyter_dependencies_are_installed(
	warn=verbose or not quiet
	):
	continue
	include_match = include.search(normalized_path) if include else True
	if include_match:
	yield child

black/src/black/files.py

Lines 250 to 276 in 3dcacdd

	def normalize_path_maybe_ignore(
	path: Path,
	root: Path,
	report: Optional[Report] = None,
	) -> Optional[str]:
	"""Normalize `path`. May return `None` if `path` was ignored.
	`report` is where "path ignored" output goes.
	"""
	try:
	abspath = path if path.is_absolute() else Path.cwd() / path
	normalized_path = abspath.resolve()
	try:
	root_relative_path = normalized_path.relative_to(root).as_posix()
	except ValueError:
	if report:
	report.path_ignored(
	path, f"is a symbolic link that points outside {root}"
	)
	return None
	except OSError as e:
	if report:
	report.path_ignored(path, f"cannot be read because {e}")
	return None
	return root_relative_path

[A-CGray]

This seems to have been fixed in v24.8.0 by #4415