hatchling reads .gitignore files outside the project tree, possibly generating broken wheels

[mgorny]

Originally reported as https://bugs.gentoo.org/918677

Hatchling aggressively scans parent directories for .gitignore files, going as far as to read files that are outside the project tree (i.e. above pyproject.toml), even if they are not inside any git repository (which is against the principle of least surprise). If one of these files include the project directory, then hatchling creates an empty wheel.

To reproduce:

mkdir -p test/sub/sub
echo sub > test/.gitignore
cat > test/sub/pyproject.toml <<-EOF
    [build-system]
    requires = ["hatchling"]
    build-backend = "hatchling.build"

    [project]
    name = "sub"
    version = "0"
    description = "test"
EOF
echo > test/sub/sub/__init__.py
cd test/sub
python -m build -w

The result is:

$ unzip -l dist/sub-0-py2.py3-none-any.whl 
Archive:  dist/sub-0-py2.py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
       57  02-02-2020 00:00   sub-0.dist-info/METADATA
      105  02-02-2020 00:00   sub-0.dist-info/WHEEL
      181  02-02-2020 00:00   sub-0.dist-info/RECORD
---------                     -------
      343                     3 files

I think there are two issues here:

1. .gitignore are being used even if they're not inside a git repository. In this case, the user assumed that renaming .git should be sufficient not to have packages affected by .git* files.
2. .gitignore are being used even when they explicitly ignore the whole project tree. This naturally implies (at least to git) that the project tree is not a part of the git repository, so hatchling shouldn't be using any rules from that repository.

[mgorny]

Oh, I think it also reads .gitignore from parent directories outside the current repository. It really shouldn't do that.

[ofek]

Hello again! Yes I plan to improve this situation next year.

[mgorny]

Thanks! Would it make sense, in the meantime, to throw an error if all package files end up being ignored? This could at least make it easier to figure out what's wrong.

[ofek]

Do you mean if there was no file that matched for inclusion in the artifact like a wheel?

[mgorny]

Yes.

[philippgl]

I ran into this problem with hatch shell and hatch run (and with rye as well).

It actually is a hatchling problem and affects the pip install -e part during the env creation.

The workaround is to just put a .gitignore besides the pyproject.toml in the same directory.

I also have my home directory set up as git repo and have a .gitignore with * inside, so this basically affects all my newly created venvs until I place a .gitignore there.

(Stopping at the pyproject.toml might break mono repos, but on the other hand looking outside the project is extremely surprising. Maybe have some non-default switch in a hatchling section of the pyproject.toml, that stops at the .git directory instead.)

[CroMarmot]

#903 reported the same issue in 2023-jul-04 , and is about 5 months ahead of this issue and https://bugs.gentoo.org/918677

[jameshilliard]

2. .gitignore are being used even when they explicitly ignore the whole project tree. This naturally implies (at least to git) that the project tree is not a part of the git repository, so hatchling shouldn't be using any rules from that repository.

I think #1643 should fix this.