install_requires question, might be a bug?

[damies13]

Hi, I have a question,

Background:
I have a project that needs matplotlib as a dependancy, I found that for some reason if the pip version installed was less than pip 21 (e.g. pip 19.x) then matplotlib would fail to install it's dependancies, so to solve the issue I included pip version >= 21 before matplotlib and this resolved that issue.

My Question:
Just recently I've encountered machines with pip 22 and my package was failing to install because pip was detecting 22 as not being greater or equal to 21, I'm wonder if this is expected behaviour? how should I configure the install_requires so that any version 21 or higher including 22, 23, 25, 30 etc is recognised as meeting the requirements.

I found a workaround that works for now, so it's not an urgent issue for me.

What I was using prior to pip 22:

	# I needed a recent version of pip (pip 21.0.1 worked my previous <20 version didn't) for matplotlib
	# 	to actually install without error
	install_requires=['pip>=21', 'matplotlib'],

My workaround for pip 22:

	# I needed a recent version of pip (pip 21.0.1 worked my previous <20 version didn't) for matplotlib
	# 	to actually install without error
	install_requires=['pip>=21,>=22', 'matplotlib'],

Obviously I don't want to have to do something like: 'pip>=21,>=22,>=23,>=24,>=25,>=26,>=27,>=28,>=29,>=30,>=31'

[abravalheri]

Hi @damies13 that is a very curious behaviour... Both for matplotlib and pip.

To understand what is going on, on the setuptools side, could you provide us some extra information?
Specifically, is there any chance you can follow the steps bellow?

1. After building your package with python -m build,
   in a POSIX system you should be able to see the contents of the generated wheel with:

   unzip -l dist/*.whl

2. In the output of the previous line, you should be able to identify a file inside the zip named
   with the following structure <$YOUR_PACKAGE_NAME>-<$VERSION>.dist-info/METADATA

3. Then you should be able to inspect the contents of this file with

   unzip -c <$WHEEL_FILE_PATH> <$YOUR_PACKAGE_NAME>-<$VERSION>.dist-info/METADATA

4. In the output of the previous command, you should be able to identify a line starting with

   Requires-Dist: pip

5. Could you please show us the contents of this line? (for both install_requires=['pip>=21'... and install_requires=['pip>=21,>=22'...)

(If you are using Windows you can instead try to inspect the generated wheel file with a Windows program that is able to open zip files, e.g. 7zip).

While we check if the problem is somehow related to setuptools, I also recommend you checking if newer versions of matplotlib are really not supposed to work with pip<21... Maybe the is a problem there?
Some ways of investigating that is to see if there are any notes in their CHANGELOG or any open issue in their bug tracker. You can also create an empty virtual environment, force-install an old version of pip and try a matplotlib installation to see if it works...

I am suggesting that, because I think adding pip as a dependency of your package is not a good workaround.

Also, are you trying to install your package in a brand new virtual environment (no other packages installed)? If not I highly recommend you to try that and see if the problem persists...

[damies13]

Hi Anderson,

Actually you can download the whl and gz files and pull them apart if you like as they are on test.pypi.org

This version (and earlier) was built with install_requires=['pip>=21'
https://test.pypi.org/project/rfswarm-manager/0.8.2.1644473236/#files

This version (and newer) was built with install_requires=['pip>=21,>=22'
https://test.pypi.org/project/rfswarm-manager/0.8.2.1644543876/#files

The source setup.py for this package is here:
https://github.com/damies13/rfswarm/blob/v1.0.0/setup-manager.py

The only difference between these 2 packages was the install_requires=['pip>=21,>=22' change.

If you want to reproduce the symptom, this is a reliable way to do so:

1. Clean install of Ubuntu 21.10 desktop (minimal install is fine, don't need all the apps)
2. open terminal
3. sudo apt install python3-pip (installs pip 20.3.4)
4. sudo pip install -U pip (upgrade to pip 22.0.3)
5. If we try to install now from test.pypi.org we'll get an error no matplotlib found because it's not on test.pypi.org
6. sudo pip install -U matplotlib
7. pip install -i https://test.pypi.org/simple/ rfswarm-manager==0.8.2.1644473236

Here's where it gets interesting, if I do the same with sudo, it'll actually install, but this is a client application so there should be no requirement for installing as admin, installing under a user profile should be fine.
7. sudo pip install -i https://test.pypi.org/simple/ rfswarm-manager==0.8.2.1644473236

Also with the change to install_requires=['pip>=21,>=22', pip install -i https://test.pypi.org/simple/ rfswarm-manager==0.8.2.1644543876 will install without sudo.

Also I got the same error on Windows, where there is no sudo, I suppose a user could run a windows command prompt as admin if they have the permissions, but I can't be sure all users will have that and the change to install_requires=['pip>=21,>=22' allows installing on windows without the elevated permissions.

(if you want to run this application you'll also need tkinter installed, the ubuntu install doesn't have that, and it's not relevant to this issue)

Hope that additional information is helpful,

Dave.

[abravalheri]

Hi @damies13, I don't know if you accidentally send the wrong links, but the 2 versions look very similar to me.

I did:

$ wget https://test-files.pythonhosted.org/packages/93/74/8aca88d8efd6e6254010dc3f0b672e5be5140b493c6eaab57a3e0b2a09f3/rfswarm_manager-0.8.2.1644473236-py3-none-any.whl
$ wget https://test-files.pythonhosted.org/packages/1a/49/0f686777bc704ba6aadd9054c6dc7dc55964e932c6d007adeb636f4545e6/rfswarm_manager-0.8.2.1644543876-py3-none-any.whl

$ unzip rfswarm_manager-0.8.2.1644473236-py3-none-any.whl -d rfswarm1
$ unzip rfswarm_manager-0.8.2.1644543876-py3-none-any.whl -d rfswarm2

$ diff -qr rfswarm1 rfswarm2
Files rfswarm1/rfswarm_manager/rfswarm.py and rfswarm2/rfswarm_manager/rfswarm.py differ
Only in rfswarm1: rfswarm_manager-0.8.2.1644473236.dist-info
Only in rfswarm2: rfswarm_manager-0.8.2.1644543876.dist-info

$ diff rfswarm1/rfswarm_manager/rfswarm.py rfswarm2/rfswarm_manager/rfswarm.py
5c5
< #    Version 0.8.2.1644473236
---
> #    Version 0.8.2.1644543876
459c459
<       version="0.8.2.1644473236"
---
>       version="0.8.2.1644543876"

$ diff -qr rfswarm1/rfswarm_manager-0.8.2.1644473236.dist-info rfswarm2/rfswarm_manager-0.8.2.1644543876.dist-info
Files rfswarm1/rfswarm_manager-0.8.2.1644473236.dist-info/METADATA and rfswarm2/rfswarm_manager-0.8.2.1644543876.dist-info/METADATA differ
Files rfswarm1/rfswarm_manager-0.8.2.1644473236.dist-info/RECORD and rfswarm2/rfswarm_manager-0.8.2.1644543876.dist-info/RECORD differ

$ diff rfswarm1/rfswarm_manager-0.8.2.1644473236.dist-info/METADATA rfswarm2/rfswarm_manager-0.8.2.1644543876.dist-info/METADATA
3c3
< Version: 0.8.2.1644473236
---
> Version: 0.8.2.1644543876

# Record files are irrelevant here and are mostly used to uninstall things...

As you can see the distribution archives are mostly identical, the only thing varying is the version number. Maybe something is cached?

Anyway, considering that your commands match these exact versions, I am assuming that you are indeed installing exactly these most identical files that only differ in the version string and achieving different outcomes... which is very curious.

Can it be the case the installation of 0.8.2.1644543876 only succeeds because you attempted to install 0.8.2.1644473236 first?

[abravalheri]

Please also note that everything seems to install fine in a virtual environment:

$ virtualenv rfswarm-venv
...
added seed packages: pip==21.3.1, setuptools==60.5.0, wheel==0.37.1
...

$ rfswarm-venv/bin/python -m pip install -U pip
...
Successfully installed pip-22.0.3

$ rfswarm-venv/bin/pip install rfswarm_manager-0.8.2.1644473236-py3-none-any.whl
...
Successfully installed HTTPServer-1.1.0 configparser-5.2.0 cycler-0.11.0 docopt-0.6.2 fonttools-4.29.1 kiwisolver-1.3.2 matplotlib-3.5.1 numpy-1.22.2 packaging-21.3 pillow-9.0.1 psutil-5.9.0 pyparsing-3.0.7 python-dateutil-2.8.2 rfswarm-manager-0.8.2.1644473236 six-1.16.0

[abravalheri]

Given that you seem to be installing two mostly similar version of the same package (only differing the version string), and that the installation proceeds with no problem when performed in a virtual environment, I think this problem is very unlikely to be related to setuptools.

The fact that the package installs fine in a virtual environment suggests to me that the generated METADATA is valid.

Please note that setuptools only generates the METADATA file and does not install or lookup for the dependencies itself. pip is the tool that contains a dependency solver and all the code for consuming the PyPI API.

[abravalheri]

@damies13 please let me know if you can somehow link the problem to an incorrect METADATA file being generated.

[abravalheri]

@damies13 also note that the install_requires field only ensures the given dependencies are present in runtime. It should not affect (or at least its side-effect should not be counted upon) build-time or installation-time.

[damies13]

I guess what I was really looking for was confirmation if my original setting install_requires=['pip>=21' was the correct way to use this? and expected to work?

if it's just a bug in pip, I can live with that, short term if I can go back to my original setting long term.

Also from your comment is there another field I should be using? I'm fairly new as setup_tools and still learning.

also note that the install_requires field only ensures the given dependencies are present in runtime. It should not affect (or at least its side-effect should not be counted upon) build-time or installation-time.