Unpin numpy

[hauntsaninja]

See also https://iscinumpy.dev/post/bound-version-constraints/

The workaround here is unnecessary (discussed in #43). Since numpy declares python_requires, pip's resolver will automatically discard newer numpy solutions when installing on Python 3.8.

This also saves you a chore every year when Python makes a new release :-)

[reyammer]

Thanks for the PR! Interesting... do you know if numpy started using python_requires only recently? IIRC I couldn't get anything working with the simpler setup that you are proposing... but indeed, the tests seem to pass on all 3.8 => 3.12 python versions, so it looks good to me...

[reyammer]

OK, I now remember what the problem was. I was just trying to bump numpy version to 2.1.0, which only supports python>=3.10. If I update pyproject.toml with numpy = ">=1.24" as you suggested, and then I try to bump numpy version with poetry update numpy, poetry actually tries to downgrade numpy to: Downgrading numpy (1.26.4 -> 1.24.4), and then crashes. (I'm running python 3.12.)

The only way I found to fix this is to update pyproject.toml with:

numpy = [
    {version = "^1.24", python = ">=3.8,<3.9"},
    {version = "^1.26", python = ">=3.9,<3.10"},
    {version = "^2.1.0", python = ">=3.10,<3.13"},
]

Then, poetry update numpy actually brings numpy to 2.1.0. Here is my PR: #625

This seems more like a "limitation"/requirement from poetry, rather than a requirement from pip?

[gaby]

I'm pretty sure the pin is needed. Not all tools behave the same way.

[reyammer]

We are working to port magika project management from poetry to uv (see #663), and it seems the right way of handling our situation with numpy is still to be explicit with target versions vs. python markers such as python version. This should help uv to know what to do and it does not rely on the python_requires from numpy (I'm not sure when this info would be available to pip/poetry/uv? would the resolve need to first download the latest version of numpy to determine "this specific version is not OK given the current setup"?).

This is how it currently looks like:

magika/python/pyproject.toml

Lines 15 to 17 in 6cf77a5

	"numpy>=1.24; python_version >= '3.8' and python_version < '3.9'",
	"numpy>=1.26; python_version >= '3.9' and python_version < '3.10'",
	"numpy>=2.1.0; python_version >= '3.10'",

The python_version constraints are now specified without the <3.13 upper bound, which means that we shouldn't need to update pyproject.toml for every new version of python.

@hauntsaninja, @gaby, thoughts?

[gaby]

@reyammer That should be fine, the main thing will be keeping track of which versions numpy is dropping. I'm surprised their lowest supported is 3.10 when 3.8/3.9 are still not EOL.

Also 👍 for using uv.

[reyammer]

Closing this for now; @hauntsaninja please feel free to reopen if you have more context, thanks!

[hauntsaninja]

Sorry, just seeing this!

All of pip/uv/poetry will look at Requires-Python metadata when determining the version of numpy to install. They will not let you install a version of numpy that has a Requires-Python incompatible with your Python version. I promise you that this is so! :-)

I'm not sure when this info would be available to pip/poetry/uv

The way all Python resolvers work is approximately like the following:

• User requests magika to be installed
• Tool now needs to figure out what version of magika to install
• Tool will request metadata from PyPI
• You can do this yourself! curl https://pypi.org/simple/magika/ -H 'Accept: application/vnd.pypi.simple.v1+json'
• Tools will then pick a version of magika. This is usually the latest, but tools will filter the available versions based on the requires-python you see in that metadata. In magika's case, installers will probably look at the magika 0.5.1 wheel first
• Tools will then request the metadata for magika to see what magika's dependencies are in the specific 0.5.1 wheel
• You can do this yourself! curl https://files.pythonhosted.org/packages/66/79/e1c167ec35060692b70bfc4f2d0aa9314dd7e37ba8e30c1c27965e2f1daa/magika-0.5.1-py3-none-any.whl.metadata
• This will have a Requires-Dist field that corresponds to what you have in pyproject.toml. Notably, nothing in your poetry.lock is ever exposed to users (will discuss this more later)
• In magika 0.5.1's case, this is:

Requires-Dist: numpy (>=1.24,<2.0) ; python_version >= "3.8" and python_version < "3.9"
Requires-Dist: numpy (>=1.26,<2.0) ; python_version >= "3.9" and python_version < "3.13"

• Tools will then recursively do this for numpy (and backtrack if some incompatible set of requirements arises)

poetry actually tries to downgrade numpy

Poetry has a concept of a lockfile. Nothing in here is ever exposed to your users, it's simply for your own development. Poetry's lockfile will try to find a single set of dependencies that you can use across different platforms and Python versions. If it's crashing, it might be because it's trying to compile numpy 1.24 for Python 3.12 from the sdist and failing, which is where your manual splitting of pins might help. uv also has a similar lockfile concept, but maybe its implementation is smart enough to avoid sdists where possible.

My recommendation:

For anything your users will see, this PR will work great, across all tools (i.e. just a simple numpy>=1.24 requirement)

For your own development / lockfiles, you could try some equivalent of:

numpy>=1.24
numpy>=1.26; python_version >= 3.12

since numpy 1.26 is the first version of numpy to ship wheels for Python 3.12

What you have on master currently also seems a little unnecessarily restrictive, e.g. your users on Python 3.10 and newer won't be able to use numpy 1.26

[reyammer]

Thank you @hauntsaninja! Super useful. Re-opened this PR while I give some thought to it :-)