A thought in process: Pipfile and pyproject.toml

[kalebmckale]

The more I work with pipenv and Pipfiles, the more I’m wondering if there isn’t an opportunity for simplification, or at least an option.

Most major Python tools have begun to implement an option to store their information and configuration in the pyproject.toml file. While there is no replacement for Pipfile.lock, I wonder if the contents of the user Pipfile itself couldn’t be incorporated into pyproject.toml. Mind you, not as replacement as most tools do still look for their own configuration files if not locating configuration in pyproject.toml.

The nice thing is that Pipfile already uses the TOML standard. Additionally, in most use cases, the dependencies specified in pyproject.toml and in Pipfile packages are identical. I personally would also welcome defining my sources in my pyproject.toml.

The one incompatibility to this idea currently, which I’ve not yet been able to figure out a nice solution for, is the differences between the specifications of versions in pyproject.toml dependencies and Pipfile. I very much prefer the latter, especially since the Pipfile.lock contains all that information, which it has verified and not up to the user to enter by hand.

Brainstorming a bit on the fly here, I wonder if using the dynamic option isn’t the path to solve this. I realize this may require an inter-project conversation and development given the Pipfile parser is a different project, but I know something like this would go light years in convincing those at least in my work place to adopt pipenv as a part of the development workflow.

Final thoughts: pip + pipenv would now have everything that Poetry has but using Python’s standard tool instead, which may just be the thing some Poetry users would find attractive enough to switch, and more users often means more developers on the project.

[hiruthikj]

Yes, poetry already does it. Will be great if pipenv also follows the same.

[o-alquimista]

YES! This duplication is bugging me.

pyproject.toml

[project]
dependencies = [
  "requests~=2.31",
]

Pipfile

[packages]
requests = "~=2.31"

[eldjh3]

This lack of a consistent means of specifying dependencies has been frustrating me too and I also hate the duplication of specifying dependencies in multiple places. As far as possible I like to use standards based tools but there doesn't seem to be a consensus in the python community for some packaging aspects.

For a package, it makes perfect sense to me that dependencies shouldn't be locked as that package, when installed into a user's environment, may need to cohabit nicely with other packages. That implies that pyproject.toml shouldn't contain pinned versions. However, when deploying an application I want to ensure the install uses the dependencies that have been tested against.

I've currently got the following workflow, it's not perfect but it seems to work. Happy to receive feedback on it.

A pyproject.toml file defining the application's properties, including its top-level dependencies. None are pinned.

# pyproject.toml
dependencies = [
    "django>=4.2",
]

A minimal Pipfile, it only specifies an internal pypi repo in addition to pypi.org and packages to resolve from there but doesn't actually specify anything beyond that dependency wise for the project.

# Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
... properties for internal repo...
name = "internal"

[packages]
internal_dependency = { index="internal" }

When I create the build environment (either for dev or for building a release)

$ pipenv install -e .
$ # Explicitly install dev specific dependencies
$ pipenv install --dev black
$ pipenv install --dev isort
$ # etc etc...

The key thing is that the editable install resolves the dependencies as specified in pyproject.toml so I don't need to repeat them in the Pipfile.

Come release time, because this is an application and I want pinned dependencies, I use pipenv to generate a requirements.txt file and install this into the environment before installing the project's own wheel.

I deploy using docker images, so I build the virtual environment, then copy it into the runtime container in a multi-stage docker image build.

pipenv requirements > requirements.txt
sed -i '/^-e/d' requirements.txt. # Remove the dependency on the editable install of the 'main' package/application

During the docker build

pipenv run pip install -r requirements.txt
pipenv install "<my_app_name> == ${VERSION}" -i internal

The explicit install via requirements.txt seems to keep the dependencies pinned as the versions specified within it when installing the application package.

No doubt there are pitfalls to this, but it seems to be working so far. As mentioned, happy to receive feedback and hope this may be of some use to avoid duplication or dependency specifications.