From 78c4086b041db07a2d560884d658e7b3990b5a6d Mon Sep 17 00:00:00 2001 From: Ethan Smith Date: Mon, 20 May 2024 16:48:49 -0400 Subject: [PATCH 1/8] Add PEP 778 and reserve 777 --- peps/pep-0777.rst | 81 +++++++++++++++ peps/pep-0778.rst | 260 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 341 insertions(+) create mode 100644 peps/pep-0777.rst create mode 100644 peps/pep-0778.rst diff --git a/peps/pep-0777.rst b/peps/pep-0777.rst new file mode 100644 index 00000000000..e6b54786549 --- /dev/null +++ b/peps/pep-0777.rst @@ -0,0 +1,81 @@ +PEP: 777 +Title: Reinventing the Wheel (Wheel 2.0) +Author: Ethan Smith +Status: Draft +Type: Standards Track +Topic: Packaging +Created: 20-May-2024 + +- Should mention https://discuss.python.org/t/speculative-wheel-2-0-and-migration-strategies/21600/13 + +Abstract +======== + +[A short (~200 word) description of the technical issue being addressed.] + + +Motivation +========== + +[Clearly explain why the existing language specification is inadequate to address the problem that the PEP solves.] + + +Rationale +========= + +[Describe why particular design decisions were made.] + + +Specification +============= + +[Describe the syntax and semantics of any new language feature.] + + +Backwards Compatibility +======================= + +[Describe potential impact and severity on pre-existing code.] + + +Security Implications +===================== + +[How could a malicious user take advantage of this new feature?] + + +How to Teach This +================= + +[How to teach users, new and experienced, how to apply the PEP to their work.] + + +Reference Implementation +======================== + +[Link to any existing implementation and details about its state, e.g. proof-of-concept.] + + +Rejected Ideas +============== + +[Why certain ideas that were brought while discussing this PEP were not ultimately pursued.] + + +Open Issues +=========== + +[Any points that are still being decided/discussed.] + + +Footnotes +========= + +[A collection of footnotes cited in the PEP, and a place to list non-inline hyperlink targets.] + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. diff --git a/peps/pep-0778.rst b/peps/pep-0778.rst new file mode 100644 index 00000000000..018833a91f5 --- /dev/null +++ b/peps/pep-0778.rst @@ -0,0 +1,260 @@ +PEP: 778 +Title: Supporting Symlinks in Wheels +Author: Ethan Smith +Sponsor: Barry Warsaw +Status: Draft +Type: Standards Track +Topic: Packaging +Content-Type: text/x-rst +Requires: 777 +Created: 18-May-2024 + +Abstract +======== + +Wheels currently do not handle symlinks well, copying content instead of making symlinks when +installed. To properly handle distributing libraries in wheels, we propose a new ``LINKS`` +metadata file to handle symlinks in a platform portable manner. This specification requires +a new wheel major version, discussed in :pep:`777`. + +Motivation +========== + +Today, symlinks in wheels get created as copies of files, as `the zipfile module +`_ in CPython `does not support handling symlinks +in-place `_ for security reasons. + +This `presents problems to projects that would like to ship large compiled libraries +`_ in +wheels, as they must choose to either greatly increase the install size of the project on disk, +or omit the symlink and potentially break some downstream use cases. + +To ship a library that can properly be loaded for runtime use or build time linking on Unix, a +library should follow conventions of Unix loader and linker search. The two main file names for +the loader to use is the "soname" and the "real name". The "soname" is a file like +``libfoo.so.3`` where ``3`` is a number that is incremented when the interface of the library +changes. The "real name" is a file named like ``libfoo.so.3.1.4``, where the extra version +information lets the loader find a specific version of a library. Finally, when compiling code to +link against a library, the linker searches for a "linker name", named like ``libfoo.so``. A more +detailed description is described in `this Linux documentation on shared libraries +`_. To fully support all +runtime and build time use cases, a project requires shipping all 3 files. Normally, this is +handled on Unix by using symlinks, so that the library is not duplicated on disk 3 times. + +Returning to Python packaging, there are many popular projects which ship binary libraries, such as +``numpy``, ``scipy``, and ``pyarrow``. Other site-packages ``dlopen`` libraries in other wheels, such as +``pytorch`` and ``jax``. These projects currently rely on a single library in the wheel, but +this can cause the linker to find the wrong library if there are system libraries that have a +"real name" library version available. + +There is also the potential benefit that symlinks in wheels would allow for simpler editable +installs by simply placing a symlink in the user's ``site-packages`` directory, but this PEP +leaves that as an open question to be explored in a future PEP. + +Rationale +========= + +To support the 3 main namings of a library used in loading and library linking on Unix, we +propose adding support for symlinks in Python wheels. To allow for tracking symlinks made, and to +potentially support other platforms that may not support Unix symlinks directly, we propose the +use of a new wheel metadata file ``LINKS``, which will exist in the ``.dist-info`` directory alongside +``METADATA``, ``RECORD``, and other metadata files. + +Using a ``LINKS`` file will allow for more cross-platform uses of symlink-like usage. On Windows, +symlinks require either `a group policy allowing the user to make symlinks +`_ +(e.g. by enabling `Dev Mode +`_) +or Administrative permissions. This means that it may be the case that symlinks are unsupported on +some user systems. By using a ``LINKS`` file, installers will be able to potentially use other +methods of handlings symlinks, such as junctions on Windows, where otherwise the installer would +have to fail. + +This PEP also describes checks that installers must make when installing an updated wheel. These +checks exist to handle security risks from allowing wheels to install symlinks. For more +information on why these checks are important, see `Security Implications`_. + +Specification +============= + +Wheel Major Version Bump +------------------------ + +This PEP requires a wheel major version bump, so the ``Wheel-Version`` for wheels generated with +``LINKS`` **MUST** be at least version ``2.0``, so that older installers do not silently fail to +install symlinks and break user environments. For more see :pep:`777`. + +New ``LINKS`` Metadata File +--------------------------- + +To enable cross-platform symlinks, this PEP introduces a new wheel metadata file, ``LINKS``. An +example of a ``LINKS`` file is below:: + + my_package/libfoo.so.3.1.4,my_package/libfoo.so.3 + my_package/libfoo.so.3,my_package/libfoo.so + +The format of ``LINKS``, as can seen above, is ``source_path,target_path`` where ``source_path`` +is a path relative to the root of any namespace or package root in the wheel. ``target_path`` is a +*non-dangling* path (i.e. a path that exists on the filesystem after the wheel's contents are +extracted) in a package or namespace of any package in the wheel. This means that if a wheel +contains multiple packages, all paths in packages in the wheel are acceptable. + +Installer Behavior Specification +-------------------------------- + +Installers **MUST** resolve the paths of any link contained in the ``LINKS`` file *before* +deciding if any ``source_path`` or ``target_path`` are valid. Installers **MUST** verify that +``source_path`` and ``target_path`` are located inside any namespace or package coming from the +wheel. Installers **MUST** reject cyclic symlinks in wheels. Installers **MAY** error if a long +chain of symlinks (symlinks pointing to symlinks many times repeated) exceeds a limit set by the +installer. + +Installers **MUST** follow the following steps when handling a wheel with symlinks: + +1. Check for the existance of a ``LINKS`` file in the ``.dist-info``. If it does not exist, + no further steps are required. +2. Extract all files in the wheel packages and data directory as in wheel 1.x. +3. Verify that for each ``source_path`` and ``target_path`` pairs, the ``target_path`` exists in + one of the package namespaces just extracted. +4. Next, check that the installer can make some kind of link for each pair in the site directory. + If the installer cannot make a link for the file/folder ``target_path`` for the current + platform, an error **MUST** be raised. An example of a failure mode would be a UNIX symlink to + a file target, where the installer is running on Windows and the installer cannot make + symlinks but can make junctions. In this case the installer **MUST** error because it cannot + handle the link. +5. Finally, the installer **MUST** add a platform-relevant link between ``source_path`` and + ``target_path``. + +Installers **MUST NOT** by default copy files instead of generating a symlink when handling +symlinks. Installers **MAY** have such behavior available under an alternate configuration or +command line flag. + +Build Backend Specification +--------------------------- + +When creating a wheel, build backends **MUST** treat symlinks in the same way as its target when +deciding whether to include the symlink in a wheel. Build backends **MUST** verify that there are +no dangling symlinks in the ``LINKS`` file. Build backends **SHOULD** recognize platform-relevant +symlinks that would be included in builds. On Unix this is typically symlinks, on Windows this +includes symlinks and junctions. + +Backwards Compatibility +======================= + +Introducing symlinks would require an increment to the wheel format major version. This would mean +new wheels that use the new wheel format would raise an error on older installer tools, per the +`wheel specification +`_. + +Please see :pep:`777` on "Wheel 2.0". + +Security Implications +===================== + +Symlinks can be quite dangerous if not handled carefully. A simple example would be if a user were +to run ``sudo pip install malicious``, and there were no protections, then the malicious package +could overwrite ``/etc/shadow`` and replace the password hash on the system, allowing malicious +logins. + +This PEP lists several requirements on checks to run by installers on symlinks in wheels to ensure +attacks like the one described above cannot happen. This means it is **critical** that installers +carefully implement these security safeguards and prevent malicious use on package installation. + +In particular, the following checks **MUST** be made by installers: + +1. That the symlinks do not point outside of any packages or namespaces coming from the wheel +2. That the symlinks are not dangling (the target exists at install time) +3. That the symlinks are not cyclical, stopping after a certain depth of checking to avoid denial + of service requests + +Do not follow symlinks on removal. + +How to Teach This +================= + +End users should, once the changes have propagated through the ecosystem, transparently experience +the benefits of symlinks in wheels. It is important for installers to give clear error messages if +symlinks are unsupported on the platform, and explain why installation has failed. + +For people building libraries, documentation on ``packaging.python.org`` should describe the use +cases and caveats (especially platform suppport) of symlinks in wheels. Otherwise it should be +handled transparently by build backends in the same way any normal file would be handled. + +Reference Implementation +======================== + +TODO + +Rejected Ideas +============== + +Just Use Unix Symlinks Everywhere +--------------------------------- + +This PEP wants to allow for ``LINKS`` to be used for a potential future :pep:`660` editable +installation. This future PEP should support Windows, so it may need to use junctions. + +Don't Use Junctions in ``LINKS`` +-------------------------------- + +Junctions are a limited way to support symlinks between folders on Windows. They do not support +files. This PEP allows for junctions as users may wish to only link folders to a different +location, and future :pep:`660` implementations may need to rely on this feature. + +Put symlinks in the ``RECORD`` Metadata File +-------------------------------------------- + +While this could be done, it would clutter the ``RECORD`` file. Furthermore the most +straightforward implementation would place the target at the end of the record. This would +make it harder to scan across the line and visually see what symlinks exist in the wheel. + +Library Maintainers Should Use Python to Locate Libraries +--------------------------------------------------------- + +Using Python to locate libraries would be much easier. However, some libraries like ``libtorch`` +are used by extension modules and themselves require loading dependencies. Some compiled libraries +cannot use Python to find their loader dependencies. + +Include Support for Hardlinks +----------------------------- + +This PEP does not specify any behavior around hardlinks. This is intentional. This is left as an +extension to a future PEP. + +Open Issues +=========== + +PEP 660 and Deferring Editable Installation Support +------------------------------------------------------ + +This PEP leaves the specification and implementation of a :pep:`660` editable installation +mechanism as unresolved for a later PEP, should we specify that here? + +Security +-------- + +This PEP needs to be reviewed to make sure it would not allow for new security vulnerabilities. +Are there other restrictions we should place on the source or target of symlinks to protect users? + +Allow inter-package symlinks +---------------------------- + +This could be useful for projects that want to shard dependencies such as large libraries between +wheels but make them available in the main parent wheel. + +The Format of ``LINKS`` +----------------------- + +Currently the format is derived from ``RECORD``, but perhaps a better format exists. + +Previous Discussion +=================== + +https://discuss.python.org/t/symbolic-links-in-wheels/1945/25 + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 888278eeb9d00a6ee7d90f2848336c3ed4bbc083 Mon Sep 17 00:00:00 2001 From: Ethan Smith Date: Wed, 22 May 2024 09:00:48 -0700 Subject: [PATCH 2/8] Add Paul Moore as PEP delegate Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0778.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/peps/pep-0778.rst b/peps/pep-0778.rst index 018833a91f5..177d411a19e 100644 --- a/peps/pep-0778.rst +++ b/peps/pep-0778.rst @@ -2,6 +2,7 @@ PEP: 778 Title: Supporting Symlinks in Wheels Author: Ethan Smith Sponsor: Barry Warsaw +PEP-Delegate: Paul Moore Status: Draft Type: Standards Track Topic: Packaging From b231bca4a87f9087c9e328e4c291068f7b672609 Mon Sep 17 00:00:00 2001 From: Ethan Smith Date: Wed, 22 May 2024 09:01:00 -0700 Subject: [PATCH 3/8] Add Paul Moore as PEP delegate Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0777.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/peps/pep-0777.rst b/peps/pep-0777.rst index e6b54786549..df303331298 100644 --- a/peps/pep-0777.rst +++ b/peps/pep-0777.rst @@ -1,6 +1,8 @@ PEP: 777 Title: Reinventing the Wheel (Wheel 2.0) Author: Ethan Smith +Sponsor: Barry Warsaw +PEP-Delegate: Paul Moore Status: Draft Type: Standards Track Topic: Packaging From 48a2f0fb16dac135bcf14a505a9e20af6d9a790d Mon Sep 17 00:00:00 2001 From: Ethan Smith Date: Wed, 9 Oct 2024 12:20:50 -0700 Subject: [PATCH 4/8] Update peps/pep-0778.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0778.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/peps/pep-0778.rst b/peps/pep-0778.rst index 177d411a19e..21fc0b517e9 100644 --- a/peps/pep-0778.rst +++ b/peps/pep-0778.rst @@ -6,7 +6,6 @@ PEP-Delegate: Paul Moore Status: Draft Type: Standards Track Topic: Packaging -Content-Type: text/x-rst Requires: 777 Created: 18-May-2024 From 9d28e9ae076a971c37d11dc32dda1c7ab0356fab Mon Sep 17 00:00:00 2001 From: Ethan Smith Date: Wed, 9 Oct 2024 12:21:14 -0700 Subject: [PATCH 5/8] Remove extra underline Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0778.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0778.rst b/peps/pep-0778.rst index 21fc0b517e9..bdec1c551b5 100644 --- a/peps/pep-0778.rst +++ b/peps/pep-0778.rst @@ -225,7 +225,7 @@ Open Issues =========== PEP 660 and Deferring Editable Installation Support ------------------------------------------------------- +--------------------------------------------------- This PEP leaves the specification and implementation of a :pep:`660` editable installation mechanism as unresolved for a later PEP, should we specify that here? From 83d484dedbda14b368ff70c7ba236133ac0b5bc7 Mon Sep 17 00:00:00 2001 From: Ethan Smith Date: Wed, 9 Oct 2024 12:22:52 -0700 Subject: [PATCH 6/8] Fix misspelling Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0778.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0778.rst b/peps/pep-0778.rst index bdec1c551b5..919a72b3b14 100644 --- a/peps/pep-0778.rst +++ b/peps/pep-0778.rst @@ -111,7 +111,7 @@ installer. Installers **MUST** follow the following steps when handling a wheel with symlinks: -1. Check for the existance of a ``LINKS`` file in the ``.dist-info``. If it does not exist, +1. Check for the existence of a ``LINKS`` file in the ``.dist-info``. If it does not exist, no further steps are required. 2. Extract all files in the wheel packages and data directory as in wheel 1.x. 3. Verify that for each ``source_path`` and ``target_path`` pairs, the ``target_path`` exists in From bdf9e2dfe296fea7db80b34712c97520dcebd5ca Mon Sep 17 00:00:00 2001 From: Ethan Smith Date: Wed, 9 Oct 2024 12:28:27 -0700 Subject: [PATCH 7/8] Remove PEP 777 The PEP has been split out into https://github.com/python/peps/pull/4036 --- peps/pep-0777.rst | 83 ----------------------------------------------- 1 file changed, 83 deletions(-) delete mode 100644 peps/pep-0777.rst diff --git a/peps/pep-0777.rst b/peps/pep-0777.rst deleted file mode 100644 index df303331298..00000000000 --- a/peps/pep-0777.rst +++ /dev/null @@ -1,83 +0,0 @@ -PEP: 777 -Title: Reinventing the Wheel (Wheel 2.0) -Author: Ethan Smith -Sponsor: Barry Warsaw -PEP-Delegate: Paul Moore -Status: Draft -Type: Standards Track -Topic: Packaging -Created: 20-May-2024 - -- Should mention https://discuss.python.org/t/speculative-wheel-2-0-and-migration-strategies/21600/13 - -Abstract -======== - -[A short (~200 word) description of the technical issue being addressed.] - - -Motivation -========== - -[Clearly explain why the existing language specification is inadequate to address the problem that the PEP solves.] - - -Rationale -========= - -[Describe why particular design decisions were made.] - - -Specification -============= - -[Describe the syntax and semantics of any new language feature.] - - -Backwards Compatibility -======================= - -[Describe potential impact and severity on pre-existing code.] - - -Security Implications -===================== - -[How could a malicious user take advantage of this new feature?] - - -How to Teach This -================= - -[How to teach users, new and experienced, how to apply the PEP to their work.] - - -Reference Implementation -======================== - -[Link to any existing implementation and details about its state, e.g. proof-of-concept.] - - -Rejected Ideas -============== - -[Why certain ideas that were brought while discussing this PEP were not ultimately pursued.] - - -Open Issues -=========== - -[Any points that are still being decided/discussed.] - - -Footnotes -========= - -[A collection of footnotes cited in the PEP, and a place to list non-inline hyperlink targets.] - - -Copyright -========= - -This document is placed in the public domain or under the -CC0-1.0-Universal license, whichever is more permissive. From 01e03296ff4dd709f09a62c43272de31091c40d1 Mon Sep 17 00:00:00 2001 From: Ethan Smith Date: Wed, 9 Oct 2024 12:37:38 -0700 Subject: [PATCH 8/8] Add codeowners --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index a90a99d7547..a94f629700c 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -626,6 +626,7 @@ peps/pep-0744.rst @brandtbucher peps/pep-0745.rst @hugovk # ... # peps/pep-0754.rst +peps/pep-0778.rst @warsaw # ... peps/pep-0801.rst @warsaw # ...