Trusted Publishing Support on Crates.io

[mdtro]

/cc @rust-lang/crates-io

A big thank you to @woodruffw for co-authoring, providing prior art through PyPi's implementation, and all of the expert advice. 🙏

Rendered

[woodruffw]

Thank you so much for your hard work authoring this @mdtro! It was my honor and pleasure to be able to help.

As one of the people who built the equivalent Trusted Publishing feature on PyPI, I'm more than happy to answer any technical or policy questions the Rust community has, as well as offer insight into PyPI's experience (which IMO has been extremely successful) over the past 18 months of having Trusted Publishing deployed.

[programmerjake]

I think we should try to support 3rd-party websites that have their own gitlab/forgejo/gitea/etc. instances, so e.g. gitlab.example.com could publish to whatever crates they own even though they aren't using gitlab.com's CI and instead are running their own CI infrastructure.

this could perhaps be done by, when CI asks crates.io for an OIDC token, having crates.io use oauth/oidc to check that gitlab.example.com grants permission for CI uploads through a token provided to crates.io by CI

[woodruffw]

I think we should try to support 3rd-party websites that have their own gitlab/forgejo/gitea/etc. instances, so e.g. gitlab.example.com could publish to whatever crates they own even though they aren't using gitlab.com's CI and instead are running their own CI infrastructure.

It's ultimately up to each index to decide a subjective cutoff for IdP "popularity," but I would caution against this: the main security benefit of trusted publishing versus an API token is that large CI/CD providers have dedicated OIDC IdP maintenance and operation teams that handle the burden of maintaining an OIDC PKI. For one-off instances, the benefits of a PKI versus ordinary API tokens are marginal and may even invert, since maintaining a PKI is significantly more operationally complicated than securing a single API token.

(For PyPI, this is one of the reasons we started with GitHub, and then moved to support GitLab, Google Cloud Build, etc., but haven't yet moved to support third-party instances of GH or GL.)

[di]

(For PyPI, this is one of the reasons we started with GitHub, and then moved to support GitLab, Google Cloud Build, etc., but haven't yet moved to support third-party instances of GH or GL.)

+1 to what @woodruffw said, also to add to this: PyPI has a notion of "organizations", and one thing we are considering is for PyPI is to permit self-hosted IdPs 1:1 with organizations on a case by case basis.

[lyphyser]

I think it should really be implemented so that you need BOTH a crates.io API token AND the OpenID Connect identity token.

Otherwise, if there is a bug in the OpenID Connect implementation by GitHub/Google/etc., someone exploiting it could take over all crates using it without having to take over the actual developer machines or CI systems where the API token would be stored; this also guarantees that security is strictly improved since even if the OpenID Connect implementation on crates.io's side were totally broken, it would still be as secure as the current system.

The best way to do this seems to be to change the crates.io API token creation UI to have the option to also require an OpenID Connect identity to be provided to accept requests using that token.

[woodruffw]

Otherwise, if there is a bug in the OpenID Connect implementation by GitHub/Google/etc., someone exploiting it could take over all crates using it without having to take over the actual developer machines or CI systems where the API token would be stored; this also guarantees that security is strictly improved since even if the OpenID Connect implementation on crates.io's side were totally broken, it would still be as secure as the current system.

Could you elaborate on the threat model you're envisioning here? We considered similar scenarios when building out the threat model for trusted publishing on PyPI, and ultimately came to the conclusion that an attacker who is sufficiently powerful to control a major OIDC IdP (like Google's or GitHub's) would almost certainly also have sufficient power to control CI-side user-configured credentials.

Or in other words: we couldn't think of an internally coherent threat model in which an attacker is simultaneously strong enough to take over a major OIDC IdP but too weak to compromise an individual CI process on that IdP's platform (and thereby exfiltrate a manually-configured crates.io API token).

(More broadly, I think Trusted Publishing's security and usability benefits become moot if they require two credentials - one manual - instead of just an automatic one: the goal is to remove error prone manual steps and opportunities for over-scoping/accidental disclosure, both of which would still exist if the user would still need to configure a crates.io API token.)

[Turbo87]

good work @mdtro and everyone involved! I'm excited for this to land in crates.io :)

[Turbo87]

very minor nitpick, we've actually changed the default last week to expire in 90 days 😄

[Turbo87]

what is the rationale for requiring this instead of making it optional?

[di]

It's not strictly required, but I would strongly recommend requiring it: making this claim optional would mean that any workflow in the source repository could generate an OIDC token with the ability publish. Requiring it to be scoped to a single workflow significantly reduces the opportunity for compromise here.

Additionally, the use case of wanting to publish from multiple workflows with a single configuration is not a strong one, I think. Usually there is just a single workflow which can publish, and if a user really needs to publish from multiple workflows, they can always configure multiple publishers with varying workflow filenames instead.

[Turbo87]

Requiring it to be scoped to a single workflow significantly reduces the opportunity for compromise here.

I'm not sure what the threat model here is. If an attacker is able to add a new workflow to the repository then wouldn't they also be able to overwrite the existing publish workflow?

the use case of wanting to publish from multiple workflows with a single configuration is not a strong one, I think

yeah, I agree with that. I was thinking more about an unknowing dev renaming a workflow file in some refactoring without realizing that this breaks the publish pipeline.

[di]

The threat would be third-party code that is necessary for the workflow to function being compromised to steal OIDC credentials (or API tokens, for that matter).

Consider two separate workflows: one that does testing & linting, another that handles publishing on release. The former may introduce a lot of dependencies that aren't required for publishing, but are required for testing/linting. If this test/lint workflow also has the ability to generate OIDC credentials that can be used for publishing, this unnecessarily increases the surface area that the OIDC token is exposed to.

(Note that for GitHub as an IdP specifically, this may be sort of a moot point, because providing the token to the workflow requires an explicit configuration in the workflow with id-token: write, however this is fairly easy to over-configure.)

[Turbo87]

I see, what you're saying is that the actions used inside of the workflow might get compromised. makes sense, thanks for the clarification. I wonder if we should include a sentence about it in the RFC so that it's documented.

[riking]

And specifically, actions used inside of workflows not named release.yml being compromised is a lot easier to contemplate. This functionally reduces the scope of your audit to just the release workflow.

[Turbo87]

cargo publish should already download everything that is needed. do we need this additional step for anything?

[riking]

This makes sure the crate successfully builds before acquiring the token. If the crate takes 1+ hour to build, the token might expire during the cargo publish verification checks.

[tarcieri]

Note some related discussion below, where one idea was cargo publish itself could manage token provisioning and do it immediately before uploading an already built crate to avoid this problem

[Turbo87]

I don't think this is correct. It may be a leftover from the "pending state" idea in an earlier draft. The configs discussed here should be in a one-to-one or many-to-one relationship with a crate, but not a user or team.

[di]

Agreed, it makes more sense to make the relationship between the project and the publisher, and confer ability to modify the configuration on any user who can otherwise modify the project.

[Turbo87]

since the config in this RFC can only be added to existing crates I don't think this the token scopes question applies here

[Turbo87]

do we know what the other registries have chosen and the reasons for their choices?

I guess an "issue" is that cargo by default builds the crate before it is sent to crates.io. Depending on how long this build process takes, the token may have already expired by the time cargo sends the crate to crates.io.

I wonder how viable it would be to provide a config option for the token lifetime.

Alternatively: cargo provides support for other authentication methods these days, e.g. to integrate with 1Password. Would it be viable to integrate the authentication flow through that? It would presumably allow us to request the token from crates.io just in time before the upload happens. Admittedly I don't know at what point cargo talks to the auth provider process. It might happen before the build process too. Might be worth exploring though.

[Mark-Simulacrum]

It seems like integrating this into Cargo's default set of auth providers also avoids the separate "auth" repository/action maintenance, which feels like more of a "new thing" in our release processes etc. It also seems like native integration in Cargo is probably better for users -- the risk of using the wrong repository (or an outdated version) is probably reduced.

[Turbo87]

I guess it's up to the cargo team whether they would want to make this a built-in thing. It looks like they have quite a long to-do list already though 😅

I don't think the action would have to be integrated with the Rust release process though. All it needs is a repository with some git tags, which shouldn't be too hard to set up an maintain.

[di]

For PyPI, the token is valid for 15 minutes from the time that it is minted. Our reasoning for this is that 15 minutes is generally enough time to perform any uploading needed, and anyone that needed more time could always re-request a token.

I guess an "issue" is that cargo by default builds the crate before it is sent to crates.io. Depending on how long this build process takes, the token may have already expired by the time cargo sends the crate to crates.io.

I'm a little unfamiliar with the general workflow here, but it might not be necessary to exchange an OIDC token for an upload token prior to the build, i.e. this could happen after a build completes, but just before an upload will happen instead, so your only concern is the length of time of the upload.

[Turbo87]

15 minutes is generally enough time to perform any uploading needed

if I understand correctly this kind of originates in PyPI allowing file uploads after a version was created? (vs. crates.io where you need to upload your one crate file upfront)

this could happen after a build completes, but just before an upload will happen instead, so your only concern is the length of time of the upload.

the short version is cargo publish performs both the build and the upload and there isn't really a step in between that we could hook into from the outside (except maybe with the auth providers that I mentioned above). the uploads are currently limited to 30sec anyway due to hosting provider limitations.

[di]

if I understand correctly this kind of originates in PyPI allowing file uploads after a version was created? (vs. crates.io where you need to upload your one crate file upfront)

Yea, kind of, more like it originates from a PyPI release consisting of multiple files, which are uploaded in multiple separate requests rather than a single request, and that some of those files can be huge, which takes some time to upload.

the short version is cargo publish performs both the build and the upload and there isn't really a step in between that we could hook into from the outside

I guess the question is whether the OIDC token exchange needs to happen externally (and prior to) cargo publish being invoked, or whether the token exchange is the responsibility of cargo publish, in which case it could happen between the build & publish steps.

For PyPI, we chose to do the exchange in our canonical upload workflow prior to invoking twine upload, but we have plans to make twine upload support handling the exchange directly: pypa/twine#999. In either case, the build has already happened though, so we can keep this window relatively small.

[Mark-Simulacrum]

Can we add some discussion of machine-accessible API to provision the configuration here? In particular, for rust-lang/* repositories, I'd love to control this access via rust-lang/team, rather than the current manual configuration process. For some repositories manual configuration (even with this feature) would still mean that we're regularly asking humans to access highly privileged accounts (rust-lang-owner) to update settings, and in a way that likely lacks good review + audit trail (e.g., no 2 person review on the actual policy set).

[Turbo87]

there will definitely be an API for it since that is currently required by the crates.io user interface anyway. I would expect us to only enable it for cookie-based auth at first though until we're happy with the API and consider it stable.

[bjorn3]

Would it make sense to treat an empty environment string as mandating that the publishing is not done from an environment? This may help users fill in which environment allows publishing if they use environments at all on their CI rather than forgetting to fill it in implicitly acting as wildcard and thus accidentally allow publishing outside of an environment.