Can no longer authenticate using --k8s-keychain on AWS EKS with cosign > 2.2.1

[pgpx]

Description

I'm running cosign in an AWS EKS k8s cluster, using a service account mapped to an IAM role. Previously, using cosign 2.2.1, I could run cosign with --k8s-keychain which would automatically authenticate with the ECR repo of the image to be signed. However, from 2.2.2 onwards (including 2.3.0, 2.40, and 2.4.1) cosign fails to authenticate.

Cosign 2.2.1 example log:

cosign sign -d --k8s-keychain '--key=awskms:///arn:aws:kms:eu-central-1:<aws-account-id>:alias/aicc-models-dev-cosign' <ecr-repo>@sha256:aeb0db2294de0d396087c200ed77e415821416299a206eb8b5812bc615e1d794 '--tlog-upload=false'
2024/10/30 14:04:19 --> GET https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/v2/
2024/10/30 14:04:19 GET /v2/ HTTP/1.1
Host: <aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com
User-Agent: cosign/v2.2.1 (linux; amd64) go-containerregistry/v0.16.1
Accept-Encoding: gzip
2024/10/30 14:04:19 <-- 401 https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/v2/ (6.859044ms)
2024/10/30 14:04:19 HTTP/1.1 401 Unauthorized
Content-Length: 15
Content-Type: text/plain; charset=utf-8
Date: Wed, 30 Oct 2024 14:04:19 GMT
Docker-Distribution-Api-Version: registry/2.0
Sizes: 
Www-Authenticate: Basic realm="https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/",service="ecr.amazonaws.com"
Not Authorized
2024/10/30 14:04:19 --> GET https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/v2/<ecr-repo>/manifests/sha256:aeb0db2294de0d396087c200ed77e415821416299a206eb8b5812bc615e1d794
2024/10/30 14:04:19 GET /v2/<ecr-repo>/manifests/sha256:aeb0db2294de0d396087c200ed77e415821416299a206eb8b5812bc615e1d794 HTTP/1.1
Host: <aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com
User-Agent: cosign/v2.2.1 (linux; amd64) go-containerregistry/v0.16.1
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Authorization: <redacted>
Accept-Encoding: gzip
2024/10/30 14:04:19 <-- 200 https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/v2/<ecr-repo>/manifests/sha256:aeb0db2294de0d396087c200ed77e415821416299a206eb8b5812bc615e1d794 (90.865369ms)
2024/10/30 14:04:19 HTTP/1.1 200 OK

Note the Authorization: <redacted> line near the end. That is not found for Cosign 2.2.2:

cosign sign -d --k8s-keychain '--key=awskms:///arn:aws:kms:eu-central-1:<aws-account-id>:alias/aicc-models-dev-cosign' <ecr-repo>@sha256:efcedb5061460c663ab844219d5cff51163136f658b543fe05344a52ab549e05 '--tlog-upload=false'
2024/10/30 14:15:22 --> GET https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/v2/
2024/10/30 14:15:22 GET /v2/ HTTP/1.1
Host: <aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com
User-Agent: cosign/v2.2.2 (linux; amd64) go-containerregistry/v0.17.0
Accept-Encoding: gzip
2024/10/30 14:15:22 <-- 401 https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/v2/ (7.625738ms)
2024/10/30 14:15:22 HTTP/1.1 401 Unauthorized
Content-Length: 15
Content-Type: text/plain; charset=utf-8
Date: Wed, 30 Oct 2024 14:15:22 GMT
Docker-Distribution-Api-Version: registry/2.0
Sizes: 
Www-Authenticate: Basic realm="https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/",service="ecr.amazonaws.com"
Not Authorized
2024/10/30 14:15:22 --> GET https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/v2/<ecr-repo>/manifests/sha256:efcedb5061460c663ab844219d5cff51163136f658b543fe05344a52ab549e05
2024/10/30 14:15:22 GET /v2/<ecr-repo>/manifests/sha256:efcedb5061460c663ab844219d5cff51163136f658b543fe05344a52ab549e05 HTTP/1.1
Host: <aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com
User-Agent: cosign/v2.2.2 (linux; amd64) go-containerregistry/v0.17.0
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Accept-Encoding: gzip
2024/10/30 14:15:22 <-- 401 https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/v2/<ecr-repo>/manifests/sha256:efcedb5061460c663ab844219d5cff51163136f658b543fe05344a52ab549e05 (1.018788ms)
2024/10/30 14:15:22 HTTP/1.1 401 Unauthorized
Content-Length: 15
Content-Type: text/plain; charset=utf-8
Date: Wed, 30 Oct 2024 14:15:22 GMT
Docker-Distribution-Api-Version: registry/2.0
Sizes: 
Www-Authenticate: Basic realm="https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/",service="ecr.amazonaws.com"
Not Authorized
Error: signing [<ecr-repo>@sha256:efcedb5061460c663ab844219d5cff51163136f658b543fe05344a52ab549e05]: accessing image: GET https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/v2/<ecr-repo>/manifests/sha256:efcedb5061460c663ab844219d5cff51163136f658b543fe05344a52ab549e05: unexpected status code 401 Unauthorized: Not Authorized
main.go:74: error during command execution: signing [<ecr-repo>@sha256:efcedb5061460c663ab844219d5cff51163136f658b543fe05344a52ab549e05]: accessing image: GET https://<aws-account-id>.dkr.ecr.eu-central-1.amazonaws.com/v2/<ecr-repo>/manifests/sha256:efcedb5061460c663ab844219d5cff51163136f658b543fe05344a52ab549e05: unexpected status code 401 Unauthorized: Not Authorized

Version

2.2.2, 2.2.4, 2.3.0, 2.4.0, 2.4.1