Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a GCS deploy workflow #45

Merged
merged 3 commits into from
Mar 5, 2024
Merged

Add a GCS deploy workflow #45

merged 3 commits into from
Mar 5, 2024

Conversation

jku
Copy link
Member

@jku jku commented Feb 13, 2024

Fixes #7

This uploads the repository to GCS and invalidates the CDN cache.

  • the workflow is a custom one for root-signing
  • it is called as a reusable workflow from the TUF-on-CI publish workflow (so gets automated issue failing on errors)

GCP configuration is mostly hard coded, only service account details are inputs:

  • service account: same as online signing
  • project: projectsigstore-staging
  • GCS bucket: tuf-root-staging
  • CDN load balancer: tuf-repo-cdn-lb

@jku
Copy link
Member Author

jku commented Feb 13, 2024

Comparison to current implementation

See https://github.com/sigstore/root-signing/blob/main/.github/workflows/sync-main-to-staging.yml

The main logical change is that the git repository now only contains the current metadata versions. This means that if the publishing mechanism wants to both delete old metadata and artifacts but also ensure old metadata & artifacts are available for clients to use for a period of time, it needs to manage that. In practice there should be no pressing need for a sigstore repository to delete old metadata & artifacts: there will only be small amount of new files per year in normal operation.

The technical changes are listed below:

  • runs as part of Publish workflow
  • Removed gcloud auth login: I don't know what the purpose was
  • Removed create_credentials_file: true from google-github-actions/auth: this is the default
  • Simplified special case for timestamp.json (it should be uploaded after all the other files)
  • Removed expiry checks: Publish workflow has already run a client smoke test that fails on expiry
  • Removed deletion of files from bucket: Some form of this can be re-implemented later but there should be no major need for it for decades (considering the usage pattern)
  • Removed failure handling: this is provided by Publish workflow that is the caller of this reusable workflow

@jku
Copy link
Member Author

jku commented Feb 13, 2024

@haydentherapper feel free to have a look already, I think this should be finished (but let's not merge yet).

@jku
Copy link
Member Author

jku commented Feb 13, 2024

Two obvious feature requests:

  • Automated testing for the GCS repository: I think this can be a separate feature
  • Use a GitHub deployment environment: This would allow us to control the final deployment with release reviews etc. We probably don't want more bureaucracy for staging so I don't think we need this right now (but this might be a separate feature to implement before production is moved to tuf-on-ci)

Copy link
Contributor

@haydentherapper haydentherapper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Only gap is figuring out the automation side of things, and probably splitting up deployments to pages and deployments to GCS (pages could always be automatic, GCS automatic just for regular ts/snapshot updates)

.github/workflows/deploy-to-gcs.yml Show resolved Hide resolved
.github/workflows/deploy-to-gcs.yml Show resolved Hide resolved
jku added 3 commits March 4, 2024 12:16
This uploads the repository to GCS and invalidates the CDN cache.

* the workflow is a custom one for root-signing
* it is called as a reusable workflow from the TUF-on-CI publish workflow
  (so gets automated issue failing on errors)

GCP configuration is mostly hard coded, only service accunt details are
inputs:
 * service account: same as online signing
 * project: projectsigstore-staging
 * GCS bucket: tuf-root-staging
 * CDN load balancer: tuf-repo-cdn-lb

Signed-off-by: Jussi Kukkonen <[email protected]>
rsync features:
* exclude list (used for timestamp)
* --delete-unmatched-destination-objects may be useful in future to
  delete unused metadata files

Note that gcloud rsync CLI is not the same as unix rsync: the SOURCE
and DEST arguments in particular function completely differently.

Also put some variables into actual named variables.

Signed-off-by: Jussi Kukkonen <[email protected]>
@jku
Copy link
Member Author

jku commented Mar 4, 2024

Rebased on main.

I'm marking this ready:

  • Tests pass on current repository, I've also tested manually with sigstore-python
  • Other signers have now been added (so my yubikey is no longer critical infra)
  • This PR has no release gating: both timestamps and other changes get published automatically once they pass the client tests. I believe this is fine for staging: we can work on Deployment gating #54 afterwards
  • quick risk analysis:
    • complete upload failure is not really an issue: currently published timestamp remains valid ~forever
    • the new timestamp is set to expire in 7 days so succeeding once and then failing later would be more of an issue
    • In general, we'll have ~4 days (length of the signing period) to resolve issues in staging. This is similar to production

@jku jku marked this pull request as ready for review March 4, 2024 10:23
@jku jku requested a review from haydentherapper March 4, 2024 17:22
Copy link
Contributor

@haydentherapper haydentherapper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Let's sync away.

If I were to write a premortem, maybe there's a permissions issue, but we're using a service account in production that has the same permissions and performs the same steps in GCS and CDN. I double checked the IAM binding is for repo:sigstore/root-signing-staging:ref:refs/heads/main.

@jku jku merged commit 6dffe30 into sigstore:main Mar 5, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Publish to staging GCS
2 participants