-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decision Record: capi images mandatory #582
Draft
mxmxchere
wants to merge
5
commits into
main
Choose a base branch
from
require-k8s-images
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
--- | ||
title: Cluster-API images | ||
type: Decision Record | ||
status: Draft | ||
track: Iaas | ||
--- | ||
|
||
<!--- | ||
This is a template striving to provide a starting point for | ||
creating a decision record adhering to scs-0001. | ||
Replace at least all text in the sections not marked as OPTIONAL. | ||
See https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0001-v1-sovereign-cloud-standards.md | ||
---> | ||
|
||
## Abstract | ||
|
||
The problem that brings us here is how to update and keep track with upstream kubernetes versions when using the cluster-stacks approach on OpenStack. As the [SCS K8S Version Policy](https://docs.scs.community/standards/scs-0210-v2-k8s-version-policy#motivation) formulates, SCS is quite eager to keep track with the upstream work and does not want to fall behind: | ||
|
||
> We want to achieve an up-to-date policy, meaning that providers should be mostly in sync with the upstream and don't fall behind the official Kubernetes releases. | ||
|
||
## Context | ||
|
||
The following scenario: 1 CSP with 100 customers each deploying at least one cluster. | ||
One image will be downloaded (from upstream) and uploaded (to glance) and stored 99 times. | ||
|
||
## Decision | ||
|
||
Mandate the CSP to keep cluster-api Images up-to-date. | ||
|
||
From the meeting notes: | ||
|
||
- will save time, money, physical resources and power for both CSP and customer | ||
- would guarantee quality and timeliness of image | ||
- regardless of CSP taste, this KaaS tech is part of SCS | ||
- CSP won't have noticable disadvantages from providing the image | ||
|
||
This would lead to the following standard changes: | ||
Create a copy of the [default image list](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/scs-0104-v1-images.yaml) and change the capi block from "recommended" to "mandatory". | ||
|
||
Also, strictly speaking, we have to change [the standard image format definition](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0104-v1-standard-images.md#image-specification-class-of-images) so that "mandatory" for an image class does not mean that at least one image MUST be present in glance but all that match the [SCS K8S Version Policy](https://docs.scs.community/standards/scs-0210-v2-k8s-version-policy#motivation). Also the checking script has to be adapted. | ||
|
||
## Consequences | ||
|
||
CSPs will have the additional burden to provide cluster-api images in glance (and update them regularly). | ||
|
||
The CSP will save bandwitdth and diskspace when images are only downloaded and stored once, instead of per Customer. | ||
It will be easier and quicker for customers to update their clusters. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So let me get this right, as a CSP -who has no interest in offering k8s services based on CAPI- I'm forced to provide not one but how every many Images that are needed only in the scope of CAPI and keep them "maintained"?
I would rather like to not mix standards and keep the separation between IaaS, KaaS + Ops "clean".
We're already looking into how to align our Gardener based k8s Setup with the KaaS Standards and thus would follow the mandatory versions and thus at this point in time we see no benefit in accepting additional "mandatory" Images.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the feedback. Yes, the standard, in the above form, would force a CSP to maintain k8s CAPI images, even in a timely manner.
Maybe a compromise could be that this standard is only mandatory when your are not providing a SCS-compliant k8s service (which you plan to do, so this would not affect you). However that would also connect IaaS and KaaS layers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The idea behind this was to prepare an SCS-standardised infrastructure as well as possible for the use of SCS cluster stacks. A customer can use the SCS cluster stacks or cluster API in general completely independently of the Kubernetes as a Service preferred and offered by the CSP. If it is specified in the standard that a CSP must provide the cluster API images, then the SCS cluster stacks could be used without additional effort (providing an image for each project, which is definitely a disadvantage if you have to do that). I think it's basically the same as with the special flavors with local storage. Their existence was necessary due to the problems with Kubernetes managed by cluster API / SCS cluster stacks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the summary @berendt - this is exactly what I'd have written.
From my point of view the impact for the CSP is fairly low while having a large impact for the customer whose Cluster-API stuff simply works.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the local storage, there was nothing a user could do if they needed the performance. The images, however, can be uploaded by the user. Yes, this does require some additional effort and it might waste some time and energy and produce emissions etc., but it is possible. So from my POV it should be up to the CSP to decide whether they want to meet their customers halfway here or not (if they do, they can advertise as much). I don't see the need to make it mandatory.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for being late, but how are these images provided? Are they downloadable for the CSP? Or does the CSP have to create those images for the operated environments? That might be difficult for CSPs that don't use/provide cluster-api services - even to test those images...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At the moment we provide them at https://github.com/osism/k8s-capi-images. We do not test them at the moment. There is an issue open for this. We are currently depending on the feedback from the SCS K8s team for the tests. If there is no negative feedback from them, we assume that the images are working.
Building the images themselves is trivial with the https://github.com/kubernetes-sigs/image-builder/ .
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We've discussed this in today's PB again and the overall consensus was that these images should be RECOMMENDED and not MANDATORY.
This allows CSPs to follow the recommendation, without having the burden to maintain the images if they really don't want to carry them. Imho we should make sure the discussion is reflected in the decision record.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Recommended instead of mandatory is the status quo, from my point of view we need no decision record for that.
The only thing that can be "improved" in the standards is the concatenation between the SCS K8S Version Policy 0210-v2 and the recommended issues. But as these standards are neither binding nor would an additional concatenation document add any new informational value I see no point in doing that either. In the worst case we build in a contradiction somewhere and create confusion. So from my point of view we can just close this PR (unmerged)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mxmxchere I suggested the decision record so that the discussion is manifested outside of this PR. ;)