✨ Allow to wire an object mutation handler #2932

alculquicondor · 2024-08-23T15:54:11Z

Proposal

Allow wiring an admission.Handler to process mutations in a webhook, as part of the builder.WebhookBuilder.

Motivation

Currently, mutations can be done through a CustomDefaulter which accepts and modifies a runtime.Object. The creation of patches is hidden from the developer. The jsonpatch is created from the difference of the request's raw yaml and the marshalled object modified by the CustomDefaulter.

This works for webhooks that are always released along with CRD changes. However, it doesn't work for webhooks written for CRDs from other projects. An example of this is Kueue, which writes webhooks for k8s Jobs, kubeflow, kuberay and others. Whenever there is a new version of the third party CRD, the hidden controller-runtime handler will produce a patch that removes all unknown fields.

A mechanism to fully override the handler allows developers to write patches that are safer against these version changes.

Goals

A mechanism to override jsonpatch generation for mutation webhooks.

Non Goals

A breaking-change in the libraries to build webhooks.
A breaking-change in the way patches are built when using a CustomDefaulter.
A mechanism to wire an admission.Handler to process validations. This could easily be added in a follow up proposal, but it doesn't seem to provide as much value as processing mutations.

Alternatives

My first proposal was #2931, which is to round-trip the raw object to get rid of unknown fields, and generate a jsondiff from this object.
However, @sbueringer pointed out that the round-trip might also add fields that didn't exist in the raw object is the go types are missing omitempty, which might lead to patches that can't be applied to the raw object in the apiserver. So the suggestion was to give full control to developers over the handler to build patches as they see fit.

alculquicondor · 2024-08-23T15:54:37Z

/assign @sbueringer

k8s-ci-robot · 2024-08-23T19:10:50Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: alculquicondor
Once this PR has been reviewed and has the lgtm label, please ask for approval from sbueringer. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/builder/webhook.go

sbueringer · 2024-09-02T13:53:50Z

pkg/webhook/admission/defaulter_custom.go

@@ -76,7 +88,14 @@ func (h *defaulterForType) Handle(ctx context.Context, req Request) Response {
 		return Errored(http.StatusBadRequest, err)
 	}

-	// Default the object
+	return h.handler.Handle(ctx, req.Object.Raw, obj)


If we're going to introduce another way to implement mutation, I wonder if we should go all the way:

Should we just hand over ctx and the entire request? (~ l.83)

Should we call this interface Mutator & the builder func WithMutator

I think then it would also work well/better for resource updates

@vincepri @alvaroaleman Opinions?

I wanted to avoid doing h.decoder.Decode(req, obj) every time, but sure, better leave everything up to the developer.

Sorry, I don't quite follow how the new thing helps with the problem at hand. The problem at hand is you need the raw request or not? Wouldn't it be better to change the existing CustomDefaulter interface to add it as an arg in there?

Yes, I do need the raw request. But this is not the biggest problem, because the request is embedded in the context.

But the reason why I need the raw request is to be able to return json patches. And the CustomDefaulter currently just expects changes to the object. The creation of patches is completely hidden from the defaulter.

We could change the interface to return patches, but that's a breaking change. Isn't it better to add a fully manual mechanism to building webhook handlers?

My thinking was that we can lose information during json.Unmarshal and json.Marshal (either if the CRDs are not prefectly designed with omitempty, or if we have the wrong version of the CRD because it's not ours). This can lead to incorrect / unintended patches

So I thought if we provide a low-level way to handle this better, let's go all the way and give folks all the info (req) and allow them to return the patches directly

Additional context here:

🐛 Fix custom defaulter: avoid deleting unknown fields #2931 (comment)

🐛 Fix custom defaulter: avoid deleting unknown fields #2931 (comment)

Somewhat related issue #2800

Would this work: #2800 (comment) ?

alculquicondor · 2024-09-03T20:20:20Z

The panic seems to be on a different suite

/retest

alculquicondor · 2024-09-04T16:17:59Z

/hold cancel

alculquicondor · 2024-09-05T13:45:05Z

@sbueringer I addressed all the comments, PTAL.

alculquicondor · 2024-09-10T14:36:19Z

Can we move this forward? In Kueue, we are risking dropping fields all the time because we don't control the versions of the CRDs that we have webhooks for.

alvaroaleman · 2024-09-10T14:47:05Z

Can we move this forward? In Kueue, we are risking dropping fields all the time because we don't control the versions of the CRDs that we have webhooks for.

I understand, but c-r is a widely used project and this is an external interface change. Please have some patience until we find time to properly review it.

sbueringer · 2024-09-10T14:52:27Z

Just to be clear. This can be implemented entirely outside of controller-runtime.

alculquicondor · 2024-09-10T15:02:01Z

This can be implemented entirely outside of controller-runtime.

I understand that, but it's a lot of repetitive code to maintain.

I was hoping that the change was not controversial, but if you don't like the high level idea, please let me know so we can propose other alternatives. Or just implement the whole thing in Kueue.

sbueringer · 2024-09-10T15:04:49Z

If you are under pressure to implement a fix now in Kueue, I would highly recommend to implement an alternative on Kueue side. Otherwise we need some time until folks have time to review this PR.

None of us is paid to maintain controller-runtime. So we are working on CR when we find the time

vincepri · 2024-09-11T05:05:04Z

Having read through this discussion, the best path forward imo would be a small proposal that outlines the goals/non-goals and the problem we're trying to solve for all Controller Runtime users, not just this specific use case or what the current code does.

alculquicondor · 2024-09-11T13:06:07Z

See updated description.

vincepri

@alculquicondor Thanks for the description and extra context. The problem statement makes sense to me, although the proposed solution bring in a lot of questions for most users of this library.

The provided goal A mechanism to wire an admission.Handler to process mutations is an implementation detail, rather an an actual goal. Also to note, our design proposals are collected in the root repository and discussed separately from the actual implementation.

vincepri · 2024-09-11T13:49:05Z

pkg/builder/webhook_test.go

+func mutatorFactoryForTestDefaulter(scheme *runtime.Scheme) admission.HandlerFactory {
+	return func(obj runtime.Object, _ admission.Decoder) admission.Handler {
+		return admission.WithCustomDefaulter(scheme, obj, &TestCustomDefaulter{}).Handler
+	}
+}


This is very confusing. We're using a mutator, but returning a custom defaulter handler?

This is proving that you can implement a custom defaulter using the mutationhandler.

I could add a completely different handler, but I didn't want to spend too much time in the test before getting green light on the implementation approach.

I moved this somewhere else, but I'm still using a CustomDefaulter. Let me know if you prefer adding a different handler.

I added a custom handler.

vincepri · 2024-09-11T13:49:59Z

pkg/builder/webhook.go

+// WithMutatorFactory takes an admission.HandlerFactory, a MutatingWebhook will be wired for the handler that this factory creates.
+func (blder *WebhookBuilder) WithMutatorFactory(factory admission.HandlerFactory) *WebhookBuilder {
+	blder.mutatorFactory = factory
+	return blder
+}


Adding this makes the WithDefaulter a noop, which doesn't return an error and it's a misleading UX

Since this function doesn't return an error, I'll leave it to the Complete function to report the error. Unless you are ok with a panic?

vincepri · 2024-09-11T13:50:34Z

pkg/webhook/admission/webhook.go

+// WithHandlerFactory creates a new Webhook for a handler factory.
+func WithHandlerFactory(scheme *runtime.Scheme, obj runtime.Object, factory HandlerFactory) *Webhook {


Why is this called Factory?

Because it knows how to create a handler. It allows you to receive a decoder from the WebhookBuilder.

Another option could be just to accept a Handler (renaming the method to WithMutationHandler) and pass the decoder through the context.

I decided to remove the factory and leave it to the implementation to decide how to build the object.

pkg/builder/webhook.go

alculquicondor · 2024-09-11T14:09:02Z

Also to note, our design proposals are collected in the root repository and discussed separately from the actual implementation.

Up to you if you prefer me to write it there. This change seems so minimal that it shouldn't need such a permanent location for the design.

alculquicondor · 2024-09-24T20:54:35Z

There seems to be a difference in how the logger is set up. I'm investigating.

alculquicondor · 2024-09-25T17:02:36Z

/hold cancel

This is ready for review now

alculquicondor · 2024-09-26T16:56:11Z

/retest

mimowo

LGTM

pkg/builder/webhook_test.go

alvaroaleman

So if I understand it correctly, the root of the issue is a mismatch between the go types in the project and what is configured as the CRD, resulting in the patch removing fields it doesn't know about.

Can you use unstructured.Unstructured in your CustomDefaulter to avoid this?

alvaroaleman · 2024-09-29T16:09:40Z

pkg/builder/webhook.go

@@ -65,6 +66,12 @@ func (blder *WebhookBuilder) For(apiType runtime.Object) *WebhookBuilder {
 	return blder
 }

+// WithMutationHandler takes an admission.Handler, a MutatingWebhook will be wired for it.
+func (blder *WebhookBuilder) WithMutationHandler(h admission.Handler) *WebhookBuilder {


Its super not obvious when and why to use this versus the defaulter and almost guaranteed to lead to questions

Can we leave it as "more advanced use cases where you want control over the construction of patches"?

Or we could explicitly add the use case of go types that might be in a different version than the CRD.

alvaroaleman · 2024-09-29T16:16:46Z

How about #2800 (comment) instead?

alculquicondor · 2024-09-30T13:16:51Z

Can you use unstructured.Unstructured in your CustomDefaulter to avoid this?

I don't think so. I need to modify either the raw object or the returning patches.

How about #2800 (comment) instead?

That was my first proposal, which leads to different problems.

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 23, 2024

k8s-ci-robot requested review from joelanford and varshaprasad96 August 23, 2024 15:54

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 23, 2024

k8s-ci-robot assigned sbueringer Aug 23, 2024

alculquicondor force-pushed the patch_defaulter branch from ab06f04 to 922ee77 Compare August 23, 2024 19:11

troy0820 reviewed Aug 23, 2024

View reviewed changes

pkg/builder/webhook.go Outdated Show resolved Hide resolved

alculquicondor mentioned this pull request Aug 26, 2024

🐛 Fix custom defaulter: avoid deleting unknown fields #2931

Closed

sbueringer reviewed Sep 2, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 2, 2024

alculquicondor force-pushed the patch_defaulter branch 2 times, most recently from 5fd81ad to cd7afc5 Compare September 3, 2024 19:25

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 3, 2024

alculquicondor force-pushed the patch_defaulter branch 3 times, most recently from df4be17 to 56c3467 Compare September 3, 2024 20:03

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 4, 2024

alculquicondor force-pushed the patch_defaulter branch from 56c3467 to a4f023c Compare September 4, 2024 16:09

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 4, 2024

vincepri reviewed Sep 11, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 17, 2024

alculquicondor mentioned this pull request Sep 24, 2024

Drop remove operations from webhook patches kubernetes-sigs/kueue#3132

Merged

Allow to wire a mutation handler

ea977b9

alculquicondor force-pushed the patch_defaulter branch 2 times, most recently from 44738ee to d48b844 Compare September 24, 2024 19:35

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 24, 2024

alculquicondor mentioned this pull request Sep 25, 2024

Remove webhook builder when controller-runtime allows overriding a mutation handler kubernetes-sigs/kueue#3137

Open

alculquicondor force-pushed the patch_defaulter branch from d48b844 to a375c55 Compare September 25, 2024 16:58

Simplify design and error on multiple mutation mechanisms

50c2afb

alculquicondor force-pushed the patch_defaulter branch from a375c55 to 50c2afb Compare September 26, 2024 16:57

mimowo reviewed Sep 27, 2024

View reviewed changes

pkg/builder/webhook_test.go Outdated Show resolved Hide resolved

revert formatting

ed242f4

alvaroaleman reviewed Sep 29, 2024

View reviewed changes

		// WithHandlerFactory creates a new Webhook for a handler factory.
		func WithHandlerFactory(scheme runtime.Scheme, obj runtime.Object, factory HandlerFactory) Webhook {

✨ Allow to wire an object mutation handler #2932

Are you sure you want to change the base?

✨ Allow to wire an object mutation handler #2932

Conversation

alculquicondor commented Aug 23, 2024 • edited Loading

Proposal

Motivation

Goals

Non Goals

Alternatives

alculquicondor commented Aug 23, 2024

k8s-ci-robot commented Aug 23, 2024

sbueringer Sep 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbueringer Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor commented Sep 3, 2024

alculquicondor commented Sep 4, 2024

alculquicondor commented Sep 5, 2024

alculquicondor commented Sep 10, 2024

alvaroaleman commented Sep 10, 2024

sbueringer commented Sep 10, 2024

alculquicondor commented Sep 10, 2024

sbueringer commented Sep 10, 2024

vincepri commented Sep 11, 2024

alculquicondor commented Sep 11, 2024

vincepri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor commented Sep 11, 2024

alculquicondor commented Sep 24, 2024

alculquicondor commented Sep 25, 2024

alculquicondor commented Sep 26, 2024

mimowo left a comment

Choose a reason for hiding this comment

alvaroaleman left a comment

Choose a reason for hiding this comment

alvaroaleman Sep 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alvaroaleman commented Sep 29, 2024

alculquicondor commented Sep 30, 2024

alculquicondor commented Aug 23, 2024 •

edited

Loading

sbueringer Sep 2, 2024 •

edited

Loading

sbueringer Sep 9, 2024 •

edited

Loading

alvaroaleman Sep 29, 2024 •

edited

Loading