Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERROR: meshRef shouldn't be nil, please check webhook setup #532

Open
RafalMaleska opened this issue Nov 5, 2021 · 2 comments
Open

ERROR: meshRef shouldn't be nil, please check webhook setup #532

RafalMaleska opened this issue Nov 5, 2021 · 2 comments
Assignees
@sshver
Labels
bug Something isn't working needs investigation

Comments

@RafalMaleska
Copy link

RafalMaleska commented Nov 5, 2021 •
edited
Loading

Describe the bug

when creating (in this order):

  1. mesh with namespaceSelector: {}
  2. virtualgateway
  3. appmesh-controller

this happens:

  1. the controller-pod is up
  2. the mesh gets an arn
  3. the virtualgateway is stuck and the controller logs:
{
   "level": "debug",
   "ts": 1.63611661E9,
   "logger": "controller-runtime.webhook.webhooks",
   "msg": "received request",
   "webhook": "/validate-appmesh-k8s-aws-v1beta2-virtualgateway",
   "UID": "4077c412-6891-4cbb-b27f-e0bfe8b2ec05",
   "kind": "appmesh.k8s.aws/v1beta2, Kind=VirtualGateway",
   "resource": {
      "group": "appmesh.k8s.aws",
      "version": "v1beta2",
      "resource": "virtualgateways"
   }
}

and

{
   "level": "debug",
   "ts": 1.63611661E9,
   "logger": "validating_handler",
   "msg": "validating webhook request",
   "request": {
      "uid": "4077c412-6891-4cbb-b27f-e0bfe8b2ec05",
      "kind": {
         "group": "appmesh.k8s.aws",
         "version": "v1beta2",
         "kind": "VirtualGateway"
      },
      "resource": {
         "group": "appmesh.k8s.aws",
         "version": "v1beta2",
         "resource": "virtualgateways"
      },
      "requestKind": {
         "group": "appmesh.k8s.aws",
         "version": "v1beta2",
         "kind": "VirtualGateway"
      },
      "requestResource": {
         "group": "appmesh.k8s.aws",
         "version": "v1beta2",
         "resource": "virtualgateways"
      },
      "name": "appmesh-gateway-cnp-example-gshl6",
      "namespace": "appmesh-ingress",
      "operation": "UPDATE",
      "userInfo": {
         "username": "system:serviceaccount:appmesh-system:appmesh-controller",
         "uid": "57f1bf92-dbd6-4f9f-a4e8-dc7399904ce3",
         "groups": [
            "system:serviceaccounts",
            "system:serviceaccounts:appmesh-system",
            "system:authenticated"
         ],
         "extra": {
            "authentication.kubernetes.io/pod-name": [
               "appmesh-controller-cnp-example-gshl6-64c68db879-shv4r"
            ],
            "authentication.kubernetes.io/pod-uid": [
               "912d7996-8087-4dd5-8c67-02d2da72c314"
            ]
         }
      },
      "object": {
         "apiVersion": "appmesh.k8s.aws/v1beta2",
         "kind": "VirtualGateway",
         "metadata": {
            "annotations": {
               "meta.helm.sh/release-name": "appmesh-gateway-cnp-example-gshl6",
               "meta.helm.sh/release-namespace": "appmesh-ingress"
            },
            "creationTimestamp": "2021-11-05T12:40:59Z",
            "deletionGracePeriodSeconds": 0,
            "deletionTimestamp": "2021-11-05T12:51:07Z",
            "finalizers": [
               "finalizers.appmesh.k8s.aws/aws-appmesh-resources"
            ],
            "generation": 2,
            "labels": {
               "app.kubernetes.io/instance": "appmesh-gateway-cnp-example-gshl6",
               "app.kubernetes.io/managed-by": "Helm",
               "app.kubernetes.io/name": "appmesh-gateway",
               "app.kubernetes.io/version": "1.0.0",
               "helm.sh/chart": "appmesh-gateway-0.1.14"
            },
            "managedFields": [
               {
                  "apiVersion": "appmesh.k8s.aws/v1beta2",
                  "fieldsType": "FieldsV1",
                  "fieldsV1": {
                     "f:metadata": {
                        "f:annotations": {
                           ".": {},
                           "f:meta.helm.sh/release-name": {},
                           "f:meta.helm.sh/release-namespace": {}
                        },
                        "f:labels": {
                           ".": {},
                           "f:app.kubernetes.io/instance": {},
                           "f:app.kubernetes.io/managed-by": {},
                           "f:app.kubernetes.io/name": {},
                           "f:app.kubernetes.io/version": {},
                           "f:helm.sh/chart": {}
                        }
                     },
                     "f:spec": {
                        ".": {},
                        "f:backendDefaults": {
                           ".": {},
                           "f:clientPolicy": {
                              ".": {},
                              "f:tls": {
                                 ".": {},
                                 "f:enforce": {},
                                 "f:validation": {
                                    ".": {},
                                    "f:trust": {
                                       ".": {},
                                       "f:acm": {
                                          ".": {},
                                          "f:certificateAuthorityARNs": {}
                                       }
                                    }
                                 }
                              }
                           }
                        },
                        "f:listeners": {},
                        "f:logging": {
                           ".": {},
                           "f:accessLog": {
                              ".": {},
                              "f:file": {
                                 ".": {},
                                 "f:path": {}
                              }
                           }
                        },
                        "f:namespaceSelector": {},
                        "f:podSelector": {
                           ".": {},
                           "f:matchLabels": {
                              ".": {},
                              "f:app.kubernetes.io/name": {}
                           }
                        }
                     }
                  },
                  "manager": "crossplane-helm-provider",
                  "operation": "Update",
                  "time": "2021-11-05T12:40:59Z"
               },
               {
                  "apiVersion": "appmesh.k8s.aws/v1beta2",
                  "fieldsType": "FieldsV1",
                  "fieldsV1": {
                     "f:metadata": {
                        "f:finalizers": {
                           ".": {},
                           "v:\"finalizers.appmesh.k8s.aws/aws-appmesh-resources\"": {}
                        }
                     }
                  },
                  "manager": "controller",
                  "operation": "Update",
                  "time": "2021-11-05T12:41:06Z"
               }
            ],
            "name": "appmesh-gateway-cnp-example-gshl6",
            "namespace": "appmesh-ingress",
            "resourceVersion": "399940839",
            "uid": "466ea09f-7cd4-4d9e-83b9-2378dfc6e42c"
         },
         "spec": {
            "backendDefaults": {
               "clientPolicy": {
                  "tls": {
                     "enforce": true,
                     "validation": {
                        "trust": {
                           "acm": {
                              "certificateAuthorityARNs": [
                                 "arn:aws:acm-pca:eu-central-1:1234567890:certificate-authority/0dd3f599-5786-46df-9881-c9194147e3c6"
                              ]
                           }
                        }
                     }
                  }
               }
            },
            "listeners": [
               {
                  "healthCheck": {
                     "healthyThreshold": 3,
                     "intervalMillis": 5000,
                     "path": "/",
                     "protocol": "http",
                     "timeoutMillis": 5000,
                     "unhealthyThreshold": 5
                  },
                  "portMapping": {
                     "port": 8088,
                     "protocol": "http"
                  },
                  "tls": {
                     "certificate": {
                        "acm": {
                           "certificateARN": "arn:aws:acm:eu-central-1:1234567890:certificate/9bf6a78d-757c-4f22-b7fa-4b09107db7ca"
                        }
                     },
                     "mode": "STRICT"
                  }
               }
            ],
            "logging": {
               "accessLog": {
                  "file": {
                     "path": "/dev/stdout"
                  }
               }
            },
            "namespaceSelector": {},
            "podSelector": {
               "matchLabels": {
                  "app.kubernetes.io/name": "appmesh-gateway-cnp-example-gshl6"
               }
            }
         }
      },
      "oldObject": {
         "apiVersion": "appmesh.k8s.aws/v1beta2",
         "kind": "VirtualGateway",
         "metadata": {
            "annotations": {
               "meta.helm.sh/release-name": "appmesh-gateway-cnp-example-gshl6",
               "meta.helm.sh/release-namespace": "appmesh-ingress"
            },
            "creationTimestamp": "2021-11-05T12:40:59Z",
            "deletionGracePeriodSeconds": 0,
            "deletionTimestamp": "2021-11-05T12:51:07Z",
            "finalizers": [
               "finalizers.appmesh.k8s.aws/virtualgateway-members",
               "finalizers.appmesh.k8s.aws/aws-appmesh-resources"
            ],
            "generation": 2,
            "labels": {
               "app.kubernetes.io/instance": "appmesh-gateway-cnp-example-gshl6",
               "app.kubernetes.io/managed-by": "Helm",
               "app.kubernetes.io/name": "appmesh-gateway",
               "app.kubernetes.io/version": "1.0.0",
               "helm.sh/chart": "appmesh-gateway-0.1.14"
            },
            "name": "appmesh-gateway-cnp-example-gshl6",
            "namespace": "appmesh-ingress",
            "resourceVersion": "399940839",
            "uid": "466ea09f-7cd4-4d9e-83b9-2378dfc6e42c"
         },
         "spec": {
            "backendDefaults": {
               "clientPolicy": {
                  "tls": {
                     "enforce": true,
                     "validation": {
                        "trust": {
                           "acm": {
                              "certificateAuthorityARNs": [
                                 "arn:aws:acm-pca:eu-central-1:1234567890:certificate-authority/0dd3f599-5786-46df-9881-c9194147e3c6"
                              ]
                           }
                        }
                     }
                  }
               }
            },
            "listeners": [
               {
                  "healthCheck": {
                     "healthyThreshold": 3,
                     "intervalMillis": 5000,
                     "path": "/",
                     "protocol": "http",
                     "timeoutMillis": 5000,
                     "unhealthyThreshold": 5
                  },
                  "portMapping": {
                     "port": 8088,
                     "protocol": "http"
                  },
                  "tls": {
                     "certificate": {
                        "acm": {
                           "certificateARN": "arn:aws:acm:eu-central-1:1234567890:certificate/9bf6a78d-757c-4f22-b7fa-4b09107db7ca"
                        }
                     },
                     "mode": "STRICT"
                  }
               }
            ],
            "logging": {
               "accessLog": {
                  "file": {
                     "path": "/dev/stdout"
                  }
               }
            },
            "namespaceSelector": {},
            "podSelector": {
               "matchLabels": {
                  "app.kubernetes.io/name": "appmesh-gateway-cnp-example-gshl6"
               }
            }
         }
      },
      "dryRun": false,
      "options": {
         "kind": "UpdateOptions",
         "apiVersion": "meta.k8s.io/v1"
      }
   }
}

For some reason the VirtualGateway dosen't get it's meshRef.

Steps to reproduce

creating in this order within 1second:

  1. mesh with namespaceSelector: {}
  2. virtualgateway
  3. appmesh-controller

Expected outcome
virtualgateway gets an arn

Environment

  • App Mesh controller version 1.4.1
  • Envoy version 1.19.1.1
  • Are you using any integrations? X-ray, Jaeger etc. If so versions? ACM/CloudMap/IRSA
  • Kubernetes version 1.21
  • Using EKS (yes/no), if so version? Yes

Additiona-Info

Before the Error with the VirtualGateway I see the following errors:

{
   "level": "error",
   "ts": 1.63611827E9,
   "logger": "controller-runtime.controller",
   "msg": "Reconciler error",
   "controller": "mesh",
   "request": "/provider-in-cluster",
   "error": "Internal error occurred: failed calling webhook \"mmesh.appmesh.k8s.aws\": Post \"https://appmesh-controller-cnp-example-vnhms-webhook-service.appmesh-system.svc:443/mutate-appmesh-k8s-aws-v1beta2-mesh?timeout=10s\": dial tcp 100.64.240.144:9443: connect: connection refused",
   "stacktrace": "github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"
}
@RafalMaleska RafalMaleska added the bug Something isn't working label Nov 5, 2021
@RafalMaleska
Copy link
Author

a workaround that helped was to include

timeoutSeconds: 30

in the webhook configuration

@cgchinmay cgchinmay self-assigned this Nov 10, 2021
@cgchinmay cgchinmay assigned sshver and unassigned cgchinmay Nov 18, 2021
@ar4096
Copy link

ar4096 commented Jan 27, 2022

@sshver Any update on this ? @cgchinmay This seems like it won't be an easy fix ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sshver sshver

Labels
bug Something isn't working needs investigation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@cgchinmay @RafalMaleska @sshver @ar4096