Test/Implement IRSA without kubermatic modifications

[nce]

No description provided.

[nce]

Native

Creating a oidc provider in aws is not possible with our kubermatic setup, as the k8s api is not reachable by :443 (but instead on a random hostport).

This renders this approach invalid:

+ resource "aws_iam_openid_connect_provider" "k8s" {
      + arn             = (known after apply)
      + client_id_list  = [
          + "sts.amazonaws.com",
        ]
      + id              = (known after apply)
      + tags_all        = {
          + "Name"    = "ops-k8s-bootstrap"
          + "Owner"   = "ops"
          + "project" = "ops-k8s-bootstrap"
        }
      + thumbprint_list = [
          + "24c8b10aaaeea99aaabcde33b4f80757baad9190",
        ]
      + url             = "https://xmxxxq.adorsys.kaas.cloudpunks.io:3xx4"
    }

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html#manage-oidc-provider-console:

For Provider URL, type the URL of the IdP. The URL must comply with these restrictions:

• The URL is case-sensitive.
• The URL must begin with https://.
• The URL should not contain a port number.

S3 as oidc provider

Another valid approach could be, that we don't rely on the API server to present the .well-known folder, but on a seperate s3 bucket:

This reflects the config, initially provided by the API server; Both files - in its current state:

.well-known/openid-configuration

{
  "issuer": "https://kubermatic-staging-irsa.s3.eu-central-1.amazonaws.com",
  "jwks_uri": "https://kubermatic-staging-irsa.s3.eu-central-1.amazonaws.com/keys.json",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ],
  "claims_supported": [
    "sub",
    "iss"
  ]
}

key.json

{
  "keys": [
    {
      "use": "sig",
      "kty": "RSA",
      "kid": "zfBdILL4MlE-PxAtrRC8RhC3LVCpNkJUQRVEfQPWaTg",
      "alg": "RS256",
      "n": "ucGNVznjPUPTYJalzIijaOnhdOAwF0BrBrSQXEBQs-bctuD7pTn-PT2axNejlydlDaDVmIY3YvBOq2_pITVGG2pCErrX-b7QK-IEBB4Jmz9aR6MgfzfUPbprOez-9iPplhydi9O0zjAZHM1FnbgK4LmSJ8qPKOVI5Qo-unwzfeHr8psp9nXUCgw5V0FvIHq-Jbd6-WBfugcEOIjwXzB9Ta0-P2Bv_zIXa7kUOn6RMP5hoWMC8Q7D0rrYt4CP8PjC-kCcq39ZU-N-JRqGYY5hCrfb7RaqvyErl_XLLGwplqDGVzOh0YHTdOzXdWBxkZEy9U7apz6RwSXQgQqDmKK-PQ",
      "e": "AQAB"
    }
  ]
}

With this setup & the current test.tf (in the irsa module). We get an error in sts get-caller-identity like

theres no valid oidc provider for kass.kubermatic.port

I guess that's because --service-account-issuer is set to the kubermatic url (maybe?)

[nce]

After several tickets we decided to not move forward on this. We'r currently on 2.20 which makes this tricky to configure. Apparently the LB needs to be set on a seed level which we can't risk.

There might be a customer facing webconfiguration option to set this (?) in 2.22.

We now need to remove poilcy/iam/s3