Can No Longer Connect to MongoDB ReplicaSet using K8s Port Forwarding

[ZachThill]

Name and Version

bitnami/mongodb:16.1.0

What architecture are you using?

amd64

What steps will reproduce the bug?

Deploy a fresh resplicaSet with externalAccess disabled, mTLS enabled, and using the same version of the bitnami mongodb helm chart that I have stated above. Run the following commands:

kubectl cp mongodb/mongodb-0:/certs/mongodb.pem ~/mongodb-client.pem
kubectl cp mongodb/mongodb-0:/certs/mongodb-ca-cert ~/mongodb-ca-cert
kubectl port-forward svc/mongodb-headless -n mongodb 27017:27017

Using MongoDB Compass, or some other mongodb client application, configure your connection string to localhost:27017, and try to connect.

Are you using any custom parameters or values?

global:
  storageClass: gp2

image:
  debug: true

clusterDomain: cluster.local

commonLabels:
  app: mongodb
commonAnnotations: {}

architecture: replicaset

replicaCount: 3
replicaSetHostnames: true

podLabels: {}
podAnnotations: {}

resources: {}

arbiter:
  enabled: true
  podLabels: {}
  podAnnotations: {}
  resources: {}

tls:
  enabled: true
  mTLS:
    enabled: true
  autoGenerated: true
  mode: requireTLS

automountServiceAccountToken: true

externalAccess:
  enabled: false
  service:
    type: LoadBalancer
    portName: "mongodb"
    ports:
      mongodb: 27017
    publicNames:
      - "test-mongodb-0-internal.my-domain.com"
      - "test-mongodb-1-internal.my-domain.com"
      - "test-mongodb-2-internal.my-domain.com"
    annotationsList:
      - external-dns.alpha.kubernetes.io/hostname: "test-mongodb-0-internal.my-domain.com"
      - external-dns.alpha.kubernetes.io/hostname: "test-mongodb-1-internal.my-domain.com"
      - external-dns.alpha.kubernetes.io/hostname: "test-mongodb-2-internal.my-domain.com"

podSecurityContext:
  enabled: true
  fsGroupChangePolicy: Always
  supplementalGroups: []
  fsGroup: 1001

containerSecurityContext:
  enabled: true
  seLinuxOptions: {}
  runAsUser: 1001
  runAsGroup: 1001
  runAsNonRoot: true
  privileged: false
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false

persistence:
  enabled: true
  # storageClass: ""      # Inherited from `global:` block
  accessModes:
    - ReadWriteOnce
  size: 100Gi
  annotations: {}
  mountPath: /bitnami/mongodb
  subPath: ""

persistentVolumeClaimRetentionPolicy:
  enabled: true
  whenScaled: Retain
  whenDeleted: Retain

serviceAccount:
  create: true
  name: mongodb
  annotations: {}
  automountServiceAccountToken: true

rbac:
  create: true
  rules:
    - apiGroups:
        - ""
      resources:
        - pods
      verbs:
        - get
        - list
        - watch
        - update

What is the expected behavior?

The expected behavior would be that I can access my unexposed database using kubectl port-forward.

What do you see instead?

I want to make it very clear that the replicaSet initializes perfectly and can be accessed by other client applications living within the same Kubernetes cluster because they can use the k8s internal DNS names created from the headless service. My problem is my lack of ability to access the MongoDB replicaSet via MongoDB Compass. Here are the screenshots of my connectivity errors when using either localhost or 127.0.0.1:

Looking into the error and viewing the /certs/openssl.conf file I see that localhost and 127.0.0.1 aren't being added to the SAN like it used to in previous versions of the helm chart:

$ cat certs/openssl.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = mongodb-headless
DNS.2 = mongodb-0
DNS.3 = mongodb-0.mongodb-headless.mongodb.svc.cluster.local
DNS.5 = test-mongodb-0-internal.spectruminternal.blue
DNS.6 = test-mongodb-1-internal.spectruminternal.blue
DNS.7 = test-mongodb-2-internal.spectruminternal.blue

For more context, we used to use version 15.1.4 of the Helm chart and the SAN would always have 127.0.0.1 and localhost, allowing users to access the replicaSet via a kubectl port-forward. I don't know if this was a change in the gen_certificates init container, but maybe something to look into.

[ZachThill]

Found the line that was removed from the generate-tls-certs.sh script:

charts/bitnami/mongodb/templates/common-scripts-cm.yaml

Line 86 in e364fc1

IP.1 = 127.0.0.1

[carrodher]

The latest changes in that file were implemented some months ago in a previous major version, are you able to reproduce the issue using a chart version containing that line?

[ZachThill]

@carrodher,
It appears that those IP lines were removed in version 15.3.0 of the helm chart. Using previous versions of the chart, everything is working fine. My question for this ticket is what was the reason for it being removed? Was it security related, thought to be unneeded, etc ..? And without it, how can I connect to the MongoDB replicaSet via my local machine with it being gone? Can we put it back?

[ZachThill]

@carrodher,
I figured out a workaround to my issue. I see in the git blame, that a person named @fmulero added optional flags to the gen_certificates bash script that allows grabs IPs and hostnames from the .externalAccess.service.loadBalancerIPs and tls.extraDnsNames arrays.

After adding localhost to the tls.extraDnsNames array, I was able to connect to the replicaSet using kubectl port-forward and MongoDB Compass.