Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom ReplicaSet Config Failing to be Created #30086

Closed
ZachThill opened this issue Oct 25, 2024 · 6 comments
Closed

Custom ReplicaSet Config Failing to be Created #30086

ZachThill opened this issue Oct 25, 2024 · 6 comments
Assignees
@carrodher
Labels
mongodb solved tech-issues The user has a technical issue about an application triage Triage is needed

Comments

@ZachThill
Copy link

ZachThill commented Oct 25, 2024 •
edited by carrodher
Loading

Name and Version

bitnami/mongodb:16.1.0

What architecture are you using?

amd64

What steps will reproduce the bug?

  1. Use my custom values file defined below and update the go template variables to your liking.
  2. Deploy it using version 16.1.0 of the bitnami/mongodb helm chart.

Are you using any custom parameters or values?

Custom Values File

global:
  storageClass: gp2

clusterDomain: cluster.local

commonLabels:
  app: mongodb
commonAnnotations: {}

architecture: replicaset

replicaCount: 3
replicaSetHostnames: true

replicaSetConfigurationSettings:
  enabled: {{ .Values.cluster.eks.add_ons.mongodb.externalAccess.enabled }}
  configuration:
    members[0]._id: 0
    members[0].host: "{{ .Values.cluster.name }}-mongodb-0-internal.{{ .Values.cluster.hosted_zone.name }}:27017"
    members[0].priority: 5
    members[1]._id: 1
    members[1].host: "{{ .Values.cluster.name }}-mongodb-1-internal.{{ .Values.cluster.hosted_zone.name }}:27017"
    members[1].priority: 1
    members[2]._id: 2
    members[2].host: "{{ .Values.cluster.name }}-mongodb-2-internal.{{ .Values.cluster.hosted_zone.name }}:27017"
    members[2].priority: 1
    members[3]._id: 3
    members[3].host: mongodb-arbiter-0.mongodb-arbiter-headless.mongodb.svc.cluster.local:27017
    members[3].arbiterOnly: true
    members[3].buildIndexes: true
    members[3].hidden: true
    members[3].priority: 0

podLabels: {}
podAnnotations: {}

resources: {{ .Values.cluster.eks.add_ons.mongodb.resources }}

arbiter:
  enabled: true
  podLabels: {}
  podAnnotations: {}
  resources: {{ .Values.cluster.eks.add_ons.mongodb.arbiter.resources }}

tls:
  enabled: true
  mTLS:
    enabled: true
  autoGenerated: true
  mode: requireTLS

automountServiceAccountToken: true

externalAccess:
  enabled: {{ .Values.cluster.eks.add_ons.mongodb.externalAccess.enabled }}
  service:
    type: LoadBalancer
    portName: "mongodb"
    ports:
      mongodb: 27017
    publicNames:
      - "{{ .Values.cluster.name }}-mongodb-0-internal.{{ .Values.cluster.hosted_zone.name }}"
      - "{{ .Values.cluster.name }}-mongodb-1-internal.{{ .Values.cluster.hosted_zone.name }}"
      - "{{ .Values.cluster.name }}-mongodb-2-internal.{{ .Values.cluster.hosted_zone.name }}"
    annotationsList:
      - external-dns.alpha.kubernetes.io/hostname: "{{ .Values.cluster.name }}-mongodb-0-internal.{{ .Values.cluster.hosted_zone.name }}"
      - external-dns.alpha.kubernetes.io/hostname: "{{ .Values.cluster.name }}-mongodb-1-internal.{{ .Values.cluster.hosted_zone.name }}"
      - external-dns.alpha.kubernetes.io/hostname: "{{ .Values.cluster.name }}-mongodb-2-internal.{{ .Values.cluster.hosted_zone.name }}"

persistence:
  enabled: true
  # storageClass: ""      # Inherited from `global:` block
  accessModes:
    - ReadWriteOnce
  size: {{ .Values.cluster.eks.add_ons.mongodb.persistentVolumeSize }}
  annotations: {}
  mountPath: /bitnami/mongodb
  subPath: ""

persistentVolumeClaimRetentionPolicy:
  enabled: true
  whenScaled: Retain
  whenDeleted: Retain

serviceAccount:
  create: true
  name: mongodb
  annotations: {}
  automountServiceAccountToken: true

rbac:
  create: true
  rules:
    - apiGroups:
        - ""
      resources:
        - pods
      verbs:
        - get
        - list
        - watch
        - update

What is the expected behavior?

The rs.conf will be created with the given configuration and allow me to access my replicaSet without needing to specify a direct connection.

What do you see instead?

When first initializing the replicaSet, the pods seem to startup and pass their readiness probes just fine but restart soon after due to there being no configuration I presume. When execing into the primary pod and accessing the DB, I see the following error when trying to view the rs.conf():

$ mongosh admin --host "mongodb-0.mongodb-headless.mongodb.svc.cluster.local" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD --tls --tlsCertificateKeyFile=/certs/mongodb.pem --tlsCAFile=/certs/mongodb-ca-cert
Current Mongosh Log ID:	671be19dc5cc31c3b8fe6910
Connecting to:		mongodb://<credentials>@mongodb-0.mongodb-headless.mongodb.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&tls=true&tlsCertificateKeyFile=%2Fcerts%2Fmongodb.pem&tlsCAFile=%2Fcerts%2Fmongodb-ca-cert&appName=mongosh+2.3.2
Using MongoDB:		8.0.1
Using Mongosh:		2.3.2

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

------
   The server generated these startup warnings when booting
   2024-10-25T17:38:21.666+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2024-10-25T17:38:22.555+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2024-10-25T17:38:22.555+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2024-10-25T17:38:22.555+00:00: We suggest setting the contents of sysfsFile to 0.
   2024-10-25T17:38:22.555+00:00: Your system has glibc support for rseq built in, which is not yet supported by tcmalloc-google and has critical performance implications. Please set the environment variable GLIBC_TUNABLES=glibc.pthread.rseq=0
   2024-10-25T17:38:22.555+00:00: vm.max_map_count is too low
------

admin> rs.conf()
MongoServerError[NotYetInitialized]: no replset config has been received

Reading the pod logs doesn't help at all because the /scripts/replicaSetConfigurationSettings.sh bash script responsible for setting up this configuration doesn't seem to output anything in the pod logs.

However, when concatenating the /scripts/replicaSetConfigurationSettings.sh bash script, I see that my custom configs are being passed at runtime:

# load Values.replicaSetConfigurationSettings.configuration into associtive array which makes iterating and string manipulation easy
declare -A desiredRsConf
desiredRsConf[members[0]._id]='0'
desiredRsConf[members[0].host]='mongodb-test-dev-mongodb-0-internal.spectruminternal.blue:27017'
desiredRsConf[members[0].priority]='5'
desiredRsConf[members[1]._id]='1'
desiredRsConf[members[1].host]='mongodb-test-dev-mongodb-1-internal.spectruminternal.blue:27017'
desiredRsConf[members[1].priority]='1'
desiredRsConf[members[2]._id]='2'
desiredRsConf[members[2].host]='mongodb-test-dev-mongodb-2-internal.spectruminternal.blue:27017'
desiredRsConf[members[2].priority]='1'
desiredRsConf[members[3]._id]='3'
desiredRsConf[members[3].arbiterOnly]='true'
desiredRsConf[members[3].buildIndexes]='true'
desiredRsConf[members[3].hidden]='true'
desiredRsConf[members[3].host]='mongodb-arbiter-0.mongodb-arbiter-headless.mongodb.svc.cluster.local:27017'
desiredRsConf[members[3].priority]='0'

My guess as to why this is failing, may be because I'm trying to configure the member[X].host values? The reason I'm doing this at all is because I cannot access all the nodes in the replicaSet EXTERNALLY unless the values are updated to match their advertised hostnames (externalAccess.service.publicNames). Without this, I'm only able to connect via direct connections which eliminates the whole purpose of deploying a replicaSet at all.

Additional information

If there is any way someone could help me either figure out how to connect to the replicaSet externally without needing to update the rs.conf, or how to stop the error provided above that would be much appreciated.
@ZachThill ZachThill added the tech-issues The user has a technical issue about an application label Oct 25, 2024
@github-actions github-actions bot added the triage Triage is needed label Oct 25, 2024
@ZachThill
Copy link
Author

ZachThill commented Oct 25, 2024 •
edited by carrodher
Loading

UPDATE:

Tried running the /scripts/replicaSetConfigurationSettings.sh bash script manually to see what the logger function would output:

$ . ./scripts/replicaSetConfigurationSettings.sh
MongoServerSelectionError: Hostname/IP does not match certificate's altnames: IP: 127.0.0.1 is not in the cert's list:
replicaSetConfigurationSettings.sh -- not yet able to read rs.conf settings from the currently running rs (after 1 attempts)
MongoServerSelectionError: Hostname/IP does not match certificate's altnames: IP: 127.0.0.1 is not in the cert's list:
replicaSetConfigurationSettings.sh -- not yet able to read rs.conf settings from the currently running rs (after 2 attempts)
MongoServerSelectionError: Hostname/IP does not match certificate's altnames: IP: 127.0.0.1 is not in the cert's list:
replicaSetConfigurationSettings.sh -- not yet able to read rs.conf settings from the currently running rs (after 3 attempts)
MongoServerSelectionError: Hostname/IP does not match certificate's altnames: IP: 127.0.0.1 is not in the cert's list:
replicaSetConfigurationSettings.sh -- not yet able to read rs.conf settings from the currently running rs (after 4 attempts)
MongoServerSelectionError: Hostname/IP does not match certificate's altnames: IP: 127.0.0.1 is not in the cert's list:
replicaSetConfigurationSettings.sh -- not yet able to read rs.conf settings from the currently running rs (after 5 attempts)

Ran into the error above indefinitely. Here is the SAN for the certificate generated via the init-container:

$ cat certs/openssl.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = mongodb-headless
DNS.2 = mongodb-0
DNS.3 = mongodb-0.mongodb-headless.mongodb.svc.cluster.local
DNS.5 = mongodb-test-dev-mongodb-0-internal.spectruminternal.blue
DNS.6 = mongodb-test-dev-mongodb-1-internal.spectruminternal.blue
DNS.7 = mongodb-test-dev-mongodb-2-internal.spectruminternal.blue

I remember that an older version of the chart would add both 127.0.0.1 and the IPv4 address of the LB to the SAN as well. Maybe this was removed for security purposes, or only added when using auto-discovery?

@ZachThill
Copy link
Author

ZachThill commented Oct 28, 2024 •
edited
Loading

This is probably pretty stupid and I don't know if this will help, but asking ChatGPT, it says that the values defined in the advertised hostnames, and rs.conf hosts need to match in order for clients to be able to communicate with the entire replicaSet (not just a single direct connection to the current primary node). After updating the rs.conf manually to match the FQDNs defined in the externalAccess: block (i.e. the advertised hostnames), I was able to connect to the entire replicaSet. With this said, this brings me to the following question:

Why doesn't the helm chart and entrypoint bash scripts configure this out the gate when external access is enabled? The chart has init containers that run to verify that everything is setup beforehand anyway, so even if we change the default rs.conf the replicaSet should be able to initialize/communicate internally with itself.

@carrodher
Copy link
Member

Thank you for bringing this issue to our attention. We appreciate your involvement! If you're interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

@ZachThill
Copy link
Author

ZachThill commented Oct 28, 2024 •
edited by carrodher
Loading

@carrodher,
Thank you for taking a look into my issue. Sorry for jumping around all over the place. After doing some more looking, I enabled debugging on the image and have the following logs:

Pod Logs when enabling debug

$ kl mongodb-0
Defaulted container "mongodb" out of: mongodb, dns-check (init), log-dir (init), generate-tls-certs (init)
mongodb 20:13:44.57 INFO  ==> Advertised Hostname: mongodb-test-dev-mongodb-0-internal.spectruminternal.blue
mongodb 20:13:44.57 INFO  ==> Advertised Port: 27017
realpath: /bitnami/mongodb/data/db: No such file or directory
mongodb 20:13:44.57 INFO  ==> Data dir empty, checking if the replica set already exists
MongoNetworkError: getaddrinfo ENOTFOUND mongodb-1.mongodb-headless.mongodb.svc.cluster.local
mongodb 20:13:44.94 INFO  ==> Pod name matches initial primary pod name, configuring node as a primary
mongodb 20:13:44.95 INFO  ==>
mongodb 20:13:44.95 INFO  ==> Welcome to the Bitnami mongodb container
mongodb 20:13:44.95 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
mongodb 20:13:44.96 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
mongodb 20:13:44.96 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
mongodb 20:13:44.96 INFO  ==>
mongodb 20:13:44.96 DEBUG ==> Copying files from /opt/bitnami/mongodb/conf.default to /opt/bitnami/mongodb/conf
mongodb 20:13:44.96 INFO  ==> ** Starting MongoDB setup **
mongodb 20:13:44.97 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 20:13:45.02 INFO  ==> Initializing MongoDB...
mongodb 20:13:45.07 INFO  ==> Writing keyfile for replica set authentication...
mongodb 20:13:45.07 INFO  ==> Deploying MongoDB from scratch...
mongodb 20:13:45.08 DEBUG ==> Starting MongoDB in background...
{"t":{"$date":"2024-10-28T20:13:45.109Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serverParameters":{"enableLocalhostAuthBypass":{"default":true,"value":true}}}}
about to fork child process, waiting until server is ready for connections.
forked process: 100
child process started successfully, parent exiting
MongoNetworkError: getaddrinfo ENOTFOUND mongodb-test-dev-mongodb-0-internal.spectruminternal.blue
mongodb 20:13:46.17 INFO  ==> Creating users...
mongodb 20:13:46.17 INFO  ==> Creating root user...
Current Mongosh Log ID:	671ff07a08990d919cfe6910
Connecting to:		mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fcerts%2Fmongodb.pem&tlsCAFile=%2Fcerts%2Fmongodb-ca-cert&tlsAllowInvalidHostnames=true&appName=mongosh+2.3.2
Using MongoDB:		8.0.1
Using Mongosh:		2.3.2

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.

test> { ok: 1 }
mongodb 20:13:46.85 INFO  ==> Creating user 'DB_Admin'...
test> Current Mongosh Log ID:	671ff07b1d2a580c94fe6910
Connecting to:		mongodb://<credentials>@127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fcerts%2Fmongodb.pem&tlsCAFile=%2Fcerts%2Fmongodb-ca-cert&tlsAllowInvalidHostnames=true&appName=mongosh+2.3.2
Using MongoDB:		8.0.1
Using Mongosh:		2.3.2

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

------
   The server generated these startup warnings when booting
   2024-10-28T20:13:45.159+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2024-10-28T20:13:45.822+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2024-10-28T20:13:45.822+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2024-10-28T20:13:45.822+00:00: We suggest setting the contents of sysfsFile to 0.
   2024-10-28T20:13:45.822+00:00: Your system has glibc support for rseq built in, which is not yet supported by tcmalloc-google and has critical performance implications. Please set the environment variable GLIBC_TUNABLES=glibc.pthread.rseq=0
   2024-10-28T20:13:45.822+00:00: vm.max_map_count is too low
------

test> { ok: 1 }
mongodb 20:13:47.51 INFO  ==> Users created
mongodb 20:13:47.52 INFO  ==> Enabling authentication...
mongodb 20:13:47.54 INFO  ==> Configuring MongoDB replica set...
mongodb 20:13:47.54 INFO  ==> Stopping MongoDB...
mongodb 20:13:48.55 DEBUG ==> Starting MongoDB in background...
test> {"t":{"$date":"2024-10-28T20:13:48.582Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serverParameters":{"enableLocalhostAuthBypass":{"default":true,"value":false}}}}
about to fork child process, waiting until server is ready for connections.
forked process: 272
child process started successfully, parent exiting
MongoServerSelectionError: Server selection timed out after 30000 ms
mongodb 20:14:19.95 INFO  ==> Configuring MongoDB primary node

$ kl mongodb-0  --follow
Defaulted container "mongodb" out of: mongodb, dns-check (init), log-dir (init), generate-tls-certs (init)
mongodb 20:13:44.57 INFO  ==> Advertised Hostname: mongodb-test-dev-mongodb-0-internal.spectruminternal.blue
mongodb 20:13:44.57 INFO  ==> Advertised Port: 27017
realpath: /bitnami/mongodb/data/db: No such file or directory
mongodb 20:13:44.57 INFO  ==> Data dir empty, checking if the replica set already exists
MongoNetworkError: getaddrinfo ENOTFOUND mongodb-1.mongodb-headless.mongodb.svc.cluster.local
mongodb 20:13:44.94 INFO  ==> Pod name matches initial primary pod name, configuring node as a primary
mongodb 20:13:44.95 INFO  ==>
mongodb 20:13:44.95 INFO  ==> Welcome to the Bitnami mongodb container
mongodb 20:13:44.95 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
mongodb 20:13:44.96 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
mongodb 20:13:44.96 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
mongodb 20:13:44.96 INFO  ==>
mongodb 20:13:44.96 DEBUG ==> Copying files from /opt/bitnami/mongodb/conf.default to /opt/bitnami/mongodb/conf
mongodb 20:13:44.96 INFO  ==> ** Starting MongoDB setup **
mongodb 20:13:44.97 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 20:13:45.02 INFO  ==> Initializing MongoDB...
mongodb 20:13:45.07 INFO  ==> Writing keyfile for replica set authentication...
mongodb 20:13:45.07 INFO  ==> Deploying MongoDB from scratch...
mongodb 20:13:45.08 DEBUG ==> Starting MongoDB in background...
{"t":{"$date":"2024-10-28T20:13:45.109Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serverParameters":{"enableLocalhostAuthBypass":{"default":true,"value":true}}}}
about to fork child process, waiting until server is ready for connections.
forked process: 100
child process started successfully, parent exiting
MongoNetworkError: getaddrinfo ENOTFOUND mongodb-test-dev-mongodb-0-internal.spectruminternal.blue
mongodb 20:13:46.17 INFO  ==> Creating users...
mongodb 20:13:46.17 INFO  ==> Creating root user...
Current Mongosh Log ID:	671ff07a08990d919cfe6910
Connecting to:		mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fcerts%2Fmongodb.pem&tlsCAFile=%2Fcerts%2Fmongodb-ca-cert&tlsAllowInvalidHostnames=true&appName=mongosh+2.3.2
Using MongoDB:		8.0.1
Using Mongosh:		2.3.2

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.

test> { ok: 1 }
mongodb 20:13:46.85 INFO  ==> Creating user 'DB_Admin'...
test> Current Mongosh Log ID:	671ff07b1d2a580c94fe6910
Connecting to:		mongodb://<credentials>@127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fcerts%2Fmongodb.pem&tlsCAFile=%2Fcerts%2Fmongodb-ca-cert&tlsAllowInvalidHostnames=true&appName=mongosh+2.3.2
Using MongoDB:		8.0.1
Using Mongosh:		2.3.2

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

------
   The server generated these startup warnings when booting
   2024-10-28T20:13:45.159+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2024-10-28T20:13:45.822+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2024-10-28T20:13:45.822+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2024-10-28T20:13:45.822+00:00: We suggest setting the contents of sysfsFile to 0.
   2024-10-28T20:13:45.822+00:00: Your system has glibc support for rseq built in, which is not yet supported by tcmalloc-google and has critical performance implications. Please set the environment variable GLIBC_TUNABLES=glibc.pthread.rseq=0
   2024-10-28T20:13:45.822+00:00: vm.max_map_count is too low
------

test> { ok: 1 }
mongodb 20:13:47.51 INFO  ==> Users created
mongodb 20:13:47.52 INFO  ==> Enabling authentication...
mongodb 20:13:47.54 INFO  ==> Configuring MongoDB replica set...
mongodb 20:13:47.54 INFO  ==> Stopping MongoDB...
mongodb 20:13:48.55 DEBUG ==> Starting MongoDB in background...
test> {"t":{"$date":"2024-10-28T20:13:48.582Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serverParameters":{"enableLocalhostAuthBypass":{"default":true,"value":false}}}}
about to fork child process, waiting until server is ready for connections.
forked process: 272
child process started successfully, parent exiting
MongoServerSelectionError: Server selection timed out after 30000 ms
mongodb 20:14:19.95 INFO  ==> Configuring MongoDB primary node
mongodb 20:14:50.78 WARN  ==> Problem initiating replica set
            request: rs.initiate({"_id":"rs0", "members":[{"_id":0,"host":"mongodb-test-dev-mongodb-0-internal.spectruminternal.blue:27017","priority":5}]})
            response: Current Mongosh Log ID:	671ff09c730f3702adfe6910
Connecting to:		mongodb://<credentials>@127.0.0.1:27017/admin?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fcerts%2Fmongodb.pem&tlsCAFile=%2Fcerts%2Fmongodb-ca-cert&tlsAllowInvalidHostnames=true&appName=mongosh+2.3.2
Using MongoDB:		8.0.1
Using Mongosh:		2.3.2

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

------
   The server generated these startup warnings when booting
   2024-10-28T20:13:48.635+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2024-10-28T20:13:49.556+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2024-10-28T20:13:49.556+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2024-10-28T20:13:49.556+00:00: We suggest setting the contents of sysfsFile to 0.
   2024-10-28T20:13:49.556+00:00: Your system has glibc support for rseq built in, which is not yet supported by tcmalloc-google and has critical performance implications. Please set the environment variable GLIBC_TUNABLES=glibc.pthread.rseq=0
   2024-10-28T20:13:49.556+00:00: vm.max_map_count is too low
------

MongoServerError[InvalidReplicaSetConfig]: No host described in new configuration with {version: 1, term: 0} for replica set rs0 maps to this node

Looks like it could be similar to this ticket: #17297

@ZachThill
Copy link
Author

ZachThill commented Oct 28, 2024 •
edited by carrodher
Loading

Looking at some other forums, a lot of people are saying that this error is caused by the pod not having the custom advertised hostname in the /etc/hosts file:

/etc/hosts file

$ cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.8.5.235	mongodb-0.mongodb-headless.mongodb.svc.cluster.local	mongodb-0

source forums:
https://dba.stackexchange.com/questions/180189/cant-run-rs-initiate-when-do-mongodb-replication-set
https://stackoverflow.com/questions/29211285/errmsg-no-host-described-in-new-configuration-1-for-replica-set-rs0-maps-to

@ZachThill
Copy link
Author

ZachThill commented Oct 29, 2024 •
edited
Loading

@carrodher,
With my new found knowledge of MongoDB and this helm chart, I will create 2 other, more clear and concise issues stemming from this one.

#30128

#30129

I will close out this issue now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@carrodher carrodher

Labels
mongodb solved tech-issues The user has a technical issue about an application triage Triage is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@carrodher @ZachThill