Trouble getting Bitnami MongoDB container to start via Helm

[JoeLudwig]

Everything below is about a new install. I have no existing data in the PVC, and in fact the PVC is recreated on each attempt because the namespace is empty before I do anything. I cloned the repo a few days ago, so I should have nearly the latest from main.

I'm deploying to a Kubernetes cluster via Helm, and have a persistent problem getting the MongoDB pod to start. It come up the first time time with this log spew, and then restarts (for reasons I don't understand):

 2025-08-15T00:58:01.497795515Z mongodb  00:58:01.49  INFO  ==> Welcome to the Bitnami mongodb container
 2025-08-15T00:58:01.499423408Z mongodb  00:58:01.49  INFO  ==> Subscribe to project updates by watching mhttps://github.com/bitnami/containers
 2025-08-15T00:58:01.501283667Z mongodb  00:58:01.50  INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
 2025-08-15T00:58:01.502564138Z mongodb  00:58:01.50  INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit �[1mhttps://bitnami.com/enterprise�[0m
 2025-08-15T00:58:01.503597246Z mongodb  00:58:01.50  INFO  ==> 
 2025-08-15T00:58:01.509950932Z mongodb  00:58:01.50  INFO  ==> ** Starting MongoDB setup **
 2025-08-15T00:58:01.523072910Z mongodb  00:58:01.52  INFO  ==> Validating settings in MONGODB_* env vars...
 2025-08-15T00:58:01.664182691Z mongodb  00:58:01.66  WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
 2025-08-15T00:58:01.696726515Z mongodb  00:58:01.69  INFO  ==> Initializing MongoDB...
 2025-08-15T00:58:01.769593251Z mongodb  00:58:01.76  INFO  ==> Deploying MongoDB from scratch...

Then it crashes (or maybe just restarts) the pod and from that point on the pod tries to come up over and over and hits this:

2025-08-15T00:58:04.437670144Z {"t":{"$date":"2025-08-15T00:58:04.437+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":17,"message":{"ts_sec":1755219484,"ts_usec":437494,"thread":"1:0x7fb450e34200","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:924:/bitnami/mongodb/data/db/WiredTiger.wt: handle-open: open","error_str":"File exists","error_code":17}}}
2025-08-15T00:58:04.456659757Z {"t":{"$date":"2025-08-15T00:58:04.456+00:00"},"s":"I",  "c":"WT",       "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1755219484,"ts_usec":456536,"thread":"1:0x7fb450e34200","session_name":"connection","category":"WT_VERB_BLOCK","category_id":4,"verbose_level":"NOTICE","verbose_level_id":-1,"msg":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.4"}}}
2025-08-15T00:58:04.482469661Z {"t":{"$date":"2025-08-15T00:58:04.481+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":1,"message":{"ts_sec":1755219484,"ts_usec":481963,"thread":"1:0x7fb450e34200","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:924:/bitnami/mongodb/data/db/WiredTiger.wt: handle-open: open","error_str":"Operation not permitted","error_code":1}}}
2025-08-15T00:58:04.574246101Z {"t":{"$date":"2025-08-15T00:58:04.574+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":17,"message":{"ts_sec":1755219484,"ts_usec":574066,"thread":"1:0x7fb450e34200","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:924:/bitnami/mongodb/data/db/WiredTiger.wt: handle-open: open","error_str":"File exists","error_code":17}}}
2025-08-15T00:58:04.593276010Z {"t":{"$date":"2025-08-15T00:58:04.593+00:00"},"s":"I",  "c":"WT",       "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1755219484,"ts_usec":593154,"thread":"1:0x7fb450e34200","session_name":"connection","category":"WT_VERB_BLOCK","category_id":4,"verbose_level":"NOTICE","verbose_level_id":-1,"msg":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.5"}}}
2025-08-15T00:58:04.618697426Z {"t":{"$date":"2025-08-15T00:58:04.618+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":1,"message":{"ts_sec":1755219484,"ts_usec":618572,"thread":"1:0x7fb450e34200","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:924:/bitnami/mongodb/data/db/WiredTiger.wt: handle-open: open","error_str":"Operation not permitted","error_code":1}}}
2025-08-15T00:58:04.634412608Z {"t":{"$date":"2025-08-15T00:58:04.634+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
2025-08-15T00:58:04.634433267Z {"t":{"$date":"2025-08-15T00:58:04.634+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"1: Operation not permitted"}}
2025-08-15T00:58:04.634441602Z {"t":{"$date":"2025-08-15T00:58:04.634+00:00"},"s":"F",  "c":"ASSERT",   "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":760}}
2025-08-15T00:58:04.634448706Z {"t":{"$date":"2025-08-15T00:58:04.634+00:00"},"s":"F",  "c":"ASSERT",   "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

Searching around for solutions to those errors of course led me here: #3972

I'm using Helm instead of docker compose, but the problem could have been the same, so I tried every major version of Bitnami MongoDB containers. Some of those were simple and I could just run them. Some of those have an issue with the permissions on the default Mongo config file, so I had to make a local version of the images to fix that, and then ran them. I tried everything V4 and above (because the Bitnami scripts seem to start failing when going back to MongoDB 3.6). None of that helped. With every version I still get one successful run, followed by N failed runs (depending on how long I let it keep retrying.)

I can't find anywhere that the Helm charts pre-populate the PVC with the raw wiretiger files. As near as I can tell those are being created by MongoDB itself, and then after it restarts, it chokes on its own files.

I've more or less run out of things to try at this point. I would really appreciate any help you can provide.

[JoeLudwig]

This turned out to be caused by a mismatch between the uid/gid on the filesystem used by the PVC for MongoDB and the IDs that Bitnami really wants to use. (1001:1001) I used containerSecurityContext in values.yaml to tell Bitnami to use the filesystem IDs and now MongoDB and LibreChat are starting up!

Leaving these breadcrumbs for the next person and closing this one. Whatever issues remain are unlikely to be related to MongoDB filesystem shenanigans.