E11000 Duplicate Key Error seems to be causing Librechat to crash #5776
Replies: 1 comment 13 replies
-
A change was made that should help with this already but I will see if I can prevent any errors in the relevant code from bubbling up. I will cut a release candidate today that you can test with. Thanks |
Beta Was this translation helpful? Give feedback.
-
Are you using the latest? There was this fix introduced but we don't want to use the latest for production to test it. #6248 |
Beta Was this translation helpful? Give feedback.
-
We are using 0.7.6 atm. We have no date to update to 0.7.7, but we will eventually - so to answer your question, no we don't have that fix yet :/ With that said, i've not noticed that PR but that might fix it yeah. While investigating the logs, on my end I'm pretty sure the exception starts in I tried to see where it would bobble up without being caught but I had little time to do so |
Beta Was this translation helpful? Give feedback.
-
|
The error still happens for us even after installing the latest with this patch: #6248 |
Beta Was this translation helpful? Give feedback.
-
|
and still after: #6392 ? |
Beta Was this translation helpful? Give feedback.
-
The error gets caught properly now so I hope it doesn't crash anymore. Hasn't happened yet! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
What happened?
In our organization (a University), we run a production LibreChat instance (currently v0.7.6) using OpenID Connect for authentication and connecting users via their university credentials. This instance services hundreds of users concurrently and is integrated with the OpenAI API.
We are experiencing occasional duplicate key errors on the “messageId” field when saving messages. One sample error log reads as follows:
{"code":11000, "codeName":"DuplicateKey", "errorResponse":{… "errmsg": "Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: LibreChat.messages index: messageId_1 dup key: { messageId: \"462df4b0-f1e5-4e4f-804e-78f0d84024c3\" }", …}, …}It appears that:
• The duplicate key error is raised during upsert/update operations on messages.
• The error seems to occur right after an API failure (from our integration with OpenAI).
• We haven’t seen this error in our testing environment, which uses the same configuration, suggesting that higher concurrency in production may be triggering a race condition or duplicate retry logic.
• We are using MongoDB as the backend.
We suspect that there might be edge conditions in the Message.js upsert logic (or related concurrent abort handling) that are leading to duplicate insertion attempts.
Version Information
ghcr.io/danny-avila/librechat v0.7.6 6cdfd9f7ce88 7 weeks ago 893MB
Steps to Reproduce
Not fully reproducible on demand, but the issue occurs under high concurrency with production load and following intermittent API failures.
What browsers are you seeing the problem on?
No response
Relevant log output
{ "code": 11000, "codeName": "DuplicateKey", "errorResponse": { "code": 11000, "codeName": "DuplicateKey", "errmsg": "Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: LibreChat.messages index: messageId_1 dup key: { messageId: \"462df4b0-f1e5-4e4f-804e-78f0d84024c3\" }", "keyPattern": { "messageId": 1 }, "keyValue": { "messageId": "462df4b0-f1e5-4e4f-804e-78f0d84024c3" }, "ok": 0 }, "keyPattern": { "messageId": 1 }, "keyValue": { "messageId": "462df4b0-f1e5-4e4f-804e-78f0d84024c3" }, "level": "error", "message": "Error saving message: Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: LibreChat.messages index: messageId_1 dup key: { messageId: \"462df4b0-f1e5-4e4f-804e-78f0d84024c3\" }", "ok": 0, "stack": "MongoServerError: Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: LibreChat.messages index: messageId_1 dup key: { messageId: \"462df4b0-f1e5-4e4f-804e-78f0d84024c3\" }\n at Connection.sendCommand (/app/api/node_modules/mongodb/lib/cmap/connection.js:290:27)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Connection.command (/app/api/node_modules/mongodb/lib/cmap/connection.js:317:26)\n at async Server.command (/app/api/node_modules/mongodb/lib/sdam/server.js:167:29)\n at async FindOneAndUpdateOperation.executeCommand (/app/api/node_modules/mongodb/lib/operations/command.js:73:16)\n at async FindOneAndUpdateOperation.execute (/app/api/node_modules/mongodb/lib/operations/find_and_modify.js:85:24)\n at async tryOperation (/app/api/node_modules/mongodb/lib/operations/execute_operation.js:199:20)\n at async executeOperation (/app/api/node_modules/mongodb/lib/operations/execute_operation.js:69:16)\n at async Collection.findOneAndUpdate (/app/api/node_modules/mongodb/lib/collection.js:491:16)\n at async model.Query._findOneAndUpdate (/app/api/node_modules/mongoose/lib/query.js:3462:13)" }{ "code": 11000, "codeName": "DuplicateKey", "errorResponse": { "code": 11000, "codeName": "DuplicateKey", "errmsg": "Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: LibreChat.messages index: messageId_1 dup key: { messageId: \"462df4b0-f1e5-4e4f-804e-78f0d84024c3\" }", "keyPattern": { "messageId": 1 }, "keyValue": { "messageId": "462df4b0-f1e5-4e4f-804e-78f0d84024c3" }, "ok": 0 }, "keyPattern": { "messageId": 1 }, "keyValue": { "messageId": "462df4b0-f1e5-4e4f-804e-78f0d84024c3" }, "level": "error", "message": "[handleAbortError] AI response error; aborting request: Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: LibreChat.messages index: messageId_1 dup key: { messageId: \"462df4b0-f1e5-4e4f-804e-78f0d84024c3\" }", "ok": 0, "stack": "MongoServerError: Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: LibreChat.messages index: messageId_1 dup key: { messageId: \"462df4b0-f1e5-4e4f-804e-78f0d84024c3\" }\n at Connection.sendCommand (/app/api/node_modules/mongodb/lib/cmap/connection.js:290:27)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Connection.command (/app/api/node_modules/mongodb/lib/cmap/connection.js:317:26)\n at async Server.command (/app/api/node_modules/mongodb/lib/sdam/server.js:167:29)\n at async FindOneAndUpdateOperation.executeCommand (/app/api/node_modules/mongodb/lib/operations/command.js:73:16)\n at async FindOneAndUpdateOperation.execute (/app/api/node_modules/mongodb/lib/operations/find_and_modify.js:85:24)\n at async tryOperation (/app/api/node_modules/mongodb/lib/operations/execute_operation.js:199:20)\n at async executeOperation (/app/api/node_modules/mongodb/lib/operations/execute_operation.js:69:16)\n at async Collection.findOneAndUpdate (/app/api/node_modules/mongodb/lib/collection.js:491:16)\n at async model.Query._findOneAndUpdate (/app/api/node_modules/mongoose/lib/query.js:3462:13)" }{ "code": "ERR_HTTP_HEADERS_SENT", "level": "error", "message": "[handleAbortError] error while trying to abort message Cannot set headers after they are sent to the client", "stack": "Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n at ServerResponse.setHeader (node:_http_outgoing:655:11)\n at ServerResponse.header (/app/node_modules/express/lib/response.js:794:10)\n at ServerResponse.send (/app/node_modules/express/lib/response.js:174:12)\n at ServerResponse.json (/app/node_modules/express/lib/response.js:278:15)\n at ServerResponse.send (/app/node_modules/express/lib/response.js:162:21)\n at abortMessage (/app/api/server/middleware/abortMiddleware.js:30:28)\n at handleAbortError (/app/api/server/middleware/abortMiddleware.js:213:20)\n at AskController (/app/api/server/controllers/AskController.js:178:5)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /app/api/server/routes/ask/openAI.js:24:5" }{ "code": 11000, "codeName": "DuplicateKey", "errorResponse": { "code": 11000, "codeName": "DuplicateKey", "errmsg": "Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: LibreChat.messages index: messageId_1 dup key: { messageId: \"e54164bf-a317-4916-b13c-d7e5129cd6a5\" }", "keyPattern": { "messageId": 1 }, "keyValue": { "messageId": "e54164bf-a317-4916-b13c-d7e5129cd6a5" }, "ok": 0 }, "keyPattern": { "messageId": 1 }, "keyValue": { "messageId": "e54164bf-a317-4916-b13c-d7e5129cd6a5" }, "level": "error", "message": "Error saving message: Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: LibreChat.messages index: messageId_1 dup key: { messageId: \"e54164bf-a317-4916-b13c-d7e5129cd6a5\" }", "ok": 0, "stack": "MongoServerError: Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: LibreChat.messages index: messageId_1 dup key: { messageId: \"e54164bf-a317-4916-b13c-d7e5129cd6a5\" }\n at Connection.sendCommand (/app/api/node_modules/mongodb/lib/cmap/connection.js:290:27)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Connection.command (/app/api/node_modules/mongodb/lib/cmap/connection.js:317:26)\n at async Server.command (/app/api/node_modules/mongodb/lib/sdam/server.js:167:29)\n at async FindOneAndUpdateOperation.executeCommand (/app/api/node_modules/mongodb/lib/operations/command.js:73:16)\n at async FindOneAndUpdateOperation.execute (/app/api/node_modules/mongodb/lib/operations/find_and_modify.js:85:24)\n at async tryOperation (/app/api/node_modules/mongodb/lib/operations/execute_operation.js:199:20)\n at async executeOperation (/app/api/node_modules/mongodb/lib/operations/execute_operation.js:69:16)\n at async Collection.findOneAndUpdate (/app/api/node_modules/mongodb/lib/collection.js:491:16)\n at async model.Query._findOneAndUpdate (/app/api/node_modules/mongoose/lib/query.js:3462:13)" }{ "level": "error", "message": "[OpenAIClient.chatCompletion] Unhandled error type Error: The server had an error while processing your request. Sorry about that!", "stack": "Error: Error: The server had an error while processing your request. Sorry about that!\n at OpenAIClient.chatCompletion (/app/api/app/clients/OpenAIClient.js:1410:15)\n at async OpenAIClient.sendCompletion (/app/api/app/clients/OpenAIClient.js:659:15)\n at async OpenAIClient.sendMessage (/app/api/app/clients/BaseClient.js:577:24)\n at async AskController (/app/api/server/controllers/AskController.js:131:20)\n at async /app/api/server/routes/ask/openAI.js:24:5" }{ "level": "error", "message": "[handleAbortError] AI response error; aborting request: Error: The server had an error while processing your request. Sorry about that!", "stack": "Error: Error: The server had an error while processing your request. Sorry about that!\n at OpenAIClient.chatCompletion (/app/api/app/clients/OpenAIClient.js:1410:15)\n at async OpenAIClient.sendCompletion (/app/api/app/clients/OpenAIClient.js:659:15)\n at async OpenAIClient.sendMessage (/app/api/app/clients/BaseClient.js:577:24)\n at async AskController (/app/api/server/controllers/AskController.js:131:20)\n at async /app/api/server/routes/ask/openAI.js:24:5" }{ "code": "ERR_HTTP_HEADERS_SENT", "level": "error", "message": "[handleAbortError] error while trying to abort message Cannot set headers after they are sent to the client", "stack": "Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n at ServerResponse.setHeader (node:_http_outgoing:655:11)\n at ServerResponse.header (/app/node_modules/express/lib/response.js:794:10)\n at ServerResponse.send (/app/node_modules/express/lib/response.js:174:12)\n at ServerResponse.json (/app/node_modules/express/lib/response.js:278:15)\n at ServerResponse.send (/app/node_modules/express/lib/response.js:162:21)\n at abortMessage (/app/api/server/middleware/abortMiddleware.js:30:28)\n at handleAbortError (/app/api/server/middleware/abortMiddleware.js:213:20)\n at AskController (/app/api/server/controllers/AskController.js:178:5)\n at async /app/api/server/routes/ask/openAI.js:24:5" }Screenshots
No response
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions