MongoDB crashes due to out of memory exception-CodePudding

I process a huge amount of data in MongoDB and recently my server hosting the data lake consistently crashes when data processing is done. The log files indicate that it is out of memory with the following error message:

[conn29] out of memory.

This is weird because there is still a lot of space on the disk of my server so I am confused why mongoDB crashes with this error. Previously, I have been running the compact command to reclaim any free space after the processing is done. Is it possible that this just, somehow, takes space away from mongoDB gradually and give it back to the OS, hence the out of memory exception? How can I monitor the amount of memory available for mongoDB?

Edit: The RAM size of my server is around 8GB. I don't share the mongoDB with other applications. The cache size (using the command db.serverStatus().wiredTiger.cache and selecting maximum bytes configured) is around 3.7GB.

The MongoDB system log files show the following:

2021-12-01T07:07:54.843 0000 I COMMAND  [conn29] command nimbus.tmp.agg_out.55 command: renameCollection { renameCollection: "nimbus.tmp.agg_out.55", to: "nimbus.ACTIVITY_STAGE", dropTarget: true, $db: "admin" } numYields:0 reslen:38 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { W: 1 } } } protocol:op_msg 684ms
2021-12-01T07:07:55.146 0000 I COMMAND  [conn29] command nimbus.ACTIVITY_STAGE command: aggregate { aggregate: "RMPlus", pipeline: [ { $match: { $and: [ { DATE_LOADED: "2021-11-30" }, { DATA_TYPE: "activities" } ] } }, { $unwind: { path: "$activities", preserveNullAndEmptyArrays: true } }, { $project: { activities.rowLevelSecurity: 0 } }, { $project: { activities.additionalRMs: 0 } }, { $addFields: { activities.DATA_TYPE: "$DATA_TYPE" } }, { $addFields: { activities.DATE_LOADED: "$DATE_LOADED" } }, { $addFields: { activities.FILE_NAME: "$FILE_NAME" } }, { $addFields: { activities.region: "$activities.rowLevelSecurity.region" } }, { $replaceRoot: { newRoot: "$activities" } }, { $project: { additionalRMCode: 0 } }, { $project: { activityTopics: 0 } }, { $project: { userParticipations: 0 } }, { $project: { entityParticipations: 0 } }, { $project: { rowLevelSecurity: 0 } }, { $project: { text: 0 } }, { $project: { textPreviewLong: 0 } }, { $unwind: { path: "$additionalRMCode", preserveNullAndEmptyArrays: true } }, { $group: { _id: { FILE_NAME: "$FILE_NAME", DATA_TYPE: "$DATA_TYPE", DATE_LOADED: "$DATE_LOADED" }, count: { $sum: 1 }, Values: { $push: "$$ROOT" } } }, { $project: { Values.DATA_TYPE: 0, Values.DATE_LOADED: 0, Values.FILE_NAME: 0 } }, { $project: { DATA_TYPE: "$_id.DATA_TYPE", DATE_LOADED: "$_id.DATE_LOADED", FILE_NAME: "$_id.FILE_NAME", Values: 1, _id: 0, ROW_COUNT: "$count" } }, { $out: "ACTIVITY_STAGE" } ], allowDiskUse: true, cursor: {}, lsid: { id: UUID("c255f805-2156-42d4-8664-e2f851358570") }, $db: "nimbus", $readPreference: { mode: "primaryPreferred" } } planSummary: IXSCAN { DATE_LOADED: 1, DATA_TYPE: 1 } keysExamined:2293 docsExamined:2293 fromMultiPlanner:1 replanned:1 cursorExhausted:1 numYields:2301 nreturned:0 reslen:102 locks:{ Global: { acquireCount: { r: 6804, w: 2250, W: 1 } }, Database: { acquireCount: { r: 4551, w: 2248, W: 2 } }, Collection: { acquireCount: { r: 4550, w: 2247 } } } protocol:op_msg 306799ms
2021-12-01T07:31:03.274 0000 F -        [conn29] out of memory.
mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(247)           mongo::printStackTrace 0x67x
mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(361)  ???
mongod.exe    ...\src\mongo\util\allocator.cpp(46)                     mongo::mongoMalloc 0x25x
mongod.exe                                                             ???
mongod.exe                                                             ???

There is some stuff there that needs context but the main takeaway is that everything executes as planned up until it says out of memory

CodePudding user response：

The MongoDB system log files show the following:

2021-12-01T07:07:54.843 0000 I COMMAND  [conn29] command nimbus.tmp.agg_out.55 command: renameCollection { renameCollection: "nimbus.tmp.agg_out.55", to: "nimbus.ACTIVITY_STAGE", dropTarget: true, $db: "admin" } numYields:0 reslen:38 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { W: 1 } } } protocol:op_msg 684ms
2021-12-01T07:07:55.146 0000 I COMMAND  [conn29] command nimbus.ACTIVITY_STAGE command: aggregate { aggregate: "RMPlus", pipeline: [ { $match: { $and: [ { DATE_LOADED: "2021-11-30" }, { DATA_TYPE: "activities" } ] } }, { $unwind: { path: "$activities", preserveNullAndEmptyArrays: true } }, { $project: { activities.rowLevelSecurity: 0 } }, { $project: { activities.additionalRMs: 0 } }, { $addFields: { activities.DATA_TYPE: "$DATA_TYPE" } }, { $addFields: { activities.DATE_LOADED: "$DATE_LOADED" } }, { $addFields: { activities.FILE_NAME: "$FILE_NAME" } }, { $addFields: { activities.region: "$activities.rowLevelSecurity.region" } }, { $replaceRoot: { newRoot: "$activities" } }, { $project: { additionalRMCode: 0 } }, { $project: { activityTopics: 0 } }, { $project: { userParticipations: 0 } }, { $project: { entityParticipations: 0 } }, { $project: { rowLevelSecurity: 0 } }, { $project: { text: 0 } }, { $project: { textPreviewLong: 0 } }, { $unwind: { path: "$additionalRMCode", preserveNullAndEmptyArrays: true } }, { $group: { _id: { FILE_NAME: "$FILE_NAME", DATA_TYPE: "$DATA_TYPE", DATE_LOADED: "$DATE_LOADED" }, count: { $sum: 1 }, Values: { $push: "$$ROOT" } } }, { $project: { Values.DATA_TYPE: 0, Values.DATE_LOADED: 0, Values.FILE_NAME: 0 } }, { $project: { DATA_TYPE: "$_id.DATA_TYPE", DATE_LOADED: "$_id.DATE_LOADED", FILE_NAME: "$_id.FILE_NAME", Values: 1, _id: 0, ROW_COUNT: "$count" } }, { $out: "ACTIVITY_STAGE" } ], allowDiskUse: true, cursor: {}, lsid: { id: UUID("c255f805-2156-42d4-8664-e2f851358570") }, $db: "nimbus", $readPreference: { mode: "primaryPreferred" } } planSummary: IXSCAN { DATE_LOADED: 1, DATA_TYPE: 1 } keysExamined:2293 docsExamined:2293 fromMultiPlanner:1 replanned:1 cursorExhausted:1 numYields:2301 nreturned:0 reslen:102 locks:{ Global: { acquireCount: { r: 6804, w: 2250, W: 1 } }, Database: { acquireCount: { r: 4551, w: 2248, W: 2 } }, Collection: { acquireCount: { r: 4550, w: 2247 } } } protocol:op_msg 306799ms
2021-12-01T07:31:03.274 0000 F -        [conn29] out of memory.
mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(247)           mongo::printStackTrace 0x67x
mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(361)  ???
mongod.exe    ...\src\mongo\util\allocator.cpp(46)                     mongo::mongoMalloc 0x25x
mongod.exe                                                             ???
mongod.exe                                                             ???

There is some stuff there that needs context but the main takeaway is that everything executes as planned up until it says out of memory

CodePudding user response：

storage section where you can change cacheSizeGB looks more or less as follow:

storage:
   dbPath: <the data directory>
   journal:
     enabled: true
   directoryPerDB: true
   engine: wiredTiger
   wiredTiger:
     engineConfig:
       cacheSizeGB: 2

CodePudding user response：

After making the modifications to the .cfg file and restarting the MongoDB server, the log files log the following error:

2021-12-02T15:02:14.473 0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2021-12-02T15:02:15.028 0000 I CONTROL  [main] Trying to start Windows service 'MongoDB'
2021-12-02T15:02:15.031 0000 I CONTROL  [initandlisten] MongoDB starting : pid=5072 port=27017 dbpath=G:\MongoDB\4.0\data 64-bit host=ggws2006007
2021-12-02T15:02:15.031 0000 I CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2021-12-02T15:02:15.032 0000 I CONTROL  [initandlisten] db version v4.0.4
2021-12-02T15:02:15.032 0000 I CONTROL  [initandlisten] git version: f288a3bdf201007f3693c58e140056adf8b04839
2021-12-02T15:02:15.032 0000 I CONTROL  [initandlisten] allocator: tcmalloc
2021-12-02T15:02:15.033 0000 I CONTROL  [initandlisten] modules: none
2021-12-02T15:02:15.033 0000 I CONTROL  [initandlisten] build environment:
2021-12-02T15:02:15.033 0000 I CONTROL  [initandlisten]     distmod: 2008plus-ssl
2021-12-02T15:02:15.034 0000 I CONTROL  [initandlisten]     distarch: x86_64
2021-12-02T15:02:15.034 0000 I CONTROL  [initandlisten]     target_arch: x86_64
2021-12-02T15:02:15.035 0000 I CONTROL  [initandlisten] options: { config: "G:\MongoDB\4.0\bin\mongod.cfg", net: { bindIp: "127.0.0.1,ggws2006007.eur.ad.hedani.net", port: 27017 }, security: { authorization: "disabled" }, service: true, storage: { dbPath: "G:\MongoDB\4.0\data", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { engineConfig: { cacheSizeGB: 6.0 } } }, systemLog: { destination: "file", logAppend: true, path: "G:\MongoDB\4.0\log\mongod.log" } }
2021-12-02T15:02:15.055 0000 I STORAGE  [initandlisten] exception in initAndListen: InvalidOptions: Requested option conflicts with current storage engine option for directoryPerDB; you requested true but the current server storage is already set to false and cannot be changed, terminating
2021-12-02T15:02:15.056 0000 I NETWORK  [serviceStopWorker] shutdown: going to close listening sockets...
2021-12-02T15:02:15.057 0000 I CONTROL  [serviceStopWorker] now exiting