Mongo not creating all docs in array and not giving errors-CodePudding

I have a long array of docs to create. When I create them I get no errors.

const docsJson =[some array of json of docs to create]
 
const orders = await MySchema.create(ordersJSON);
// orders.length returns the same number of docs as  docsJson

But when I search for the new docs, only some were created.


const actualOrdersCreated = await MySchema.find({ _id: { $in: orders.map((p) => p._id) } });
// actualOrdersCreated.length returns less docs than in docsJson

What's causing this?

CodePudding user response：

I think your data is to large. The maximum BSON document size is 16 megabytes. Reference: https://www.mongodb.com/docs/manual/reference/limits/

CodePudding user response：

This was due to having a ttl (time to live) index on one mongo database and not the other. I was copying docs over from a database. The index on the first database was:

$ mongo "mongodb srv://....database-1-url"
>> db.myschema.getIndexes()
[
        {
                "v" : 2,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_"
        },
        {
                "v" : 2,
                "key" : {
                        "paidOn" : 1
                },
                "name" : "paidOn_1",
                "background" : true
        }
]

But the database I was working with had expireAfterSeconds.

$ mongo "mongodb srv://....database-2-url"
>> db.myschema.getIndexes()
[
        {
                "v" : 2,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_"
        },
        {
                "v" : 2,
                "key" : {
                        "expireAt" : 1
                },
                "name" : "expireAt_1",
                "background" : true,
                "expireAfterSeconds" : 86400
        },
        {
                "v" : 2,
                "key" : {
                        "paidOn" : 1
                },
                "name" : "paidOn_1",
                "background" : true
        }
]

So mongo was deleting the new docs where the expireAt field had an old date.

To fix it I ran await Order.syncIndexes(); in a script. This cleared the index to [ { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_" } ]. This is fine for my purpose. But the index isn't the same as the 1st database. The paidOn key is no longer indexes.

What I thought helped but didn't

At first I thought the issue was due to the large size of the jsonDocs.

I had objects with fields that had large base64 strings for images. These were placeholders, supposed to be replaced with http urls for the images.

After I removed the base64 strings I was able to upload the documents. I thought this helped but it was just speeding things up. It takes 1 minute for mongo to checkout up on expired docs.