Home > Software engineering >  Mongoose bulk insert misses critical information
Mongoose bulk insert misses critical information

Time:12-20

image

The picture above shows an example of a request getting sent to the following route:

image

and the following picture shows what have been inserted to the database in compass (notice how there are three entries):

image

As we know, Model.create() accepts an array of objects, or an object.

In this example, I am sending an array of objects, to insert them.

Model.create([]) will insert the documents one by one to the database, it doesn't skip the validation part, which is why I chose it. and when it finds a document with a validation error, it skips it, and moves to the next one. until it finishes, then it reports the errors it encounters.

That's what it should be, However it's not exactly working like that.

Note that I have two documents which holds validation errors:

image

However, mongoose is only reporting the first one, it's not reporting the second one, even though it passes by it, and it sees it.

Why this information is critical?

Because on the client side, I have to know which documents got inserted, and which did not.

In this case, (when I will know which are the ones got inserted and the ones that did not), I can show the client for example that the documents x, y, z has been inserted, while the documents f, g, h did not. So the user can correct his mistake and send the request again.

The current error report is useless, because it only tells "there was a validation error", but it doesn't tell you the "where"

The error report should include all the documents which refused to be written to the database in an array.

Update

I realized that

const data = await User.insertMany(req.body)

Has exactly the same behavior. It doesn't only apply to Model.create(). Model.insertMany() has the same problem as well.

How to make mongoose report the full errors?


since we don't have a fold code option yet, I will include the code shown in the images, down here. and I hope this isn't going to polute the question.

mongoose.connect('mongodb://localhost:27017/temp', (err) => {
  if (err) return log.error(log.label, 'an error occured while connecting to DB!')
  log.success(log.label, 'Successfully connected to the database!')
})

app.use(express.json())

app.post('/users', async (req, res, next) => {
  try {
    const data = await User.collection.insertMany(req.body)
    res.json({
      message: 'success!',
      data,
    })
  } catch (error) {
    console.log(error)
    res.json({
      message: 'an error',
      data: error,
    })
  }
})

CodePudding user response:

I checked the source code of create() method. It indeed only saves the first error and not all the errors.

However, since the create() method will send one request for each item anyway, you can implement your own logic where you will wrap all the items with Promise.allSettled() and use the create() method for each item. That way, you will know exactly which item was successfully added, and which threw an error:

app.post('/users', async (req, res, next) => {
  try {
    const items = req.body;

    const results = await Promise.allSettled(
      items.map((item) => User.create(item))
    );

    constole.log(results.map((result) => result.status);
    // Items that were successfully created will have the status "fulfilled", 
    // and items that were not successfully created will have the status "rejected".
   
    return res.status(200).json({ message: 'success', results })
  } catch (error) {
    console.log(error)
    return res.status(400).json({ message: 'an error', data: error })
  }
})

CodePudding user response:

Just want to draw your attention to insertMany issue-5337 which is similar to your question, they resolved it differently, like below:

Comment on issue: https://github.com/Automattic/mongoose/issues/5783#issuecomment-341590245

In hind sight, going to have to punt on this one until a later release because we need to return a different structure if rawResult is false. We can't just return a ValidationError like in #5698 because that would cause a promise rejection, which isn't correct with insertMany() with ordered: false because that's very inconsistent with how the driver handles it. Using rawResult: true is the way to go right now.

  • ordered: false should give you multiple validation errors, but if ordered is not set (true by default) we should fall back to the current behavior.
User.collection.insertMany(req.body, { rawResult: true, ordered: false })
  .then(users => {
    console.log(users)
  })
  .catch(err => {
    console.log(`Error: ${err}`);
  });

Console Print:

{
  acknowledged: true,
  insertedCount: 3,
  insertedIds: {
    '0': new ObjectId("63a09dcaf4f03d04b07ec1dc"),
    '1': new ObjectId("63a09dcaf4f03d04b07ec1de")
    '2': new ObjectId("63a0a0bdfd94d1d4433e77da")
  },
  mongoose: { 
    validationErrors: [
      [
        Error: WASetting validation failed: ....
          at ValidationError.inspect (...)
          ...
          errors: { itemId: [ValidatorError] },
          _message: '.....'
      ],
      [
        Error: WASetting validation failed: ....
          at ValidationError.inspect (...)
          ...
          errors: { itemId: [ValidatorError] },
          _message: '....'
      ]
    ] 
  }
}

You can see the above response, this is not giving which object failed.


Currently, I would suggest @NeNaD's solution.

  • Related