Let's say I have a model as below

import {model, Schema, Types} from 'mongoose'
import { IResource } from '../interfaces'

const ResourceSchema = new Schema<IResource>({
    user : {type : Schema.Types.ObjectId, ref : 'users'},
    type : {type: Schema.Types.ObjectId , ref : 'resource_datas'},
    building : {type : Schema.Types.ObjectId , ref : 'buildings'},
    lastUpdate : {type : Date , default : Date.now},
    value : {type : Number, default : 500},
    valuePerHours : {type : Number, default : 0}
})

const Resources = model<IResource>('resources' , ResourceSchema)

export default Resources

And sample records :

[
   {
      user : User_ID,
      type : Type_ID,
      building : Building_ID,
      lastUpdate : Date("2022-03-21T08:32:40.866 00:00"),
      value : 500,
      valuePerHours : 120
   },
   {
      user : User_ID,
      type : Type_ID,
      building : Building_ID,
      lastUpdate : Date("2022-03-21T08:22:40.866 00:00"),
      value : 540,
      valuePerHours : 150
   },
   {
      user : User_ID,
      type : Type_ID,
      building : Building_ID,
      lastUpdate : Date("2022-03-21T08:36:40.866 00:00"),
      value : 1200,
      valuePerHours : 180
   },
]

How can I update field "value" as often as possible?

Currently, I do like bellow

while (true) {
        const resources = await Resources.find({})
    .limit(100)
    .sort({lastUpdate : 1})
    .populate('building')

    const promiseSave = []
    for (let index = 0; index < resources.length; index  ) {
        const resource = resources[index];
        const now = Date.now()
        const diffTime = (now - new Date(resource.lastUpdate).getTime()) / 1000
        const percentDiffTimePerHour = diffTime / 3600000
        const generate = resource.valuePerHours
        const valueAfterDiff = generate * percentDiffTimePerHour
        resource.value  = valueAfterDiff
        resource.lastUpdate = now         
        promiseSave.push(resource.save())
    }
    await Promise.all(promiseSave)
 }

//update value follow lastUpdate and valuePerHours 

Every round for find 100 records and save them, It took about 200ms if I have 1mil record, It will take 2000seconds to update all records. I can update it once per hour or per day. But better it should nearly "real-time". Any better way to do that? Thanks for reading.

CodePudding user response：

You don't need any loop or fancy logic. You can run a simple update aggregation pipeline:

db.Resources.updateMany({}, [
   {
      $set: {
         lastUpdate: "$$NOW",
         value: {
            $sum: ["$value", {
               $multiply: [{
                  $divide: [
                     { $dateDiff: { startDate: "$lastUpdate", endDate: "$$NOW", unit: "second" } },
                     60 * 60 * 1000
                  ]
               }, "$valuePerHours"]
            }]
         }
      }
   }
])

or a bit shorter:

db.Resources.updateMany({}, [
   {
      $set: {
         lastUpdate: "$$NOW",
         value: {
            $sum: ["$value", {
               $multiply: [
                  { $dateDiff: { startDate: "$lastUpdate", endDate: "$$NOW", unit: "second" } },
                  "$valuePerHours",
                  1 / 60 / 60 / 1000
               ]
            }]
         }
      }
   }
])

This updates value on resolution of Seconds. If you require resolution of Milliseconds use

db.Resources.updateMany({}, [
   {
      $set: {
         lastUpdate: "$$NOW",
         value: {
            $sum: ["$value", {
               $multiply: [{
                  $divide: [
                     { $dateDiff: { startDate: "$lastUpdate", endDate: "$$NOW", unit: "millisecond" } },
                     60 * 60 * 1000 * 1000
                  ]
               }, "$valuePerHours"]
            }]
         }
      }
   }
])