Rotate MongoDb collection automatically in a periodic manner-CodePudding

Let's say I have a mongo collection ("resultData"). I want that every month collection rotation happens (similar to log rotation or archiving) and automatically rename the old collection to "{{old_collection_name}}_{{month_year}}."

This will help me to keep all data without increasing the size of the collection.

NOTE: I am using golang for development.

CodePudding user response：

In MongoDB collections do not need to exist prior to using them: you can insert into a collection that does not yet exist, and it will be created automatically.

So a simple and automatic solution is to always use a collection whose name is generated using the current month. As a result, documents will be inserted into a new collection once a new month starts (named after the month).

Here's a simple helper that implements this logic:

func getResultColl(db *mongo.Database) *mongo.Collection {
    name := "resultData_"   time.Now().Format("01_2006")
    return db.Collection(name)
}

This month (2022 November) this function returns a collection whose name is resultData_11_2022. In the next month, the name of the returned collection will be resultData_12_2022.

Using it:

var db *mongo.Database // initialize your Mongo DB

c := getResultColl(db)

if _, err := c.InsertOne(ctx, resultData); err != nil {
    // Handle error
}

Tip: if you move the year in front of the month, then the alphabetical order of the collections will be identical to the chronological order. So I suggest to use "2006_01" format (layout) instead of "01_2006".

Also note that there's nothing wrong having many documents in a collection. You could simply add the month as a field to the documents, and you can filter the results by month if you need, with an index it won't be slower. If you also store the timestamp or use an ObjectId as the _id, you can even do this without adding the month as an extra.