How to iterate firestore batch writes to perform more than 500 operations in Flutter?-CodePudding

This isn't a duplicate because I did not find any question or solution for writing more than 500 documents using batches in FLUTTER. I've seen answers for this in other frameworks or languages but I can't understand how to implement this in flutter. I need to update documents in my collections using batch write but a batched write can only contain 500 operations. So how do I write more than 500 documents to firestore? I've seen iterating as one of the solution but how do I implement that in flutter?

This is how I am performing single batch write:

Future<void> batchUpdate() {
  WriteBatch batch = FirebaseFirestore.instance.batch();

  return users
      .get()
      .then((snapshot) {
        for (DocumentSnapshot document in snapshot.docs) {
          document.reference.update(
            {
              'totalScore': 0,
            },
          );
        }
        return batch.commit();
      })
      .then((value) => ScaffoldMessenger.of(context).showSnackBar(snackBar))
      .catchError(
        (error) => ScaffoldMessenger.of(context).showSnackBar(
          SnackBar(
            content: Text(error),
          ),
        ),
      );
}

CodePudding user response：

A batched write has a limit of 500 operations, a hard limit. You can simply perform multiple batches, to add more documents, but you can't exceed the limit.

One thing you can do is, by creating a counter variable and incrementing its value every time an operation is started to the batch. Then create an if statement and every time you increment the counter, check to see if it reached 500. At that time, commit the current batch, reset the counter and start a new batch, picking up where you left off. Do this till you finish all the batch writes. Or we can create a BatchedWrite object by calling batch(), fill that until it has a maximum capacity of 500 documents, and then write it to Firestore.

Writing 1,000 documents takes about 2.8 seconds with this approach, so the throughput is roughly 357 document writes per second.

You can try to implement the above logic in Flutter code as I am not much familiar with Flutter. But anyone who is looking for a JS code for the above logic can see below code references:

async function testBatchedWrites(datas) { 
let batch = admin.firestore().batch(); 
let count = 0; 
while (datas.length) {
 batch.set(collection.doc(Math.random().toString(36).substring(2, 15)), datas.shift()); 
if (  count >= 500 || !datas.length) { 
await batch.commit(); 
batch = admin.firestore().batch(); 
count = 0;
 } } }

Or this,

// prepare the batch 
let currentBatch = firebase.firestore().batch(); 
let currentBatchSize = 0; 
const batches = [ currentBatch ]; // add each doc's deletion to the batch 
docs.forEach((doc) => { 
// when batch is too large, start a new one 
if (  currentBatchSize >= 500) { 
currentBatch = firebase.firestore.batch(); 
batches.push(currentBatch); 
currentBatchSize = 1; } // add operation to batch 
currentBatch.delete(doc.ref); }) // commit the changes 
await Promise.all(batches.map(batch => batch.commit()));