I have an app that contains millions of documents per Collection. My app is created using Android Java with Firestore Database. The problem is that Firestore's batched write can only update up to 500 documents. What if I need to update more than 500 documents? I cannot create multiple batched write just to solve my problem because I am using Android with Java and there's no such thing as Promise or AsyncTask in Java. Therefore, creating multiple batched write will not return a Promise. So I cannot assure the consistency of my data with Batched Write.
I read about Firebase Realtime Database's multipath updates and it can really solve my problem because it has no documented limit. But the problem with Realtime Database is that it doesn't support queries on multiple where clauses like in the Firestore database.
In my case where I want to have support for multiple where clauses and update up to millions of data, which database should I use?
Please help me because I am confused right now and I am planning to migrate to SQL if my problem with Firebase can't be solved.
CodePudding user response：
I cannot create multiple batched write just to solve my problem because I am using Android with Java and there's no such thing as Promise or AsyncTask in Java.
There is no concept of Promise in Java. However, the AsyncTask exists, but as you can see is deprecated even from API level 30.
Therefore, creating multiple batched write will not return a Promise.
It cannot return a Promise, it will return a Task. When you call WriteBatch#commit() the type of the object that is returned is a Task<Void>. This means that all those operations are performed in another thread, and the main thread will not be affected. Since you have a Task object, you can attach a listener using addOnCompleteListener() and wait for the operations to complete. So you can perform a batch one after another, or in parallel since all operations are pipelined over a single connection.
If you don't want to use this really simple approach, then you should consider using the RxJava Library.
If you want to improve your skills when it comes to Android development, I highly recommend you learn Kotlin. So when it comes to Kotlin, one of the best features in my opinion is represented by the Kotlin Coroutines.
For simple Firestore reads I recommend you read the following resource:
When it comes to updating millions of documents, doing that in your Android application's code, doesn't sound like a good solution. The best option that you have, in my opinion, is to use Cloud Functions for Firebase. So you'll be able to create a query using multiple whereEqualTo() calls, and everything will be performed on the server.
CodePudding user response：
You could still create multiple batches and run each batch individually. Promise.all() essentially runs all those promises simultaneously but each batch returns a separate promise at the end. If you don't want to handle this on app side, then you can use Cloud Functions and run it on server side, in case of any error, you can revert any documents added if required.
I read about Firebase Realtime Database's multipath updates and it can really solve my problem because it has no documented limit
The limits are documented in the documentation. The total data in each write operation should be less than 256 MB for REST APIs or 16 MB in case of SDKs
In my case where I want to have support for multiple where clauses and update up to millions of data, which database should I use?
This totally depends on what queries you need. Are read operations more frequent than write? Firestore might be a better choice as it can support queries with multiple constraints as in your question.