Cloud Firestore/Firebase is awesome in its ability to sync documents. However, I have an interesting use-case and I was wondering if the platform might be able to help. I have parquet files that I want to be able to sync to a central server -- and possibly other users. These are large binary files, for example, 100MB parquet files or so.

The max size of a document is 1 MiB. Are there any other ways to accomplish what Im looking to do with Firebase (I suppose partitioning the file into multiple 1MB pieces?) Or does this seem like the wrong tool for what I'm looking to do?

CodePudding user response：

I think you are mixing up the limit for Cloud Firestore (one of the NoSQL dbs offered by Firebase/GCP) and Cloud Storage, the service for objects/file storage.

The maximum size limit for individual objects stored in Cloud Storage is 5 TiB. Of course, uploading such a file from a web or mobile app may not make sense, but this is more a functional limit.

So you can very well upload large files with Cloud Storage and associate to each Cloud Storage file a Firestore document which contains, for example, a download URL. With that you have the elementary bricks of a file sharing system: The storage is based on Cloud Storage and Firestore contains documents that point to the files and handles the collaborative file-sharing part based on these docs..

To associate the Cloud Storage file and the Firestore doc you can adopt several strategies, depending on your exact use case:

• Use the Firestore document ID as the file name;
• Store the Cloud Storage path and/or download URL in the Firestore document;
• Store the file in a folder named with the Firestore document ID
• ...