How to handle that Firestore returns document items in inconsistent order-CodePudding

Assume that I have the following table stored in a Firestore collection:

AAA BBB CCC
6   1   15
3   19  83
100 4   2

I would like to read the entire collection and convert it to a Pandas dataframe in Python. I tried it with the following code piece:

cred = credentials.Certificate(serviceAccountKeyJson)
firebase_admin.initialize_app(cred, databaseURL)
db = firestore.client()

col_ref = db.collection(collection_name)

docs = col_ref.stream()

rows_list = []
for doc in docs:
    column_name_list = list(doc.to_dict().keys())
    values_list = list(doc.to_dict().values())
    rows_list.append(values_list)

col_data_df = pd.DataFrame(rows_list, columns=column_name_list)

It works but when I save the col_data_df into a csv file, the content appears to be like to following:

AAA BBB CCC
15  1   6
3   83  19
100 4   2

I do know that each document in the collection has identical fields (I mean "field names") in each document. But I have the challenge that I do not know the name of these fields in advance! I chose the ugly way of assigning field names to column_name_list in the for loop. But it appears to be the case that order of fields vary at each step in the for loop.

For instance, the first row is supposed to be "6 1 15" but I see "15 1 6" in the csv file.

How can I read the collection data and save it in a Pandas dataframe (with Python) without knowing field names of the documents in advance?

CodePudding user response：

The following code seems to solve this problem:

import pandas as pd
from google.cloud import firestore

db = firestore.Client()
users = list(db.collection(u'users').stream())

users_dict = list(map(lambda x: x.to_dict(), users))
df = pd.DataFrame(users_dict)

I found it here: https://gist.github.com/romicofre/ee80100b62b5fdbed42218e8239df94e

Any other methods/solutions are more than welcome!

CodePudding user response：

Just add an orderBy.

Take a look at the orderBy function in https://www.npmjs.com/package/firebase-firestore-helper

Disclaimer: I am the creator of this library. It helps to manipulate objects in Firebase Firestore (and adds Cache)

Enjoy!