Simple iteration on 100 object to create id list is super slow-CodePudding

I'm using pymongo to retrieve a cursor of 100 objects. The object only contain one key because I filter it and the mongodb query is fast enough.

The problem is that the result is a cursor of objects looking like this: {'id': 1588697856264716288}

My goal is to have a list of 'id' but only interested in the value, so something like this: [1588697856264716288,1588697856264716289,...]

In the code below, assume that idd is our cursor containing 100 object. It takes almost 20 second to eventually print mylist

mylist=[]
for idd in ids:
    mylist.append(idd["id"])
print(mylist)

Why is it so slow and how can I speed this up? I'm using Python3.10

I'm trying to create a list of id from a cursor faster. I tried using a simple for loop and maps but it always takes 15-20 seconds

Edit: I just realized what made it slow is the query (eventhough the cursor is returned really fast)

The query is something like this:

ids=mycol.find({"timestamp_ms": {"$gt":"1670979659"}}).limit(100)

In my database timestamp_ms is a string... I guess this is my problem. Should I convert it to a date to make it faster?

CodePudding user response：

For debugging purposes do this:

from time import perf_counter

duration = None
mylist = []
start = perf_counter()
for idd in ids:
    if duration is None:
        duration = perf_counter() - start
        print(f'Duration={duration:.4f}s')
    mylist.append(idd['id'])

If the query is handled lazily - i.e., not until you first try to enumerate the cursor, then this will tell you how long the query takes

CodePudding user response：

If I remember correctly, list comprehension is faster than a for loop in Python (for some cases).

You could try the following solution:

ids = [{'id': 1}, {'id': 2}]

mylist = [id['id'] for id in ids]
print(mylist)

The official docs for list comprehension: https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions