Unable to iterate over all objects in table using objects.all()-CodePudding

I am writing a migration script that will iterate over all objects of a cassandra model (Cats). There are more than 30000 objects of Cat but using Cats.objects.all(), I am only able to iterate over 10000 objects.

qs = Cats.objects.all()

print(qs.count()) # returns 30000
print(len(qs)) # returns 10000

Model:

from django_cassandra_engine.models import DjangoCassandraModel


class Cats(DjangoCassandraModel):
    ...

Cassnadra backend used: django-Cassandra-engine version 1.6.2

CodePudding user response：

The default fetch size (aka page size) is 10K so you'll only get the first 10K rows returned. If you really want to get all the records in the table, you'll need to override the session defaults:

        'cassandra': {
            ...
            'OPTIONS': {
                ...
                'session': {
                    ...
                    'default_fetch_size': 10000
                }
            }
        }

But be careful around setting it to a very high value because it can overload the coordinator node for the request and affect the performance of your cluster.

You should instead iterate through the results in a page, then request the next page until you've reached the end. Cheers!