Django custom queryset class slicing not working-CodePudding

We are using a custom object(pas1_objects). Once we perform some filter operation we are getting the below query set.

all_users_queryset = (
User.pas1_objects.select_related("userlogindetails", "usermeta", "tenant").filter(data_filter)
.exclude(
    userrolepermissions__role_id__in=["PG", "PD"], tenant_id__in=tenant_ids
    )
    .order_by(*sort_array)
)

the output of the above query is -

<KafkaPushQuerySet [<User: User object (bbc306a1-3a75-4e5e-a9b4-6dce0b31b4b3)>, <User: User object (8830ecc1-944a-43e3-a701-331122a26c2b)>, <User: User object (25255aa5-31f9-45d4-9158-5225fba24cb3)>, <User: User object (4a24f883-210b-43a6-9b57-f51fac624a09)>, <User: User object (b6b044c6-cf9b-46f9-91d9-3a31cd4f6d30)>]>

Slicing on this query set is not consistent.

(Pdb) all_users_queryset[0:1] # first slice is giving correct output
<KafkaPushQuerySet [<User: User object (bbc306a1-3a75-4e5e-a9b4-6dce0b31b4b3)>]>
(Pdb) all_users_queryset[1:2] # this should return 2nd obj in the query set but it return 1st object
<KafkaPushQuerySet [<User: User object (bbc306a1-3a75-4e5e-a9b4-6dce0b31b4b3)>]>
(Pdb) all_users_queryset[2:3] # this works fine
<KafkaPushQuerySet [<User: User object (25255aa5-31f9-45d4-9158-5225fba24cb3)>]>
(Pdb) all_users_queryset[0:2] # this is returning 1st two objects but in reversed order.
<KafkaPushQuerySet [<User: User object (8830ecc1-944a-43e3-a701-331122a26c2b)>, <User: User object (bbc306a1-3a75-4e5e-a9b4-6dce0b31b4b3)>]>
(Pdb) all_users_queryset[0:3] # this is returning correct objets and in correct order.
<KafkaPushQuerySet [<User: User object (bbc306a1-3a75-4e5e-a9b4-6dce0b31b4b3)>, <User: User object (8830ecc1-944a-43e3-a701-331122a26c2b)>, <User: User object (25255aa5-31f9-45d4-9158-5225fba24cb3)>]>

What can be causing this inconsistency?

context on this pas1_object - This is basically a custom object which is created by inheriting models.Manager. and we have overridden get_queryset, which return below custom queryset -

class KafkaPushQuerySet(models.query.QuerySet):
    """
    Overriding queryset function to push to kafka
    """

    def delete(self):
        self.produce_messages_to_kafka(action="delete")
        super(KafkaPushQuerySet, self).delete()

    def update(self, **kwargs):
        super(KafkaPushQuerySet, self).update(**kwargs)
        self.produce_messages_to_kafka(action="update")

Thanks!!!

CodePudding user response：

Querysets are evaluated only when you slice them. In other terms the database is hit each time you perform a slicing operation (cf. Django documentation regarding this point).

Then, the question is: why isn't the effect of your order_by method the same for every evaluation? There might be several (non exclusive) explanations, depending on the content of your User database table and on your variable sort_array.

The database has been updated between 2 evaluations of the queryset,
*sort_array corresponds to a tuple which is not long, precise enough for the order to be fixed and unique. From the documentation:

A particular ordering is guaranteed only when ordering by a set of fields that uniquely identify each object in the results. For example, if a name field isn’t unique, ordering by it won’t guarantee objects with the same name always appear in the same order.

A simple solution would be to force the evaluation of your queryset before to perform slicing, by calling list() on it:

my_list = list(User.pas1_objects.select_related("userlogindetails", "usermeta", "tenant")
    .filter(data_filter)
    .exclude(userrolepermissions__role_id__in=["PG", "PD"], tenant_id__in=tenant_ids)
    .order_by(*sort_array)
)
# And then you can slice your list without any risk of inconsistency
my_list[0:1]