Due to a loop before pagination Django paginator takes too much time. How can I solve this problem?-CodePudding

I was asked to map the object customer. I was trying to do the pagination before the loop but I don't know how to do it, because I think that you need to pass all the data to the paginator when creating.

This is my view

I think the problem is that when I call the function "get_customer_view_data" it runs the loop inside this function and I believe this happens everytime I change page on the paginator, causing the delay

class CustomersView(AdminStaffRequiredMixin, TemplateView):
    template_name = 'customers/tables.html'

    def get(self, request, activeCustumers, *args, **kwargs):
        controller = CustomerViewController()

        date1 = request.GET.get('date1', 1)
        date2 = request.GET.get('date2', 1)
        customer_view_data = controller.get_customer_view_data(activeCustumers, date1, date2)

        page = request.GET.get('page', 1)

        paginator = Paginator(customer_view_data, 10)
        try:
            customers_data = paginator.page(page)
        except PageNotAnInteger:
            customers_data = paginator.page(1)
        except EmptyPage:
            customers_data = paginator.page(paginator.num_pages)

        context = {'object_list': customers_data, 'num': len(customer_view_data)}

        return render(request, self.template_name, context)

And this is my controller where I map the data:

class CustomerViewController(object):


    def get_customer_view_data(self, get_active_custumers,date1,date2):
        data = []
        dates = self.set_dates(date1, date2)
        if get_active_custumers == 1:
            obj = Organization.objects.filter(organizationmainapp__application__appinfoforstore__status=2,
                                              deleted=False,
                                              status=True, to_deleted=False)
        else:
            obj = Organization.objects.all()

        for o in obj:
            customer_view_data = Customer()

            customer_view_data.Organization_id = o.id
            customer_view_data.Organization_name = o.name

            try:
                customer_view_data.monthly_price_plan = o.organizationmainapp.application.applicationselectedplan.price_plan.monthly_price
            except Exception as e:
                print(e)
            try:
                if o.organizationmainapp.application.applicationselectedplan.price_plan.transaction_percent_price is not None:
                    customer_view_data.commission = o.organizationmainapp.application.applicationselectedplan.price_plan.transaction_percent_price
                else:
                    customer_view_data.commission = o.organizationmainapp.application.applicationselectedplan.price_plan.transaction_fixed_price
            except Exception as e:
                print(e)

            try:
                customer_view_data.App_name = o.organizationmainapp.application.appinfoforstore.store_name
            except Exception as e:
                print(e)

            try:
                customer_view_data.plan = o.organizationmainapp.application.applicationselectedplan.plan_name_stamp
            except Exception as e:
                print(e)

            try:
                customer_view_data.Total_last_Month_sales = self.get_total_sales(o.id,dates['start_date'],dates['end_date'])['total']
            except Exception as e:
                print(e)

            try:
                customer_view_data.Total_last_Month_sales_without_shipping_cost = \
                    self.get_total_sales(o.id, dates['start_date'], dates['end_date'])['total_without_shipping_cost']
            except Exception as e:
                print(e)
            try:
                customer_view_data.Main_mail = o.organizationmainapp.application.user.email
            except Exception as e:
                print(e)
            data.append(customer_view_data)

        return data

CodePudding user response：

sorry

This is how I solved the problem. First I create the paginator and then I map the data. For pagination purposes I also pass the "organization_data" to the context. This is not the actual code but is something like this:

My view

class CustomersView(AdminStaffRequiredMixin, TemplateView):
template_name = 'customers/tables.html'

def get(self, request, *args, **kwargs):
    controller = CustomerViewController()
    organizations = controller.get_customer_view_data()
    
    page = request.GET.get('page', 1)

    paginator = Paginator(organizations, 15)
    try:
        organization_data = paginator.page(page)
    except PageNotAnInteger:
        organization_data = paginator.page(1)
    except EmptyPage:
        organization_data = paginator.page(paginator.num_pages)

    customers_data = controller.map_view_data(organization_data.object_list)
    # loop over each organization and return customers data
    context = {'object_list': customers_data, 'organization': organization_data}

    return render(request, self.template_name, context)

CodePudding user response：

The paginator doesn't need ALL the data.. it just needs to know how many rows of data there will be. Your queryset (in your code, you call this obj, but a more clear, common name for it would be qs or queryset).. can be passed directly to Paginator() and the it will call count() on your QuerySet to find out how many rows will be selected.

I say will be selected, because the paginator doesn't fetch all the data from the database in order to count it.. it just asks the database for a count and gets a number back.. an operation which is MUCH faster than actually fetching the data.

The paginator then fetches only the rows it needs to display the current page.

What I would do in your shoes is to eliminate your CustomerViewController completely. The CustomerView class itself is your controller here. I would put the code that creates and filters the queryset directly into the view, and then code that organizes the data for each row into a method of your Organization class.

Then, just pass the queryset to Paginator.

As you display each Organization instance within your template, call (from inside the template) the method that returns the data you need. That way you don't actually touch any rows that you won't be displaying on that page.

Avoid touching the queryset, just pass it to the paginator and then use the paginator. For example.. do not call len() on your queryset as you are doing. Because the queryset is not a list.. it doesn't contain any data until you start iterating over it, or until you do certain things, for example if you construct a list from it, or call len() on it.. then you will trigger the database query and the data will be loaded. You don't want that to happen when you're doing pagination.