Home > Enterprise >  Why in django-import-export doesn't work use_bulk?
Why in django-import-export doesn't work use_bulk?

Time:04-14

I use django-import-export 2.8.0 with Oracle 12c.
Line-by-line import via import_data() works without problems, but when I turn on the use_bulk=True option, it stops importing and does not throw any errors. Why does not it work?

resources.py

class ClientsResources(resources.ModelResource):

    class Meta:
        model = Clients
        fields = ('id', 'name', 'surname', 'age', 'is_active')
        batch_size = 1000
        use_bulk = True
        raise_errors = True

views.py

def import_data(request):
    if request.method == 'POST':
        file_format = request.POST['file-format']
        new_employees = request.FILES['importData']

        clients_resource = ClientsResources()
        dataset = Dataset()

        imported_data = dataset.load(new_employees.read().decode('utf-8'), format=file_format)
        result = clients_resource.import_data(imported_data, dry_run=True, raise_errors=True)

        if not result.has_errors():
            clients_resource.import_data(imported_data, dry_run=False)

    return HttpResponseRedirect(request.META.get('HTTP_REFERER'))

data.csv

id,name,surname,age,is_active
18,XSXQAMA,BEHKZFI,89,Y
19,DYKNLVE,ZVYDVCX,20,Y
20,GPYXUQE,BCSRUSA,73,Y
21,EFHOGJJ,MXTWVST,93,Y
22,OGRCEEQ,KJZVQEG,52,Y

--UPD--

I used django-debug-toolbar and saw a very strange behavior with import-queries.

With Admin Panel doesnt work. I see all importing rows, but next it writes "Import finished, with 5 new and 0 updated clients.", and see this strange queries

Then I use import by my form and here simultaneous situation:

use_bulk by django-import-export (more)

And for comparing my handle create_bulk()

--UPD2--

I've tried to trail import logic and look what I found:

import_export/resources.py

   def bulk_create(self, using_transactions, dry_run, raise_errors, batch_size=None):
        """
        Creates objects by calling ``bulk_create``.
        """
        print(self.create_instances)
        try:
            if len(self.create_instances) > 0:
                if not using_transactions and dry_run:
                    pass
                else:
                    self._meta.model.objects.bulk_create(self.create_instances, batch_size=batch_size)
        except Exception as e:
            logger.exception(e)
            if raise_errors:
                raise e
        finally:
            self.create_instances.clear()

This print() showed empty list in value.

CodePudding user response:

This issue appears to be due to a bug in the 2.x version of django-import-export. It is fixed in v3.

  • The bug is present when running in bulk mode (use_bulk=True)
  • The logic in save_instance() is finding that 'new' instances have pk values set, and are then incorrectly treating them as updates, not creates.
  • I cannot determine how this would happen. It's possible this is related to using Oracle (though I cannot see how).
  • Related