I would like to import a CSV in Django. The issue occurs when trying to import based on the attributes. Here is my code:

class Event(models.Model):
    id = models.BigAutoField(primary_key=True)
    amount = models.ForeignKey(Amount, on_delete=models.CASCADE)
    value = models.FloatField()
    space = models.ForeignKey(Space, on_delete=models.RESTRICT)
    time = models.ForeignKey(Time, on_delete=models.RESTRICT)

    class Meta:
        managed = True
        db_table = "event"

class Space(models.Model):
    objects = SpaceManager()

    id = models.BigAutoField(primary_key=True)
    code = models.CharField(max_length=100)
    type = models.ForeignKey(SpaceType, on_delete=models.RESTRICT)
    space_date = models.DateField(blank=True, null=True)

    def natural_key(self):
        return self.code  #   self.type   self.source_date

    def __str__(self):
        return f"{self.name}"

    class Meta:
        managed = True
        db_table = "space"

class Time(models.Model):
    objects = TimeManager()

    id = models.BigAutoField(primary_key=True)
    type = models.ForeignKey(TimeType, on_delete=models.RESTRICT)
    startdate = models.DateTimeField()
    enddate = models.DateTimeField()

    def natural_key(self):
        return self.name

    def __str__(self):
        return f"{self.name}"

    class Meta:
        managed = True
        db_table = "time"

Now, I create the resource that should find the right objects, but it seems it does not enter into ForeignKeyWidget(s) at all:

class AmountForeignKeyWidget(ForeignKeyWidget):
    def clean(self, value, row=None, **kwargs):
        logger.critical("<<<<< {AmountForeignKeyWidget} <<<<<<<")
        name_upper = value.upper()
        amount = Amount.objects.get_by_natural_key(name=name_upper)
        return amount


class SpaceForeignKeyWidget(ForeignKeyWidget):
    def clean(self, value, row, **kwargs):
        logger.critical("<<<<< {SpaceForeignKeyWidget} <<<<<<<")
        space_code = row["space_code"]
        space_type = SpatialDimensionType.objects.get_by_natural_key(row["space_type"])
        try:
            space_date = datetime.strptime(row["space_date"], "%Y%m%d")
        except ValueError:
            space_date = None
        space = Space.objects.get(
            code=space_code, type=space_type, source_date=space_date
        )
        return space


class TimeForeignKeyWidget(ForeignKeyWidget):
    def clean(self, value, row, **kwargs):
        logger.critical("<<<<< {TimeForeignKeyWidget} <<<<<<<")
        time_type = TimeType.objects.get_by_natural_key(row["time_type"])
        time_date = parse_datetime(row["time_date"])
        time = Time.objects.get_or_create(
            type=time_type, startdate=time_date), defaults={...}
        )
        return time


class EventResource(ModelResource):
    amount = Field(
        column_name="amount",
        attribute="amount",
        widget=AmountForeignKeyWidget(Amount),
    )
    space = Field(
        # column_name="space_code",
        attribute="space",
        widget=SpaceForeignKeyWidget(Space),
    )
    time = Field(
        attribute="time",
        widget=TimeForeignKeyWidget(Time),
    )

    def before_import_row(self, row, row_number=None, **kwargs):
        logger.error(f">>>> before_import_row() >>>>>>")
        time_date = datetime.strptime(row["time_date"], "%Y%m%d").date()
        time_type = TimeType.objects.get_by_natural_key(row["time_type"])
        Time.objects.get_or_create(
            type=time_type, startdate=time_date,
            defaults={
                "name": str(time_type)   str(time_date),
                "type": time_type,
                "startdate": time_date,
                "enddate": time_date   timedelta(days=1),
            },
        )

    class Meta:
        model = Event

I added some loggers, but I only print out the log at AmountForeignKeyWidget. The main question is: How to search for objects in Space by attributes (space_code,space_type,space_date) and in Time search and create by (time_date,time_type) A lesser question is why SpaceForeignKeyWidget and TimeForeignKeyWidget are not used?

CodePudding user response：

The main question is: How to search for objects in Space by attributes (space_code,space_type,space_date) and in Time search and create by (time_date,time_type)

It looks like you are searching for these objects correctly, but it might not be being called. Often with import-export you will save yourself a lot of time if you setup your debugger and step through the code.

It could be that there isn't a 'space' or a 'time' column in your source csv. If there are no such fields, then the import process will silently skip this declaration. If you need to create objects if they don't exist, it's probably best to use before_import_row() for this, as you do in your example. Ensure that you use get_or_create() so that re-runs of the import are handled correctly.

Update

I believe the use case you have is that you need to link relations (Time, Space) to an Event instance during import, but there is no single field which identifies the relations. Instead, they are defined by a combination of fields.

This use case can be handled by import-export but it requires overriding the correct functions. We need to create relations if they don't exist, and then link the created relation instances to the model instance. Therefore we need to find a method in the code base which takes both the instance and the row as params. Unfortunately this is not as well defined as it could be in the code base (before_save_instance() would be a good candidate), but there is an method called import_obj() which we can use.

def import_obj(self, obj, data, dry_run, **kwargs):
    # 'obj' is the object instance
    # 'data' is the row data
    # go ahead and create the relation objects
    time_type = TimeType.objects.get_by_natural_key(row["time_type"])
    time_date = parse_datetime(row["time_date"])
    obj.time = Time.objects.get_or_create(
        type=time_type, startdate=time_date), defaults={...}
    )
    # other relation creations omitted...
    super().import_obj(obj, data, dry_run, **kwargs)

A lesser question is why SpaceForeignKeyWidget and TimeForeignKeyWidget are not used?

As above, if there is no 'space' or 'time' column in the source data, then they will never be called.

It shouldn't make a difference but your clean() method declaration does not define row as a kwarg in SpaceForeignKeyWidget and TimeForeignKeyWidget. Change the clean() definition to:

def clean(self, value, row=None, **kwargs):
    #  your implementation here

I can't see that this will fix it but maybe when running in your context it is an issue.

Note that there are some changes you can make to improve your code.

For AmountForeignKeyWidget, if you only need to look up by one value, you can change your resource declaration to this:

class EventResource(ModelResource):
    amount = Field(
        column_name="amount",
        attribute="amount",
        widget=ForeignKeyWidget(Amount, field="name__iexact"),
    )

You don't need any extra logic, and the lookup will be case-insensitive.

CodePudding user response：

I managed to solve all the issues and make proper imports. Following is the code I used:

class EventResource(ModelResource):
    amount = Field(
        column_name="amount",
        attribute="amount",
        widget=ForeignKeyWidget(Amount, field="name__iexact"),
    )
    space_code = Field(
        attribute="space",
        widget=SpaceForeignKeyWidget(Space),
    )
    time_date = Field(
        attribute="time",
        widget=TimeForeignKeyWidget(Time),
    )

    class Meta:
        model = Event

For the amount field I don't need to make a derivative Widget, since it is using only one variable in CSV. For the two others, implementation follows. I noticed that the widgets for the two other variables were not called and the reason is the variable names were non-existent in my CSV file. When I renamed them to the column names existing in the CSV they have been called.

class SpaceForeignKeyWidget(ForeignKeyWidget):
    def clean(self, value, row, **kwargs):
        space_code = row["spacial_code"]
        space_type = SpaceDimensionType.objects.get(type=row["space_type"])
        try:
            space_date = datetime.strptime(row["space_date"], "%Y%m%d")
        except ValueError:
            space_date = None

        space = SpaceDimension.objects.get(
            code=space_code, type=space_type, source_date=space_date
        )
        return space


class TimeForeignKeyWidget(ForeignKeyWidget):
    def clean(self, value, row, **kwargs):
        time_type = TimeDimensionType.objects.get(type=row["time_type"])
        delta = T_TYPES[time_type]

        start_date = datetime.strptime(row["time_date"], "%Y%m%d").date()
        end_date = start_date   timedelta(days=delta)
        time, created = TimeDimension.objects.get_or_create(
            type=time_type,
            startdate=start_date,
            enddate=start_date   timedelta(days=delta),
            defaults={
                "name": f"{time_type}: {start_date}-{end_date}",
                "type": time_type,
                "startdate": start_date,
                "enddate": end_date,
            },
        )
        return temporal

SpaceForeignKeyWidget only searches it the record is existing and returns the object and TimeForeignKeyWidget creates if non-existing and returns the record. This way no need to use before_import_row() and all the logic is localized to this two widgets.