Django uploaded file validation leads to "InMemoryUploadedFile is not supported"-CodePudding

I have a template where the user can choose and upload a file from their computer. The file is eventually uploaded to S3, but is first validated using custom validation to check some contents of the file. For checking the contents, the script reads the lines of the file in forms.py:

from io import TextIOWrapper

class UploadForm(forms.ModelForm):
    class Meta:
        model = MyModel
        fields = ('myfilefield',)

    def clean_myfilefield(self):
        file = self.cleaned_data['myfilefield']
        read_file = TextIOWrapper(self.cleaned_data.get('myfilefield'), encoding='ASCII')
        headerCounter = 0
        for line in read_file:
            if ">" in line:
                headerCounter  = 1
        if headerCounter > 1:
            error = "Custom error message 1"
            raise ValidationError(error)
        if headerCounter == 0:
            error = "Custom error message 2"
            raise ValidationError(error)
        return file

The validation messages are correctly displayed, but if the file passes validation I get this error message (assuming myfile.txt is the uploaded file):

Input myfile.txt of type: <class 'django.core.files.uploadedfile.InMemoryUploadedFile'> is not supported.

The validation and uploading works fine if I leave out the TextIOWrapper part and the error checks. So I guess there's something about reading the file that converts it to a InMemoryUploadedFile, which is not compatible with uploading the file? I tried finding solutions for "converting" the InMemoryUploadedFile back to a "regular" (?) file, but nothing worked.

Other relevant code:

views.py

@login_required
def detail_view(request, id):
    current_object = MyModel.objects.get(id=id)
    context = {}
    context["data"] = current_object

    if request.method == 'POST':
        form = UploadForm(request.POST, request.FILES)
        if form.is_valid():
            form.save()
            return render(request, 'this_page.html', context)
        else:
            return render(request, 'this_page.html', {"form" : form})
    else:
        context["form"] = UploadForm()
        return render(request, 'this_page.html', context)

models.py (only relevant stuff)

class MyModel(models.Model):
    myfilefield = models.FileField(upload_to="media/stuff", blank=True, null=True)

Using Django 3.2

Full traceback:

Environment:


Request Method: POST
Request URL: xxxx

Django Version: 3.2.13
Python Version: 3.9.12
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'storages',
 'myapp',
 'crispy_forms']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/xxxx/views.py", line 111, in detail_view
    form.save()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/forms/models.py", line 468, in save
    self.instance.save()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1415, in execute_sql
    for sql, params in self.as_sql():
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1358, in as_sql
    value_rows = [
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1359, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1310, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/fields/files.py", line 302, in pre_save
    file.save(file.name, file.file, save=False)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/db/models/fields/files.py", line 89, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/django/core/files/storage.py", line 54, in save
    name = self._save(name, content)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/storages/backends/s3boto3.py", line 459, in _save
    obj.upload_fileobj(content, ExtraArgs=params)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/boto3/s3/inject.py", line 725, in object_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/boto3/s3/inject.py", line 636, in upload_fileobj
    return future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/s3transfer/upload.py", line 579, in _submit
    upload_input_manager = self._get_upload_input_manager_cls(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/s3transfer/upload.py", line 546, in _get_upload_input_manager_cls
    raise RuntimeError(

Exception Type: RuntimeError at /xxxx/12
Exception Value: Input myfile.txt of type: <class 'django.core.files.uploadedfile.InMemoryUploadedFile'> is not supported.

CodePudding user response：

You have to provide file-object to TextIOWrapper.

InMemoryUploadedFile is a wrapper around a file object. You can access the file object using the file attribute.

The io.TextIOWrapper object closes the wrapped file when it is garbage collected (eg __del__()). We can detach your TextIOWrapper() object.

from io import TextIOWrapper

class UploadForm(forms.ModelForm):
    ...

    def clean_myfilefield(self):
        my_file = self.cleaned_data['myfilefield']
                                   HERE ⬇️
        read_file = TextIOWrapper(my_file.file, encoding='ASCII')

        ...
        # Here use detach
        read_file.detach()

CodePudding user response：

SOLVED! I needed to close the TextIOWrapper stream by adding read_file.detach() to forms.py.

from io import TextIOWrapper

class UploadForm(forms.ModelForm):
    class Meta:
        model = MyModel
        fields = ('myfilefield',)

    def clean_myfilefield(self):
        file = self.cleaned_data['myfilefield']
        read_file = TextIOWrapper(self.cleaned_data.get('myfilefield'), encoding='ASCII')
        headerCounter = 0
        for line in read_file:
            if ">" in line:
                headerCounter  = 1
        if headerCounter > 1:
            error = "Custom error message 1"
            raise ValidationError(error)
        if headerCounter == 0:
            error = "Custom error message 2"
            raise ValidationError(error)
        read_file.detach()
        return file