Recommend way and maximum byte size for requests in a data-intensive Django application-CodePudding

I am working on an application that handles datasets with varyiing sizes. For POST requests I do a check on the attached .csv-File for byte size like and if its over 10mb, I return a BAD_REQUEST response:

MAX_FILESIZE_BYTES = 10485760 # 10mb

csvFile = request.data["file"]
if(csvFile.size > MAX_FILESIZE_BYTES):
    raise Exception("File size over " str(MAX_FILESIZE_BYTES))

except Exception as e:
    return Response("Uploaded File bigger than maximum filesize", status=BAD_REQUEST, exception=True)

Is this the recommend way of handling files too big? And is 10mb a reasonable size, or could my application handle much bigger sizes? I use rest_framework, atomic transactions with postgresql and I dont expect to have more much than 50 users at the same time.

CodePudding user response：

Is this the recommend way of handling files too big?

Well, it seems a bit weird that you'd raise an exception just to catch it.

MAX_FILESIZE_BYTES = 10 * 1024 * 1024  # 10 MiB

if request.data["file"].size > MAX_FILESIZE_BYTES:
    return Response(
        "Uploaded file bigger than maximum filesize", 
        status=BAD_REQUEST,
        exception=True,
    )

is simpler.

And is 10mb a reasonable size, or could my application handle much bigger sizes?

As I said in my comment, 10 mb is relatively tiny these days, so unless you're running on a very small machine, or your "quite heavy operations" are very heavy indeed, it's a pretty low limit.

Also, consider the case where your user actually needs to upload a file that's, say, 10.1 megabytes in size – they'll just complain that your app doesn't allow them to do that.

Simply stated: You will either need zero of a thing, one of a thing, or an arbitrary number of the thing. Programmers and architects ignore this at their own peril. Arbitrary fixed limits are a Code Smell.

– Zero-One-Infinity Rule

Sure, if your app allows just anyone to start uploading a gigabyte of data, you might want to set a very high arbitrary "safety" limit, but if it's e.g. an internal tool of some sort, I'd just get rid of the limit altogether. If your user tries to upload a large enough file, the "quite heavy" operation will likely fail, and they'll complain about that, not that your app doesn't let them do their work.