I would like to set a file upload limit with fastapi, but I have a problem where I can't check the size of the file before the person uploads the whole file. For example, if there is a 15 MB file upload limit and the person uploads more than 15 MB, I want to prevent them from uploading to the server. I don't want to use Content-Length to prevent it because it won't prevent any attack. I have found different solutions, but I haven't found a system that can check the file before it is uploaded to the system. As a result, if I can't prevent this and the person tries to upload a 100 GB file to the system and I don't have that much space on my machine, what will happen? Thank you in advance
https://github.com/tiangolo/fastapi/issues/362 I've read and tried what's written on this subject, I also tried with chatgpt, but I couldn't find anything.
CodePudding user response：
You describe a problem faced by all web servers. As per dmontagu's response to the fastapi issue, the general solution is to let a mature web server or load balancer enforce it for you. Example: Apache LimitRequestBody . These products are one of several lines of defense on a hostile Internet, so their implementations will hopefully be more resilient than anything you can write.
The client is completely untrustworthy due to the way the Internet is built peer to peer. There is no inherent identification/trust provision in the HTTP protocol (or the Internet structure), so this behaviour must be built into our applications. In order to protect your web API from malicious sized uploads, you would need to provide an authorised client program that can check the source data before transmission, and an authorisation process for your special app to connect to the API to prevent bypassing the authorised client. Such client-side code is vulnerable to reverse-engineering, and many users would object to installing your software for the sake of an upload!
It is more pragmatic to build our web services with inherent distrust of clients and block malicious requests on sight. The linked Apache directive above will prevent the full 100 GB being received, and similar options exist for nginx and other web servers. Other techniques include IP bans for excessive users, authentication to allow you to audit users individually, or some other profiling of requests.
If you must DIY in Python, then Tiangolo's own solution is the right approach. Either you spool to file to limit memory impact as he proposes, or you would have to run an in-memory accumulator on the request body and abort when you hit the threshold. The Starlette documentation talks about how to stream a request body. Something like the following starlette-themed suggestion:
body = b''
async for chunk in request.stream():
body = chunk
if (len(body) > 10000000):
return Response(status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE)
...
In going down this road, you've drained the request body and will need to send it directly to disk, or repackage it for fastapi. fastapi itself exists "above" the untrusted user problem and offers no solutions.
CodePudding user response：
Your request doesn't reach the ASGI app directly. It goes through reverse proxy (Nginx, Apache), ASGI server (uvicorn, hypercorn, gunicorn) before handled by an ASGI app.
Reverse Proxy
For Nginx, the body size is controlled by client_max_body_size, which defaults to 1MB.
For Apache, the body size could be controlled by LimitRequestBody, which defaults to 0.
ASGI Server
The ASGI servers don't have a limit of the body size. At least it's the case for gunicorn, uvicorn, hypercorn.
Large request body attack
This attack is of the second type and aims to exhaust the server’s memory by inviting it to receive a large request body (and hence write the body to memory). A poorly configured server would have no limit on the request body size and potentially allow a single request to exhaust the server.
FastAPI solution
- You could require the Content-Length header and check it and make sure that it's a valid value. E.g.
from fastapi import FastAPI, File, Header, Depends, UploadFile
async def valid_content_length(content_length: int = Header(..., lt=50_000_000)):
return content_length
app = FastAPI()
@app.post('/upload', dependencies=[Depends(valid_content_length)])
async def upload_file(file: UploadFile = File(...)):
# do something with file
return {"ok": True}
note: ⚠️ but it probably won't prevent an attacker from sending a valid Content-Length header and a body bigger than what your app can take ⚠️
- Another option would be to, on top of the header, read the data in chunks. And once it's bigger than a certain size, throw an error.
from typing import IO
from tempfile import NamedTemporaryFile
import shutil
from fastapi import FastAPI, File, Header, Depends, UploadFile, HTTPException
from starlette import status
async def valid_content_length(content_length: int = Header(..., lt=80_000)):
return content_length
app = FastAPI()
@app.post("/upload")
def upload_file(
file: UploadFile = File(...), file_size: int = Depends(valid_content_length)
):
real_file_size = 0
temp: IO = NamedTemporaryFile(delete=False)
for chunk in file.file:
real_file_size = len(chunk)
if real_file_size > file_size:
raise HTTPException(
status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE, detail="Too large"
)
temp.write(chunk)
temp.close()
shutil.move(temp.name, "/tmp/some_final_destiny_file")
return {"ok": True}