We have deployed a django server (nginx/gunicorn/django) but to scale the server there are multiple instances of same django application running.

Here is the diagram (architecture):

Each blue rectangle is a Virtual Machine.

HAProxy sends all request to example.com/admin to Server 3.other requests are divided between Server 1 and Server 2.(load balance).

Old Problem:

Each machine has a media folder and when admin Uploads something the uploaded media is only on Server 3. (normal users can't upload anything)

We solved this by sending all requests to example.com/media/* to Server 3 and nginx from Server3 serves all static files and media.

Problem right now

We are also using sorl-thumbnail.

When a requests comes for example.com/,sorl-thumbnail tries to access the media file but it doesn't exist on this machine because it's on Server3.

So now all requests to that machine(server 1 or 2) get 404 for that media file.

One solution that comes to mind is to make a shared partition between all 3 machines and use it as media. Another solution is to sync all media folders after each upload but this solution has problem and that is we have almost 2000 requests per second and sometimes sync might not be fast enough and sorl-thumbnail creates the database record of empty file and 404 happens.

Thanks in advance and sorry for long question.

CodePudding user response：

You should use an object store to save and serve your user uploaded files. django-storages makes the implementation really simple.

If you don’t want to use cloud based AWS S3 or equivalent, you can host your own on-prem S3 compatible object store with minio.

On your current setup I don’t see any easy way to fix where the number of vm s are dynamic depending on load.

If you have deployment automation then maybe try out rsync so that the vm takes care of syncing files with other vms.