docker build command took all of the space-CodePudding

I mistakenly ran, docker build command in a folder with a dockerfile, where there were few large folders with few hundred gigs of files.

folder_1 (55 gigs)
folder_2 (182 gigs)
kaldi_ub18_cuda10.Dockerfile

Inside this folder, I ran the command, nvidia-docker build -t nabil/kaldi_sre:ub18cu10 . -f kaldi_ub18_cuda10.Dockerfile

After that, I got docker sending build context to docker daemon 91gb/237gb

Unfortunately, I had only around 90 gigs free in my root /.

After that, the command stopped and it showed me not enough space.

Now, when I run df -h --total, I see my / has 0 space.

udev             94G     0   94G   0% /dev
tmpfs            19G  212M   19G   2% /run
/dev/sda6       1.7T  1.7T     0 100% /
tmpfs            94G  1.8G   92G   2% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            94G     0   94G   0% /sys/fs/cgroup
/dev/sdb1       1.8T  1.5T  213G  88% /home/tit/Data
/dev/sda1       453M  446M     0 100% /boot
tmpfs            19G   24K   19G   1% /run/user/108
tmpfs            19G     0   19G   0% /run/user/1000
/dev/loop3       27M   27M     0 100% /snap/heroku/4068
/dev/loop1       27M   27M     0 100% /snap/heroku/4076
/dev/loop2      100M  100M     0 100% /snap/core/11743
/dev/loop4      100M  100M     0 100% /snap/core/11798
total           3.8T  3.1T  549G  86% -

Output from /var/lib/docker

:/var/lib/docker$ sudo du -h --max-depth=1 | sort
180K    ./network
182G    ./tmp
205G    ./overlay2
20K     ./builder
20K     ./plugins
386G    .
4.0K    ./runtimes
4.0K    ./swarm
4.0K    ./trust
509M    ./containers
53M     ./image
72K     ./buildkit
72K     ./volumes

I am not sure what happened that took all of my space. I tried docker system prune, but it didn't give me my space.

I can not restart the server as there are many apps running on it. How can I get the space?

CodePudding user response：

The files should be in /var/lib/docker/tmp. After removing files from this directory, I'd recommend restarting the docker engine (systemctl restart docker) in case the engine had open file handles to files in that directory, or anything running that expected files in there to exist.

If you're using buildkit, I'd also recommend:

docker builder prune

CodePudding user response：

Running docker system prune --all --force is a good start (documentation).

As seen in comment, be aware it will delete docker images too.

Then, could you check your /var/lib/docker/tmp directory :

Should be safe to clean its content.
Requires root privileges
Used during build process depending of your build context

with gigs of files in your case.
If docker hold files in that directory :

# Assuming your system uses systemd
systemctl stop docker
rm -r /var/lib/docker/tmp/*
systemctl start docker

You may check available disk space like this :

cd /var/lib/docker
du -h --max-depth=1 | sort