By default Docker stores its persistent data in /var/lib/docker folder on Linux systems. This folder is part of the OS disk and in a production (or even dev) setup you might want to store persistent data on a data disk. Many OS Disks come with limited size. On a server that I was working on the /var folder only had 8GB to it.
/dev/mapper/rootvg-varlv 7.8G 7.8G 0 100% /var
It got full fast and then we had a big problem. We had running containers on the machine. What to do now? First we had to attach a data disk to the machine. Once that is done follow the recipe below to migrate Docker’s storage to another folder while keeping everything intact! The instructions below are for Red Hat Enterprise Linux (RHEL) and so should work as-is on CentOS. On other systems you might need some tweaks to the commands.
The new location is /app which resides on a different disk in this writeup.
Step 1: Stop Docker daemon
#-> systemctl stop docker
Step 2: Copy /var/lib/docker to new location
Copy the whole of /var/lib/docker not just the volumes inside it or you will lose your containers etc.
#-> rsync -aP /var/lib/docker /app
Run ls /app/docker to verify everything is there.
#-> ls /app/docker
builder containerd image overlay2 runtimes tmp volumes
buildkit containers network plugins swarm trust
Step 3: Rename /var/lib/docker to /var/lib/docker.old
For safety, always retain the old directory.
#-> mv /var/lib/docker /var/lib/docker.old
To see the file size
#-> du -ch -d 1 /var/lib/docker.old
212K /var/lib/docker.old/containerd
1.1G /var/lib/docker.old/overlay2
72K /var/lib/docker.old/buildkit
1.5M /var/lib/docker.old/containers
20K /var/lib/docker.old/builder
62M /var/lib/docker.old/swarm
4.0K /var/lib/docker.old/tmp
24K /var/lib/docker.old/plugins
995M /var/lib/docker.old/volumes
4.0K /var/lib/docker.old/runtimes
4.0K /var/lib/docker.old/trust
372K /var/lib/docker.old/network
9.5M /var/lib/docker.old/image
2.1G /var/lib/docker.old
2.1G total
WARNING: If you run mv /var/lib/docker /app/docker.old its going to hang for a while because /app is on another filesystem and so it has to really move the files to that folder – it can’t just rename the folder. So don’t do that. Run rsync again if you want to keep an archive of the data on /app.
Step 4: Add symlink from /var/lib/docker to /app/docker
Note the destination folder comes as the first argument to the ln command:
#-> ln -s /app/docker /var/lib/docker
The old containers have /var/lib/docker paths configured in them and so we need to setup this symlink so that we can restart the old containers and everything works.
Verify:
#-> ls -al /var/lib/docker
lrwxrwxrwx 1 root root 11 Sep 29 17:36 /var/lib/docker -> /app/docker
This is the error I got when trying to restart the container if I skipped this step and did not setup the symlink:
#-> docker start middleware-mysql
Error response from daemon: error evaluating symlinks from mount source "/var/lib/docker/volumes/middleware-mysql/_data": lstat /var/lib/docker/volumes: no such file or directory
Error: failed to start containers: middleware-mysql
Step 5: Edit /lib/systemd/system/docker.service and set the --data-root variable
The --data-root is explained below:
--data-root string Root directory of persistent Docker state (default "/var/lib/docker")
Edit following file:
#-> vi /lib/systemd/system/docker.service
and add entry for --data-root pointing it to /app/docker:
ExecStart=/usr/bin/dockerd --data-root /app/docker
Step 6: Reload configuration
#-> systemctl daemon-reload
Step 7: Restart Docker
#-> systemctl start docker
Run systemctl status docker to verify everything is good
Step 8: Restart containers
Finally just restart the containers:
#-> docker start foo
#-> docker start bar
That’s it! Everything is restored.
It was a nerve-racking experience for sure. Now I saw following when I did a df -h
#-> df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 4.0K 3.9G 1% /dev/shm
tmpfs 3.9G 378M 3.6G 10% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/rootvg-rootlv 7.8G 145M 7.2G 2% /
/dev/mapper/rootvg-usrlv 9.8G 2.1G 7.2G 23% /usr
/dev/sda1 976M 83M 826M 10% /boot
/dev/sdc 246G 2.1G 232G 1% /app
/dev/mapper/rootvg-homelv 976M 364M 546M 40% /home
/dev/mapper/rootvg-optlv 2.0G 1.6G 280M 85% /opt
/dev/mapper/rootvg-varlv 7.8G 7.5G 0 100% /var
/dev/mapper/rootvg-tmplv 2.0G 379M 1.5G 21% /tmp
/dev/sdb1 16G 2.1G 13G 14% /mnt/resource
tmpfs 797M 0 797M 0% /run/user/48081
overlay 246G 2.1G 232G 1% /app/docker/overlay2/b65f1433823722f8bba50ff4461f8a8789a9f652f33532326c96bc6215af9d28/merged
shm 64M 0 64M 0% /app/docker/containers/cf51fefa07b118eb252b382947650ad3fbd58a2dbeb2cd7c72e23ab1a0f9e862/mounts/shm
overlay 246G 2.1G 232G 1% /app/docker/overlay2/ca0999cd7dee9d9b3a9911150ff4e6bc06d8c5552a9faa2158f3ecc35cce3a9f/merged
shm 64M 0 64M 0% /app/docker/containers/1f1ff1dc1e4cf96decdd6ea57fe51a72c1edbfdad89771e396f217fc5d294383/mounts/shm
siddharth's space 