I saw that my Pi was full when I tried to update it with

sudo apt-get update && sudo apt-get upgrade -y

 Error writing to output file - write (28: No space left on device)

I could also see that there was no free storage anymore as the system was unable to allocate the swap file:

systemctl status dphys-swapfile.service

dphys-swapfile.service - dphys-swapfile - set up, mount/unmount, and delete a swap file
   Loaded: loaded (/lib/systemd/system/dphys-swapfile.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2020-06-28 12:35:15 BST; 4min 21s ago
[...]Jun 28 12:35:15 helios dphys-swapfile[327]: want /var/swap=100MByte, restricting to 50% of remaining disk size: 0MBytes
Jun 28 12:35:15 helios systemd[1]: Failed to start dphys-swapfile - set up, mount/unmount, and delete a swap file.

And indeed, the swap file was 0B:

free -h

      total   used   free   shared  buff/cache  available
Mem:  3.8Gi   285Mi  2.7Gi  11Mi    886Mi       3.4Gi
Swap: 0B      0B     0B

Start to investigate with:

sudo df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G   29G     0 100% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm

Or use the version for Inode which won’t include the mounted drives

sudo df -i

Filesystem Inodes  IUsed  IFree   IUse% Mounted on
/dev/root  1895552 328167 1567385 18%   /
devtmpfs   117763  409    117354  1%    /dev
tmpfs      183811  1     183810   1%    /dev/shm

Check, in the output of those commands, that the size of the root partition (/dev/root) is close the the size of your SD (Here above, I have IUsed =~ 32G). If it is not the case, enlarge it with:

sudo raspi-config

7 Advanced Options > A1 Expand Filesystem 

(reboot)

If the size of the root partition is maximum, then investigate to find the very large stuff with:

sudo du -xh / | grep -P “G\t”

1,2G /opt/openhab/userdata
1,2G /opt/openhab
1,2G /opt
1,3G /usr
11G  /var/lib/docker
11G  /var/lib
11G  /var
16G  /mnt/backup
16G  /mnt
29G  /

To only have the size of the first level folders, use:

sudo du -xh –max-depth=1 / | grep -P “G\t”

1.2G /opt
1.3G /usr
11G  /var
16G  /mnt
29G  /

Check, in the output of that command, if there as any folder or file which could be cleaned-up.

You can also us this to navigate into your SD:

sudo mount –bind /  /mnt
sudo ncdu -x /mnt

-- /mnt ------------------------
15,1 GiB [##########] /mnt
10,6 GiB [###### ] /var
1,3 GiB [ ] /usr
1,2 GiB [ ] /opt
415,3 MiB [ ] /home
353,3 MiB [ ] /lib
9,3 MiB [ ] /bin

You can navigate in this table (with keys up and down) and open folders (press enter) to see the details of their content. Exit this table by pressing “q”.

If by any accident, you don’t succeed to delete a large file or folder (especially if located under /media or /mnt), check that it’s not on a mounted drive. Auto-mount are usually in /etc/fstab and may only be :

cat /etc/fstab

You can umount all at once with

sudo umount -a -t cifs -l

 

Regarding packages, You can cleanup some space with:

sudo apt-get autoremove

sudo rm -R /var/cache/

sudo mkdir -p /var/cache/apt/archives/partial
sudo touch /var/cache/apt/archives/lock
sudo chmod 640 /var/cache/apt/archives/lock
sudo apt-get clean

 

If you are using docker,

you can check the space consumed with:

sudo du -sh /var/lib/docker/overlay2

docker system df

you can cleanup some space with:

docker system prune -a -f

docker system prune –all –volumes –force

docker volume rm $(docker volume ls -qf dangling=true)

 

If using GitLab, you can cleanup some space with:

sudo gitlab-ctl registry-garbage-collect

 

Other useful commands:

sudo find / -type f -size +500M -exec ls -lh {} \;

sudo touch /forcefsck ; sudo reboot

sudo resize2fs /dev/mmcblk0p2 ; sudo reboot

After reboot check the resize status with:

systemctl status resize2fs_once.service

If you see the error “Failed to start LSB: Resize the root filesystem to fill partition”, you can disable the resize2fs with:

sudo systemctl disable resize2fs_once

 

What was the issue in my own case: my remote folder for backup was not mounted anymore on /mnt/backup but the backup script run and stored a 16Gb file into the local folder /mnt/backup. I have been able to delete the local backup by commenting the related shared folder in /ect/stab, rebooting and then deleting /mnt/backup.

Et voilà!