No space left on device, even though there is

Question

similar questions on this site did not have a solution for me. I'm developing a web app with symfony, and tried to start a local server, which resulted in a warning and didn't start anything

che6yp@aspire:~/Projects/hometowns_api$ symfony server:start
WARNING Web Server log file cannot be tailed: unable to watch log file: no space left on device
                           
  no space left on device

I have done this a lot before and never had this problem until recently. One time I closed my Chrome and it helped, but it doesn't help anymore. It seems to me like there is a lot of space left!

df -h

Filesystem      Size  Used Avail Use% Mounted on
udev            3,8G     0  3,8G   0% /dev
tmpfs           781M  2,2M  779M   1% /run
/dev/sda5        92G   36G   52G  41% /
tmpfs           3,9G  414M  3,5G  11% /dev/shm
tmpfs           5,0M  4,0K  5,0M   1% /run/lock
tmpfs           3,9G     0  3,9G   0% /sys/fs/cgroup
/dev/loop0       75M   75M     0 100% /snap/beekeeper-studio/64
/dev/loop3      1,0M  1,0M     0 100% /snap/gnome-logs/100
/dev/loop1      180M  180M     0 100% /snap/telegram-desktop/2185
/dev/loop5      143M  143M     0 100% /snap/slack/33
/dev/loop4      1,0M  1,0M     0 100% /snap/gnome-logs/61
/dev/loop2      218M  218M     0 100% /snap/gnome-3-34-1804/60
/dev/loop7      176M  176M     0 100% /snap/postman/129
/dev/loop6       61M   61M     0 100% /snap/core20/634
/dev/loop9       15M   15M     0 100% /snap/gnome-characters/296
/dev/loop8      143M  143M     0 100% /snap/code/49
/dev/loop12     2,3M  2,3M     0 100% /snap/gnome-system-monitor/148
/dev/loop11     163M  163M     0 100% /snap/gnome-3-28-1804/145
/dev/loop10      65M   65M     0 100% /snap/gtk-common-themes/1513
/dev/loop13     179M  179M     0 100% /snap/telegram-desktop/2198
/dev/loop14      98M   98M     0 100% /snap/core/10126
/dev/loop15      56M   56M     0 100% /snap/core18/1932
/dev/loop16     4,2M  4,2M     0 100% /snap/gnome-calculator/406
/dev/loop18     147M  147M     0 100% /snap/code/50
/dev/loop17     162M  162M     0 100% /snap/gnome-3-28-1804/128
/dev/loop20     149M  149M     0 100% /snap/figma-linux/86
/dev/loop19      98M   98M     0 100% /snap/core/10185
/dev/loop21      75M   75M     0 100% /snap/beekeeper-studio/62
/dev/loop23     2,5M  2,5M     0 100% /snap/gnome-calculator/826
/dev/loop26     3,8M  3,8M     0 100% /snap/gnome-system-monitor/100
/dev/loop25     384K  384K     0 100% /snap/gnome-characters/570
/dev/loop22     174M  174M     0 100% /snap/postman/128
/dev/loop29     144M  144M     0 100% /snap/slack/32
/dev/loop27      63M   63M     0 100% /snap/gtk-common-themes/1506
/dev/loop24      56M   56M     0 100% /snap/core18/1885
/dev/loop28      68M   68M     0 100% /snap/sublime-text/85
/dev/sda2        96M   12M   85M  12% /boot/efi
tmpfs           781M   16K  781M   1% /run/user/121
tmpfs           781M   64K  781M   1% /run/user/1000

df -i

Filesystem      Inodes  IUsed   IFree IUse% Mounted on
udev            992678    614  992064    1% /dev
tmpfs           998718   1224  997494    1% /run
/dev/sda5      6119424 688310 5431114   12% /
tmpfs           998718   1206  997512    1% /dev/shm
tmpfs           998718      6  998712    1% /run/lock
tmpfs           998718     18  998700    1% /sys/fs/cgroup
/dev/loop0         133    133       0  100% /snap/beekeeper-studio/64
/dev/loop3         355    355       0  100% /snap/gnome-logs/100
/dev/loop1       26212  26212       0  100% /snap/telegram-desktop/2185
/dev/loop5       23530  23530       0  100% /snap/slack/33
/dev/loop4         354    354       0  100% /snap/gnome-logs/61
/dev/loop2       18513  18513       0  100% /snap/gnome-3-34-1804/60
/dev/loop7       44282  44282       0  100% /snap/postman/129
/dev/loop6       11573  11573       0  100% /snap/core20/634
/dev/loop9         271    271       0  100% /snap/gnome-characters/296
/dev/loop8        4197   4197       0  100% /snap/code/49
/dev/loop12        784    784       0  100% /snap/gnome-system-monitor/148
/dev/loop11      27807  27807       0  100% /snap/gnome-3-28-1804/145
/dev/loop10      63811  63811       0  100% /snap/gtk-common-themes/1513
/dev/loop13      26205  26205       0  100% /snap/telegram-desktop/2198
/dev/loop14      12815  12815       0  100% /snap/core/10126
/dev/loop15      10779  10779       0  100% /snap/core18/1932
/dev/loop16       1550   1550       0  100% /snap/gnome-calculator/406
/dev/loop18       4197   4197       0  100% /snap/code/50
/dev/loop17      27798  27798       0  100% /snap/gnome-3-28-1804/128
/dev/loop20        705    705       0  100% /snap/figma-linux/86
/dev/loop19      12826  12826       0  100% /snap/core/10185
/dev/loop21        133    133       0  100% /snap/beekeeper-studio/62
/dev/loop23       1384   1384       0  100% /snap/gnome-calculator/826
/dev/loop26        734    734       0  100% /snap/gnome-system-monitor/100
/dev/loop25        230    230       0  100% /snap/gnome-characters/570
/dev/loop22      45118  45118       0  100% /snap/postman/128
/dev/loop29      23537  23537       0  100% /snap/slack/32
/dev/loop27      62342  62342       0  100% /snap/gtk-common-themes/1506
/dev/loop24      10775  10775       0  100% /snap/core18/1885
/dev/loop28      14689  14689       0  100% /snap/sublime-text/85
/dev/sda2            0      0       0     - /boot/efi
tmpfs           998718     25  998693    1% /run/user/121
tmpfs           998718     46  998672    1% /run/user/1000

sudo du -h --max-depth=1 /

12M /sbin
0   /sys
13M /bin
4,0K    /mnt
949M    /lib
4,0K    /.cache
du: cannot access '/run/user/1000/doc': Permission denied
du: cannot access '/run/user/1000/gvfs': Permission denied
2,2M    /run
16K /dev
6,4G    /var
128K    /tmp
226M    /opt
11G /snap
126M    /boot
4,0K    /lib64
40K /root
16K /lost+found
4,9G    /usr
15M /etc
4,0K    /media
4,0K    /cdrom
du: cannot access '/proc/28237': No such file or directory
du: cannot access '/proc/28270/task/28270/fd/4': No such file or directory
du: cannot access '/proc/28270/task/28270/fdinfo/4': No such file or directory
du: cannot access '/proc/28270/fd/3': No such file or directory
du: cannot access '/proc/28270/fdinfo/3': No such file or directory
0   /proc
21G /home
4,0K    /srv
46G /

free -m

              total        used        free      shared  buff/cache   available
Mem:           7802        3912         484         472        3405        3117
Swap:          2047         152        1895

ps -a

  PID TTY          TIME CMD
 2278 tty1     00:00:00 gnome-session-b
 2415 tty1     00:00:30 gnome-shell
 2689 tty1     00:00:00 Xwayland
 4164 tty1     00:00:00 ibus-daemon
 4169 tty1     00:00:00 ibus-dconf
 4171 tty1     00:00:00 ibus-x11
 4257 tty1     00:00:00 gsd-xsettings
 4260 tty1     00:00:00 gsd-a11y-settin
 4261 tty1     00:00:00 gsd-clipboard
 4264 tty1     00:00:01 gsd-color
 4265 tty1     00:00:00 gsd-datetime
 4266 tty1     00:00:00 gsd-housekeepin
 4267 tty1     00:00:00 gsd-keyboard
 4268 tty1     00:00:00 gsd-media-keys
 4273 tty1     00:00:00 gsd-mouse
 4276 tty1     00:00:00 gsd-power
 4279 tty1     00:00:00 gsd-print-notif
 4280 tty1     00:00:00 gsd-rfkill
 4281 tty1     00:00:00 gsd-screensaver
 4286 tty1     00:00:00 gsd-sharing
 4290 tty1     00:00:00 gsd-smartcard
 4294 tty1     00:00:00 gsd-sound
 4297 tty1     00:00:00 gsd-wacom
 4307 tty1     00:00:00 ibus-engine-sim
 6614 tty2     00:17:14 Xorg
 6668 tty2     00:00:01 gnome-session-b
 6789 tty2     00:18:58 gnome-shell
 6833 tty2     00:00:35 ibus-daemon
 6843 tty2     00:00:00 ibus-dconf
 6845 tty2     00:00:00 ibus-x11
 6920 tty2     00:00:00 gsd-power
 6922 tty2     00:00:00 gsd-print-notif
 6923 tty2     00:00:00 gsd-rfkill
 6926 tty2     00:00:00 gsd-screensaver
 6927 tty2     00:00:01 gsd-sharing
 6930 tty2     00:00:00 gsd-smartcard
 6940 tty2     00:00:00 gsd-xsettings
 6951 tty2     00:00:00 gsd-sound
 6952 tty2     00:00:00 gsd-wacom
 6958 tty2     00:00:00 gsd-clipboard
 6959 tty2     00:00:00 gsd-a11y-settin
 6961 tty2     00:00:00 gsd-datetime
 6963 tty2     00:00:02 gsd-color
 6966 tty2     00:00:00 gsd-keyboard
 6967 tty2     00:00:01 gsd-housekeepin
 6968 tty2     00:00:00 gsd-mouse
 6972 tty2     00:00:01 gsd-media-keys
 7062 tty2     00:00:03 nautilus-deskto
 7065 tty2     00:00:00 gsd-printer
 7078 tty2     00:00:00 gsd-disk-utilit
 7100 tty2     00:00:11 ibus-engine-sim
 7191 tty2     00:41:54 telegram-deskto
 7374 tty2     00:00:04 gnome-software
 7377 tty2     00:00:00 update-notifier
 7503 tty2     00:00:00 deja-dup-monito
 7542 tty2     00:17:37 chrome
 7548 tty2     00:00:00 cat
 7549 tty2     00:00:00 cat
 7553 tty2     00:00:00 chrome
 7555 tty2     00:00:00 chrome
 7560 tty2     00:00:00 nacl_helper
 7561 tty2     00:00:00 nacl_helper
 7564 tty2     00:00:00 chrome
 7584 tty2     00:23:50 chrome
 7589 tty2     00:03:35 chrome
 7600 tty2     00:00:00 chrome
 7601 tty2     00:00:13 chrome
 7630 tty2     00:00:13 chrome
 7690 tty2     00:02:27 chrome
 7704 tty2     00:02:12 chrome
 7715 tty2     00:00:37 chrome
 7721 tty2     00:00:40 chrome
 7727 tty2     00:01:10 chrome
 7733 tty2     00:01:27 chrome
 8019 tty2     00:07:48 chrome
 8079 tty2     00:02:24 chrome
 8849 tty2     00:00:41 chrome
 8880 tty2     00:00:22 chrome
15904 tty2     00:00:05 chrome
24202 tty2     00:01:57 chrome
27125 tty2     00:00:12 chrome
27650 tty2     00:02:02 chrome
27669 tty2     00:00:02 chrome
27718 tty2     00:00:02 chrome
27756 tty2     00:00:02 chrome
27876 tty2     00:00:01 chrome
28119 tty2     00:00:00 chrome
28302 pts/0    00:00:00 ps

Thank you!

Weird. I was going to suggest checking inodes, but you did that already. Have you tried a restart ? — colindaven, Nov 26 '20 at 13:04
@colindaven I had this exact problem yesterday and a few days before that, and I turn off my pc at night. So I think restart won't help much. I'll try it now, but even if it helps I am sure this problem will reappear. — CHE6yp, Nov 26 '20 at 13:25
Maybe it doesn't have the rights to write the log file, try to find and check the logfile. — colindaven, Nov 26 '20 at 14:06
This can sometimes happen when the Inotify Watches Limit has been reached. Check the current limit with `cat /proc/sys/fs/inotify/max_user_watches`. If the value is somewhere around 8192, consider (temporarily) bumping it up a bit with: `echo "16384" > /proc/sys/fs/inotify/max_user_watches`. — , Nov 26 '20 at 14:53
@Matigo thanks, that did help! But I don't understand it. This problem occurs soon after I boot up, why is the limit reached so fast? — CHE6yp, Dec 07 '20 at 10:14
There are a number of reasons that the limit can be reached quickly, such as using a sync resource like CrashPlan or Dropbox on a directory with thousands of subdirectories. If you have any processes that "watch" directories or files for changes, this will eat up the number of watches available. If you run into this problem often, you may be able to double the 16,384 value once or twice. Be careful not to go *too* high, though, as it can cause a "No space left on device" error during boot. — , Dec 07 '20 at 12:07
I don't think I have any sync resources actually. Also, seems like it's not working anymore and I'm a bit afraid to double it again. — CHE6yp, Dec 16 '20 at 09:33

No space left on device, even though there is

0 Answers0