2

I am using ubuntu 18.04-4 and want to move a large directory (yocto build directory for my project) from my ~/Desktop to an external drive (ext4 formatted). The external drive is a 512 GB empty drive. Whenever i attempt to copy the folder to the external drive using cp -r or rsync -ah, After hours of copying, I get the following error:

No space left on device (28)

When I check the space on the drive (after the copy fails), I find that it is actually full!

df -hT shows the following 2 relevant lines:

Filesystem     Type      Size  Used Avail Use% Mounted on  
/dev/sda1      ext4      246G  212G   23G  91% /  
/dev/sdb1      ext4      469G  445G   24K 100% /media/builder/WorkSpace

du -sh on my source folder shows that the source is 111 GB.

Before issuing the cp (or rsync) command, df -hT shows:

Filesystem     Type      Size  Used Avail Use% Mounted on  
/dev/sdb1      ext4      469G   73M  445G   1% /media/builder/WorkSpace

So the destination drive is definitely empty.

The suggestion that I have run out of Inodes does not seem to apply to my case. as can be seen from the df -hT output above, my case is actually using all the space.

The destination drive is freshly formatted and is definitely large enough. Why is the copied data much larger than the source folder (and the entire source disk, for that matter)? What could be causing this?

EDIT: The suggestion that I have run out of Inodes does not seem to apply to my case. as can be seen from the df -hT output above, my case is actually using all the space.

The exact commands I tried using are as follows:

sudo cp -r Desktop/Yocto_test /media/builder/Workspace/
rsync -ah /home/builder/Desktop/Yocto_test /media/builder/WorkSpace

The result of the "df" command related to that (target) disk is:

Filesystem     1K-blocks      Used Available Use% Mounted on /dev/sdb1  
491173784 466153780         0 100% /media/builder/WorkSpace

df -i yields:

Filesystem       Inodes    IUsed    IFree IUse% Mounted on /dev/sdb1  
31260672 15285870 15974802   49% /media/builder/WorkSpace

Some other tests requested in the comments:

df -hi | grep -E 'Inodes|sd[ab]1'  
Filesystem     Inodes IUsed IFree IUse% Mounted on  
/dev/sda1         16M  7.4M  8.3M   48% /  
/dev/sdb1         30M   15M   16M   49% /media/builder/WorkSpace  

du -xms ~/Desktop/Yocto_test/ /media/builder/WorkSpace  
113145  /home/builder/Desktop/Yocto_test/  
455157  /media/builder/WorkSpace
roaima
  • 2,889
  • 1
  • 13
  • 27
Dis Play Name
  • 43
  • 6
  • Does this answer your question? [Linux: no space left on device error with 50% free space](https://superuser.com/questions/310076/linux-no-space-left-on-device-error-with-50-free-space) – gronostaj Sep 22 '20 at 12:35
  • I have added an EDIT that explains why i think that is not the case. Thanks for the suggestion. – Dis Play Name Sep 22 '20 at 12:54
  • Is that folder carrying many small files? What's the target file system ... exFAT? – Eugen Rieck Sep 22 '20 at 12:58
  • Please check block sizes: `sudo tune2fs | grep 'Block size'` – gronostaj Sep 22 '20 at 12:59
  • @EugenRieck Yes, the folder has many small files as well as large files. The target is ext4 just like the source. You can see this in the df -hT output i shared that shows the source filesystem (/dev/sda1) and the destination filesystem (/dev/sdb1). – Dis Play Name Sep 22 '20 at 13:01
  • @gronostaj I think you are missing a -l in that command. Anyway I checked the block size for both partitions (source /dev/sda1 and destination /dev/sdb1) and they're both 4096. – Dis Play Name Sep 22 '20 at 13:08
  • I know you say you've checked the number of inodes. Here I'm interested in the number used compared to the source. `df -hi | grep -E 'Inodes|sd[ab]1'` please. Also `du -xms ~/Desktop/directory_you_are_copying /media/builder/WorkSpace` would be interesting to compare – roaima Sep 22 '20 at 14:46
  • Final question for now. Are you copying sparse files? Try `cp --sparse=always ...` or `rsync -S ...` – roaima Sep 22 '20 at 14:49
  • I've another question for you. What are the actual `cp` and `rsync` commands (including any flags and options) you're using? – roaima Sep 24 '20 at 16:52
  • @roaima I have edited the response to include the exact copy commands i tested with. – Dis Play Name Sep 27 '20 at 09:04
  • @roaima df -hi | grep -E 'Inodes|sd[ab]1' >Filesystem Inodes IUsed IFree IUse% Mounted on >/dev/sda1 16M 7.4M 8.3M 48% / /dev/sdb1 30M 15M 16M 49% /media/builder/WorkSpace The "du -xms" command is taking long to execute, I will post the result when it completes. – Dis Play Name Sep 27 '20 at 09:12
  • Please could you add your updates to the question. Put everything in one place (not spread across the comments) so that it's easy for people to read your question -make it easy for us to help you – roaima Sep 27 '20 at 12:02
  • OK. Sorry about that. I'm new here. Is there anything else i should test? I will test the -sparse=always option and update this tomorrow. – Dis Play Name Sep 28 '20 at 18:52
  • 1
    @roaima It still behaves the same even with --sparse=always. I monitored the disk usage from another terminal and when I saw that it reached 50% disk space, I aborted the copy. I can create a backup of the folder with tar -czvf . I still don't have an answer to why it can't simply copy. – Dis Play Name Oct 07 '20 at 15:17
  • 1
    @roaima I have solved the issue finally and have posted my solution as the correct answer. Thanks for all your help! – Dis Play Name Oct 08 '20 at 06:44

2 Answers2

1

I have finally figured out why this was happening, and it was not caused by sparse files, block size issues or even running out of Inodes!

The problem was that Yocto (the build tool that created most of the files in the directory that I'm trying to copy) really loves to use hard links. and most of the millions of files it created are actually hard links to other files in the same directory. Therefore they don't consume additional space.

cp (and rsync) do not preserve hard links by default. when they encounter a hard linked file, they will create a whole new inode for it and will end up multiplying the size of an inode by the number of hard links to it!

This also explains why I could tar czvf the directory. Tar's default behavior is to preserve hard links.

I now can use cp -a to successfully copy my directory to external storage like this:

  sudo cp -a Yocto_test /media/builder/WorkSpace/

I hope this helps someone else with the same issue. Thanks everyone for your suggestions!

Dis Play Name
  • 43
  • 6
0

Most likely you are running into a block size dilemma: The ext4 file system will use a full block even for the smallest file. This implies, that if you copy a small file (e.g. 300 bytes) from a 512B blocked device to a 4K blocked device the used space will quadruple.

Along the same lines, the space used by a directory will quadruple - so a deep folder structure will use up an appreciable amoount of space.

Things you can do: Create a big file on the new disk and assign a loop device to it, then format it with a small block size and use it to store the small files.

Eugen Rieck
  • 19,950
  • 5
  • 51
  • 46
  • 1
    But The block size (4096) and filesystem type (ext4) on both my source and destination partitions are the same. I understand what you're suggesting, but wouldn't the same thing apply to the source filesystem? – Dis Play Name Sep 22 '20 at 13:19