Files taking *A LOT* more space on disk than the sum of their sizes.

[bga]

One of the tests I always do under Haiku is to download the source inside itself with svn. Right now I am using a 2 Gb VMWare image. Haiku itself takes around 200 Mb on this image and the 1.8 Gb left was usually enough to hold the source tree but with the latest revision (26815), it says device is full. "df" actually shows that 0 bytes are available but if I run "du -h ." inside the directory where I downloaded Haiku, it says the files themselves take only 511 Mb. I am pretty sure the problem is not space wasted due to block sizes as it used to work before and, also, it seems it is taking almost 4 times the actual required space which does not seem usual.

[bga]

FYI, after erasing the directory that contains the Haiku sources with "rm -rf", all the space is reclaimed and I am back at 1.8 Gb free.

[anevilyak]

I would guess this is related to hrev26794.

[bga]

Makes sense but only if we are not trimming down the files on close.

[bga]

I reverted to hrev26793 and the same happens, so it was not hrev26794 that resulted in this.

[scottmc]

Seems like this one would fall into the prop #1 for alpha? I've run into disk full issues on 2GB vmware images and when that happens there's a good chance bit splatter also happens.

[axeld]

Scott, "bit splatter" has nothing to do with a full disk, those are completely different issues. I haven't seen any issues of space waste in BFS yet (but I only looked at some exemplary files in the repository).

To Bruno: what block size are you using for BFS? The standard 2048 bytes or more?

[bga]

2048. I will try checking out the sources in a recent build and show the results here just to update this information.

[bga]

So here it is:

Space before starting check out: 1.7 Gb. Last file checked out before no space left on device message: src/libs/lp_solve/lp_solve/ccc Space used according to "du -h .": 601 Mb

So now the files are taking more than 2 times they actual size. This at least looks better than the first attempt I did. Do you think this can simply be attributed to waste due to block size? I have no hard numbers to prove or disprove it. Maybe calculating the average file size of all files in the tree would give a better insight.

[anevilyak]

Considering things like libroot have tons of tiny files in them that are a few hundred bytes but wind up taking the full 2KB, it's entirely possible that at least parts of the tree take up far more on-disk space than just the sum of the file sizes would indicate.

[Darknesss]

Disregard this if it is totally unrelated, but I thought I'd point out a possibly related thing...

Is it actually the files taking up the space or the directories?

Try creating a shitload of directories and see what happens to the diskspace. At the SkyOS bugtracker, we have a similar bug with SkyFS which is mostly the same as BFS.

I don't have a haiku installation at the moment where I can verify this but perhaps it is related to this bug:

​http://www.skyos.org/bugs/view.php?id=1570

Explanation of Robert Szeleney: Looks like directory inodes are not trimmed on close.

For instance: 10000 empty files require 20MB of diskspace, which is normal. (2048 byte block size)

10000 empty directories require 664MB of diskspace. :) Reason: Whenever a directory is created, disk space is preallocated to speed up file creation. Normally, disk preallocated size should be freed when the directory gets closed. Obvisously, this isn't so. ---end quote

[diver]

It seems #3160 could be related.

[Darknesss]

Replying to diver:

It seems #3160 could be related.

heh, didn't see that bug yet, but you are right, looks like a similar issue

[aljen]

I encounter this today, when building & installing haiku on ~6gb partition Before booting into it, i did on Linux:

aljen@Hitomi:~/Develop/Haiku$ sudo mount /dev/sda7 mnt -t befs
aljen@Hitomi:~/Develop/Haiku$ df -h | grep sda7
/dev/sda7             6,6G  353M  6,2G   6% /home/aljen/Develop/Haiku/mnt

so there is only 353M used, then, i booted into Haiku and run df:

Mount           Type      Total     Free     Flags   Device
--------------- -------- --------- --------- ------- --------------------------
/boot           bfs           6.5G      2.2G QAM-P-W /dev/disk/ata/0/master/2_2

Now after reboot into Linux, i test it again, but now with "Haiku results":

aljen@Hitomi:~/Develop/Haiku$ sudo mount /dev/sda7 mnt -t befs
aljen@Hitomi:~/Develop/Haiku$ df -h | grep sda7
/dev/sda7             6,6G  4,4G  2,2G  67% /home/aljen/Develop/Haiku/mnt

[bga]

In your specific case, it was probably just the swap file being created.

[aljen]

Stupid me, sorry for time waste, i checked and indeed there is a 4gb swap file.. But my second machine at first run created 286mb swap file instead of 4gb (first have 4gb ram, second 3gb), i didn't check that at first place and that's why i thought it was that issue

[axeld]

The directory preallocations are now only 4K since hrev31253.

Directory preallocations are only trimmed when their vnode is flushed, which doesn't happen that often when you have lots of memory. The Haiku source tree contained about 20000 directories - before, this could lead to a loss of 1.2 GB in preallocations. Now it's only 80 MB which I consider acceptable.

A more refined solution would be to trim the preallocations after a certain time of inactivity, but I guess that's not worth the effort.

[axeld]

(In #3160) Thanks for the note.