Opened 11 years ago

Closed 10 years ago

#2573 closed bug (fixed)

Files taking *A LOT* more space on disk than the sum of their sizes.

Reported by: bga Owned by: axeld
Priority: high Milestone: R1
Component: File Systems/BFS Version: R1/pre-alpha1
Keywords: Cc: obaro.ogbo@…
Blocked By: Blocking: #3160
Has a Patch: no Platform: All


One of the tests I always do under Haiku is to download the source inside itself with svn. Right now I am using a 2 Gb VMWare image. Haiku itself takes around 200 Mb on this image and the 1.8 Gb left was usually enough to hold the source tree but with the latest revision (26815), it says device is full. "df" actually shows that 0 bytes are available but if I run "du -h ." inside the directory where I downloaded Haiku, it says the files themselves take only 511 Mb. I am pretty sure the problem is not space wasted due to block sizes as it used to work before and, also, it seems it is taking almost 4 times the actual required space which does not seem usual.

Change History (18)

comment:1 Changed 11 years ago by bga

FYI, after erasing the directory that contains the Haiku sources with "rm -rf", all the space is reclaimed and I am back at 1.8 Gb free.

comment:2 Changed 11 years ago by anevilyak

I would guess this is related to hrev26794.

comment:3 Changed 11 years ago by bga

Makes sense but only if we are not trimming down the files on close.

comment:4 Changed 11 years ago by bga

I reverted to hrev26793 and the same happens, so it was not hrev26794 that resulted in this.

comment:5 Changed 11 years ago by scottmc

Seems like this one would fall into the prop #1 for alpha? I've run into disk full issues on 2GB vmware images and when that happens there's a good chance bit splatter also happens.

comment:6 Changed 11 years ago by axeld

Scott, "bit splatter" has nothing to do with a full disk, those are completely different issues. I haven't seen any issues of space waste in BFS yet (but I only looked at some exemplary files in the repository).

To Bruno: what block size are you using for BFS? The standard 2048 bytes or more?

comment:7 Changed 11 years ago by bga

  1. I will try checking out the sources in a recent build and show the results here just to update this information.

comment:8 Changed 11 years ago by bga

So here it is:

Space before starting check out: 1.7 Gb. Last file checked out before no space left on device message: src/libs/lp_solve/lp_solve/ccc Space used according to "du -h .": 601 Mb

So now the files are taking more than 2 times they actual size. This at least looks better than the first attempt I did. Do you think this can simply be attributed to waste due to block size? I have no hard numbers to prove or disprove it. Maybe calculating the average file size of all files in the tree would give a better insight.

comment:9 Changed 11 years ago by anevilyak

Considering things like libroot have tons of tiny files in them that are a few hundred bytes but wind up taking the full 2KB, it's entirely possible that at least parts of the tree take up far more on-disk space than just the sum of the file sizes would indicate.

comment:10 Changed 10 years ago by Darknesss

Disregard this if it is totally unrelated, but I thought I'd point out a possibly related thing...

Is it actually the files taking up the space or the directories?

Try creating a shitload of directories and see what happens to the diskspace. At the SkyOS bugtracker, we have a similar bug with SkyFS which is mostly the same as BFS.

I don't have a haiku installation at the moment where I can verify this but perhaps it is related to this bug:

Explanation of Robert Szeleney: Looks like directory inodes are not trimmed on close.

For instance: 10000 empty files require 20MB of diskspace, which is normal. (2048 byte block size)

10000 empty directories require 664MB of diskspace. :) Reason: Whenever a directory is created, disk space is preallocated to speed up file creation. Normally, disk preallocated size should be freed when the directory gets closed. Obvisously, this isn't so. ---end quote

comment:11 Changed 10 years ago by diver

It seems #3160 could be related.

comment:12 in reply to:  11 Changed 10 years ago by Darknesss

Replying to diver:

It seems #3160 could be related.

heh, didn't see that bug yet, but you are right, looks like a similar issue

comment:13 Changed 10 years ago by nastee

Cc: obaro.ogbo@… added

comment:14 Changed 10 years ago by aljen

I encounter this today, when building & installing haiku on ~6gb partition Before booting into it, i did on Linux:

aljen@Hitomi:~/Develop/Haiku$ sudo mount /dev/sda7 mnt -t befs
aljen@Hitomi:~/Develop/Haiku$ df -h | grep sda7
/dev/sda7             6,6G  353M  6,2G   6% /home/aljen/Develop/Haiku/mnt

so there is only 353M used, then, i booted into Haiku and run df:

Mount           Type      Total     Free     Flags   Device
--------------- -------- --------- --------- ------- --------------------------
/boot           bfs           6.5G      2.2G QAM-P-W /dev/disk/ata/0/master/2_2

Now after reboot into Linux, i test it again, but now with "Haiku results":

aljen@Hitomi:~/Develop/Haiku$ sudo mount /dev/sda7 mnt -t befs
aljen@Hitomi:~/Develop/Haiku$ df -h | grep sda7
/dev/sda7             6,6G  4,4G  2,2G  67% /home/aljen/Develop/Haiku/mnt

comment:15 Changed 10 years ago by bga

In your specific case, it was probably just the swap file being created.

comment:16 Changed 10 years ago by aljen

Stupid me, sorry for time waste, i checked and indeed there is a 4gb swap file.. But my second machine at first run created 286mb swap file instead of 4gb (first have 4gb ram, second 3gb), i didn't check that at first place and that's why i thought it was that issue

comment:17 Changed 10 years ago by axeld

Resolution: fixed
Status: newclosed

The directory preallocations are now only 4K since hrev31253.

Directory preallocations are only trimmed when their vnode is flushed, which doesn't happen that often when you have lots of memory. The Haiku source tree contained about 20000 directories - before, this could lead to a loss of 1.2 GB in preallocations. Now it's only 80 MB which I consider acceptable.

A more refined solution would be to trim the preallocations after a certain time of inactivity, but I guess that's not worth the effort.

comment:18 Changed 10 years ago by axeld

Blocking: 3160 added

(In #3160) Thanks for the note.

Note: See TracTickets for help on using tickets.