Opened 16 years ago
Closed 16 years ago
#2614 closed bug (fixed)
Indexes not being correctly maintained.
| Reported by: | bga | Owned by: | bga |
|---|---|---|---|
| Priority: | blocker | Milestone: | R1/alpha1 |
| Component: | File Systems/BFS | Version: | R1/pre-alpha1 |
| Keywords: | Cc: | ||
| Blocked By: | Blocking: | ||
| Platform: | All |
Description (last modified by )
I have been doing experiments on a real Haiku installation and playing with emails I just noticed the indexes are not being correctly maintained.
Here is what I mean:
~> query MAIL:subject=\* /boot/home/development/mdr/AGMSBayesianSpamServer/SoundUncertain /boot/home/development/mdr/outbound_filters /boot/home/config/settings/AGMSBayesianSpam/AGMSBayesianSpamServer\ Settings /boot/home/development/mdr/bemail/ObjectList.h /boot/home/development/mdr/bemail /boot/home/development/mdr/daemon/.cvsignore /boot/home/development/mdr/bemail/Enclosures.cpp /boot/home/development/mdr/lib/crypt.cpp /boot/home/development/spam_server/CVS/Entries /boot/home/development/mdr/config/ConfigWindow.cpp /boot/home/development/mdr/bemail/QueryMenu.cpp /boot/home/development/mdr/CVSROOT/verifymsg /boot/home/development/mdr/CVSROOT/modules /boot/home/development/mdr/inbound_protocols /boot/home/development/mdr/CVSROOT/notify /boot/home/development/mdr/bemail/Content.h /boot/home/development/mdr/bemail/Content.cpp /boot/home/development/mdr/lib/numailkit.cpp /boot/home/development/mdr/lib/status.cp /boot/home/development/mdr/config/ConfigWindow.h /boot/home/development/mdr/bemail/FindWindow.h /boot/home/development/mdr/bemail/Header.cpp /boot/home/development/mdr/bemail/LICENSE /boot/home/development/mdr/bemail/Mail.cpp /boot/home/development/mdr/bemail/Prefs.h /boot/home/development/mdr/inbound_protocols/makefile /boot/home/development/mdr/lib /boot/home/development/mdr/bemail/Signature.h /boot/home/development/mdr/bemail/BeMail.rsrc /boot/home/development/mdr/bemail/Signature.cpp /boot/home/development/mdr/bemail/QueryMenu.h /boot/home/development/mdr/daemon/SlowMenu.h /boot/home/development/mdr/AGMSBayesianSpamServer/BuyLogoLayers.image.x-becasso /boot/home/development/mdr/bemail/Utilities.cpp /boot/home/development/mdr/bemail/WIndex.cpp /boot/home/development/mdr/bemail/Words.cpp /boot/home/development/mdr/bemail/WIndex.h /boot/home/development/mdr/bemail/Words.h /boot/home/development/mdr/bemail/geekspeak /boot/home/development/mdr/bemail/Utilities.h /boot/home/development/mdr/config/E-mail.rsrc /boot/home/development/mdr/daemon/NavMenu.h /boot/home/development/mdr/bemail/BmapButton.h /boot/home/development/mdr/daemon/main.cp
And, just to confirm in one file:
~> listattr /boot/home/development/mdr/daemon/main.cp File: /boot/home/development/mdr/daemon/main.cp Type Size Name ----------- --------- ------------------------------- 0 bytes total in attributes.
I was working with those files before I checked the email so it looks a lot like BFS getting confused about the inodes that it puts on the indexes.
Change History (12)
comment:1 by , 16 years ago
| Description: | modified (diff) |
|---|
comment:2 by , 16 years ago
| Milestone: | R1 → R1/alpha1 |
|---|
comment:3 by , 16 years ago
| Description: | modified (diff) |
|---|
comment:4 by , 16 years ago
comment:5 by , 16 years ago
Forget the last comment about the name and size index, as the name index includes directories while the size index does not. This does not change the problem, anyway.
comment:6 by , 16 years ago
I just confirmed that this also makes the same file be added twice to one index. This messes up the BEOS:APP_SIG index, for example, which makes the Open With menu in Tracker show full paths because it gets the same exacutable twice or more in the index (note it is not 2 exacutables with the same signature, it is the exact same executable twice in the index).
Based on all of the above I guess MDR is just doing thing is a way that triggers one of these (in Inode:WritteAttribute):
TODO: we actually depend on that the contents of "buffer" are constant. If they get changed during the write (hey, user programs), we may mess up our index trees! TODO: for attribute files, we need to log the first BPLUSTREE_MAX_KEY_LENGTH bytes of the data stream, or the same as above might happen.
follow-up: 9 comment:7 by , 16 years ago
Just found a reliable way to reproduce at least part of the problem I am seeing. The script attached to this bug will trigger the "Bad data" error problem all the time here in my computer. I was able to track this error down to the fact that there are still entries in the index that point to files that were deleted. Although this is doing that by creating copies of all applications in the apps folder and checking the BEOS:APP_SIG index, it looks like it would happoen even with 7 apps being copied (but not less) and the index problem happens to all indexes (except for name, size, last_modified, which are handled as a special case).
If someone else could confirm this bug, that would be much appreciated.
comment:8 by , 16 years ago
Attaching files is not working, so here is the script:
#!/bin/sh APP_SIG_NUM_FILES=`query BEOS:APP_SIG=\* | wc -l` echo "Number of readable files in BEOS:APP_SIG index : $APP_SIG_NUM_FILES" echo -n "Copying /boot/beos/apps/* to ./test ... " mkdir test copyattr -d /boot/beos/apps/* test/. echo "Ok." APP_SIG_NUM_FILES=`query BEOS:APP_SIG=\* | wc -l` echo "Number of readable files in BEOS:APP_SIG index : $APP_SIG_NUM_FILES" APP_SIG_NUM_FILES=`query BEOS:APP_SIG=\* | sort -u | wc -l` echo "Number of unique readable files in BEOS:APP_SIG index : $APP_SIG_NUM_FILES" echo -n "Erasing test dir ... " rm -rf test echo "Ok." # This will generate errors in syslog. The index still contains references to # files that were deleted. The funny thing is that not all files resukts in # errors, whcih means that at least some are correctly removed from the index. APP_SIG_NUM_FILES=`query BEOS:APP_SIG=\* | xargs catattr BEOS:APP_SIG | wc -l` echo "Number of readable files in BEOS:APP_SIG index : $APP_SIG_NUM_FILES" APP_SIG_NUM_FILES=`query BEOS:APP_SIG=\* | xargs catattr BEOS:APP_SIG | sort -u | wc -l` echo "Number of unique readable files in BEOS:APP_SIG index : $APP_SIG_NUM_FILES"
comment:9 by , 16 years ago
Replying to bga:
If someone else could confirm this bug, that would be much appreciated.
Script tested. I confirm the behavior. ;-)
comment:10 by , 16 years ago
hrev27180 fixes at least part of this bug (although it can be just hiding a different bug). Will do some more test to see if it fixes the other problems too.
comment:11 by , 16 years ago
| Owner: | changed from to |
|---|---|
| Status: | new → assigned |
comment:12 by , 16 years ago
| Resolution: | → fixed |
|---|---|
| Status: | assigned → closed |
After some extensive testing (who would know downloading emails was a good way to stress-test a file system? :) ) it seems that this is really fixed.
Fixed in hrev27180.
More index problems:
BTW, there are only 110 emails in my system right now and it seems the MAIL:subject index have 152 entries.
Here is what I see on syslog when running the command above:
The system has only been running for a few hours (after installing hrev27057) and I checked for emails (and downloaded then) around 10 times in this period. No unexpected reboot, crash or anything else that would have a bigger chance of corrupting data.
And. just as reported originally (and considering all my email files are inside the mail directory:
So this is definitely the case that files that have nothing to do with a specific index but were accessed (in some cases, even read-only, it seems), were incorrectly added to it.
Note that my example involves MAIL:* indexes, but It is probably also affecting BEOS:APP_SIG, for example, and maybe even last_modified, name and size indexes. In fact:
Considering I have no files that are equal or bigger to the size I used, those numbers shoulkd match, right? Not that I another thing I also see from time to time is the same file added twice to one index so it could be that name has more files than it should, size has less files than it should or even both.
}}}