Tracker cant see all files / folders on NTFS partitions

[streak]

The problem i have is still present on newest nightly builds and first time i found it in 1'st alpha from sept'09. The problem is: When having in NTFS folder a lot of files / folders sometimes [ completely random ] tracker lost some files / folders. If this occurs haiku generates cpu's heavy load [ previously 10% , and while heavy load about 50-60% ].

The important fact is: When tracker's find a folders / files, and you'll reneme it , renamed folder/file disappears and reveal "not seen previously" file or folder.

example:

i have in folder files/folders like:

• dvd-folder
• dvd2-folder
• ex-file.txt
• ex-file2.txt

and point the tracker into this folder, tracker's find only "dvd-folder" and "ex-file.txt" file

when i rename "ex-file.txt" to example: "1.txt" it disappears and reveals not seen "ex-file2.txt" file.

Black magic, i say. Im guessing its problem with NTFS handling and not the tracker.

btw. This problem could be widely releated to: ​http://dev.haiku-os.org/ticket/4877 ​http://dev.haiku-os.org/ticket/4881 [ both tickets are mine ]

[3dEyes]

I think that this is a duplicate of closed my ticket # 4462. Most likely cause of lost files is the same. Check on the night build Haiku.

[streak]

Im using only nightly builds , as i stated in first sentence of my post. Currently im using nightlies hrev34049 , and the problem is still present.

[michaelvoliveira]

same problem here

for example:

/Media (ntfs). I have inside mp3, games and movies folders). I only see zines folder.

Type on tracker address bar /Media/mp3 the folders (albums) inside.

[streak]

Any progress since?

[colin]

When I'm trying to access my music folder it will not show any subdirectories and some music files, too.
Maybe this gives more insight:
I've enabled the debug output in the libntfs implementation. It revealed this error message:
"Failed to read index block: I/O Error"
Though as my insight into Haiku's filesystem in common and ntfs in concrete is very premature I can't tell what exactly this points to.
I've grepped the libntfs folder for this error message, which appears in two files only: attrib.c and dir.c. Though the I/O Error which it points to is pretty generic and is hardly usable for any further conclusions.
At least it seems the error appears when trying to read the ntfs index. I've run a complete ntfsck on the filesystem, which revealed no errors.
I also checked whether there are any compressed or encrypted files/folders in this music directory, which there aren't any.
Ubuntu 9.10 has no problems in accessing all files/folders, though.

[colin]

Adding to the comment above: Running a UserBuildConfig'd gcc2-hybrid of hrev35247. The disk we are talking about has a size of 250 GB and has only one primary ntfs partition.

[jstressman]

Just adding my note here to say that I've been seeing the same problem, and it's still here in build hrev35752 gcc4hybrid.

I have a "downloads" directory on my C: drive (one of several partitions on that drive. the 2nd primary at a glance), and there are 19 folders and 10 files in it in Windows. When I view the folder from Haiku, I see only 12 folders and 7 of the files.

I first noticed this when I downloaded WebPositive.zip in windows and then couldn't find it when I was in Haiku. When I booted back into Windows it was right where it was supposed to be. So I moved the file into a subfolder "Haiku stuff" of my downloads folder and was able to see it fine in Haiku.

Like Colin said, my Ubuntu 9.10 install on this machine has no problem seeing all the files.

[streak]

Still present in hrev36601

[michaelvoliveira]

But I think that the problem is because ntfs driver keeps trying to load all files/folder (infinite looper)

for example: until de last week wasn't possible to see my movies folder... and in this week voilá....

the blue waiting icon in the status bar continues to monitoring... please observe...

[streak]

"until de last week wasn't possible to see my movies folder... and in this week voilá...."

Because amount of your files in this folder changes, but the problem still occurs. I cannot specify which files will disappear and which will be visible normally..

[streak]

I'd love to see it corrected before R2, because it's terrible annoying while working with NTFS and files on it.

[bonefish]

Replying to colin:

When I'm trying to access my music folder it will not show any subdirectories and some music files, too.
Maybe this gives more insight:
I've enabled the debug output in the libntfs implementation. It revealed this error message:
"Failed to read index block: I/O Error"
Though as my insight into Haiku's filesystem in common and ntfs in concrete is very premature I can't tell what exactly this points to.
I've grepped the libntfs folder for this error message, which appears in two files only: attrib.c and dir.c. Though the I/O Error which it points to is pretty generic and is hardly usable for any further conclusions.
At least it seems the error appears when trying to read the ntfs index. I've run a complete ntfsck on the filesystem, which revealed no errors.
I also checked whether there are any compressed or encrypted files/folders in this music directory, which there aren't any.
Ubuntu 9.10 has no problems in accessing all files/folders, though.

Without looking at the code, missing directory entries and an error message that could originate from dir.c sounds very much like they could be related. From the lack of response so far no one seem to feel responsible. Assuming that you can reliably reproduce the problem, I recommend you track this bug down yourself. It shouldn't be too hard to find the exact point of failure by adding error output and maybe a panic() where the current error message is printed. For short turn-around times good choices are qemu under Linux or using userlandfs under Haiku. For the latter a userland add-on has to be built specifically. Have a look how that is e.g. done for packagefs. The advantage of userlandfs is that, when you enable debugging for the userlandfs kernel add-on, you automatically get tracing output for all FS interface calls. That should help you to quickly see which call fails without even touching the NTFS code.

BTW, it's always a good idea to check in the terminal what ll says. It is way more lenient than Tracker and prints error messages for problems it encounters.

[michaelvoliveira]

streak,

please delete all your binutils and haiku trunk folder

start over again, ​http://www.haiku-os.org/guides

download sources, configure with --include-gpl-addons --include-3rdparty

build... and all works fine

I can see all my folders and files again

Tested with the latest hrev36814

[streak]

tested with hrev37131 -> still the same problem [not all files / folders visible on NTFS partition].

BTW. Its releated with #4877 Tracker sometimes skips a files or reports that file not exist while copying from NTFS -> BFS. Its releated with this "not visible files on NTFS partitions.."

[streak]

still present in hrev37693

[streak]

Seems to be a TRACKER problem -> [finally i have proof :)] , check attachment

[streak]

Genesis Commander detects all files in root of NTFS folder, but tracker not.. see tracker_missing_files.png

[bonefish]

Replying to streak:

Genesis Commander detects all files in root of NTFS folder, but tracker not.. see tracker_missing_files.png

That's all nice, but as I already wrote:

Replying to bonefish:

BTW, it's always a good idea to check in the terminal what ll says. It is way more lenient than Tracker and prints error messages for problems it encounters.

... so, please open a Terminal, type ll (or even better ls -lai) and report the output.

[aldeck]

Could this have been fixed by hrev39067 ?

[streak]

I cannot check it because i'm not using Haiku anymore.

[korli]

Replying to aldeck:

Could this have been fixed by hrev39067 ?

For sure, not by this specific commit.

[michaelvoliveira]

Checking now.. but I noticed that was fixed since 3905 ...

For me, should be closed

Thank you again Korli!! You're saving the day :D

[korli]

I'll let streak test again, this bug should still be with us.

[Disreali]

Experiencing on hrev41076-4h. Tracker and ll in Terminal show all the files and dirs in the root, but not in the directories below that.

[pulkomandy]

Disraeli : still hapenning on more recent nightlies ? I can't see this problem here...

[scottmc]

sliding this one to R1/beta while we wait for it to be checked further.

[Disreali]

Replying to pulkomandy:

Disraeli : still happenning on more recent nightlies ? I can't see this problem here...

I haven't noticed this in the last month or so. However, I recall Alpha3 had the issue intermittently, so I don't know if the issue simply does not occur regularly.

[Disreali]

Experienced again on hrev43475-2hn. I was traversing down the directories of an ntfs volume with right click navmenu, and when I got to the third level down, Tracker navmenu did not display anything in the directory and completely froze. I was not able to open anything else with tracker either. ProcessController showed the Tracker Popup thread was maxed out. I had to use PC to kill Tracker.

I attempting to view the same directory in terminal, but ls froze and spiked cpu and mem usage. I was able to close Terminal regularly.

Attaching KDL stack trace of tracker freeze.

[Disreali]

I just noticed that while Terminal closed without issue, the ls thread still persisted to run. I tried to force unmount the ntfs volume, but it would not. I had to kill the ls thread before the ntfs volume could be unmounted.

[jstressman]

Just an update to say it's still here in hrev44293.

I'm adding some shots of the Tracker, terminal, and a Windows CMD prompt all showing the same directory listing.

For the record, while the Japanese file name does cause some problems with Terminal (due to character widths I'd guess), this doesn't appear to be the problem with the listing, as the same thing happened before I changed the filename to that.

On top of leaving out several files and folders like the Tracker, ls in Terminal also doubles one of the folders in the listing, shows one of the files to be hundreds of petabytes in size, and lists the folder contents as a whole to be a staggering 5 zettabytes. (the actual partition size as a whole is only around 464GB... 499,118,182,400 bytes according to Windows) so the size listing is several orders of magnitude off)

I'd be happy to try suggestions to provide more information on this...

[jstressman]

using menu navigation appears to still work for the directory with the hugely misreported size

[3dEyes]

Should be fixed in hrev44721