Opened 10 years ago

Closed 7 years ago

#4974 closed bug (fixed)

Tracker cant see all files / folders on NTFS partitions

Reported by: streak Owned by: 3dEyes
Priority: normal Milestone: R1/beta1
Component: File Systems/NTFS Version: R1/Development
Keywords: Cc: jstressman@…, korli, mdisreali@…
Blocked By: Blocking: #9060
Has a Patch: no Platform: All

Description

The problem i have is still present on newest nightly builds and first time i found it in 1'st alpha from sept'09. The problem is: When having in NTFS folder a lot of files / folders sometimes [ completely random ] tracker lost some files / folders. If this occurs haiku generates cpu's heavy load [ previously 10% , and while heavy load about 50-60% ].

The important fact is: When tracker's find a folders / files, and you'll reneme it , renamed folder/file disappears and reveal "not seen previously" file or folder.

example:

i have in folder files/folders like:

  • dvd-folder
  • dvd2-folder
  • ex-file.txt
  • ex-file2.txt

and point the tracker into this folder, tracker's find only "dvd-folder" and "ex-file.txt" file

when i rename "ex-file.txt" to example: "1.txt" it disappears and reveals not seen "ex-file2.txt" file.

Black magic, i say. Im guessing its problem with NTFS handling and not the tracker.

btw. This problem could be widely releated to: http://dev.haiku-os.org/ticket/4877 http://dev.haiku-os.org/ticket/4881 [ both tickets are mine ]

Attachments (6)

tracker_missing_files.png (121.1 KB) - added by streak 9 years ago.
syslog_hrev43475-2hn_tracker_popup_froze_viewing_ntfs_volume.txt (70.5 KB) - added by Disreali 7 years ago.
screenshot1-missingfiles4.png (131.4 KB) - added by jstressman 7 years ago.
screenshot1-missingfiles5.png (129.2 KB) - added by jstressman 7 years ago.
screenshot1-missingfiles6.png (156.9 KB) - added by jstressman 7 years ago.
using menu navigation appears to still work for the directory with the hugely misreported size
ntfs-bug-windows3-crop.png (20.2 KB) - added by jstressman 7 years ago.

Download all attachments as: .zip

Change History (43)

comment:1 Changed 10 years ago by streak

Summary: haiku not seeing all files / folders on NTFS partitionshaiku cant see all files / folders on NTFS partitions

comment:2 Changed 10 years ago by streak

Priority: normalhigh

comment:3 Changed 10 years ago by 3dEyes

I think that this is a duplicate of closed my ticket # 4462. Most likely cause of lost files is the same. Check on the night build Haiku.

comment:4 Changed 10 years ago by streak

Im using only nightly builds , as i stated in first sentence of my post. Currently im using nightlies hrev34049 , and the problem is still present.

comment:5 Changed 9 years ago by michaelvoliveira

same problem here

for example:

/Media (ntfs). I have inside mp3, games and movies folders). I only see zines folder.

Type on tracker address bar /Media/mp3 the folders (albums) inside.

comment:6 Changed 9 years ago by streak

Any progress since?

comment:7 Changed 9 years ago by colin

When I'm trying to access my music folder it will not show any subdirectories and some music files, too.
Maybe this gives more insight:
I've enabled the debug output in the libntfs implementation. It revealed this error message:
"Failed to read index block: I/O Error"
Though as my insight into Haiku's filesystem in common and ntfs in concrete is very premature I can't tell what exactly this points to.
I've grepped the libntfs folder for this error message, which appears in two files only: attrib.c and dir.c. Though the I/O Error which it points to is pretty generic and is hardly usable for any further conclusions.
At least it seems the error appears when trying to read the ntfs index. I've run a complete ntfsck on the filesystem, which revealed no errors.
I also checked whether there are any compressed or encrypted files/folders in this music directory, which there aren't any.
Ubuntu 9.10 has no problems in accessing all files/folders, though.

comment:8 Changed 9 years ago by colin

Adding to the comment above: Running a UserBuildConfig'd gcc2-hybrid of hrev35247. The disk we are talking about has a size of 250 GB and has only one primary ntfs partition.

comment:9 Changed 9 years ago by jstressman

Cc: jstressman@… added

Just adding my note here to say that I've been seeing the same problem, and it's still here in build hrev35752 gcc4hybrid.

I have a "downloads" directory on my C: drive (one of several partitions on that drive. the 2nd primary at a glance), and there are 19 folders and 10 files in it in Windows. When I view the folder from Haiku, I see only 12 folders and 7 of the files.

I first noticed this when I downloaded WebPositive.zip in windows and then couldn't find it when I was in Haiku. When I booted back into Windows it was right where it was supposed to be. So I moved the file into a subfolder "Haiku stuff" of my downloads folder and was able to see it fine in Haiku.

Like Colin said, my Ubuntu 9.10 install on this machine has no problem seeing all the files.

comment:10 Changed 9 years ago by streak

Still present in hrev36601

comment:11 Changed 9 years ago by streak

Milestone: R1R1/alpha2
Version: R1/alpha1

comment:12 Changed 9 years ago by michaelvoliveira

But I think that the problem is because ntfs driver keeps trying to load all files/folder (infinite looper)

for example: until de last week wasn't possible to see my movies folder... and in this week voilá....

the blue waiting icon in the status bar continues to monitoring... please observe...

comment:13 Changed 9 years ago by streak

"until de last week wasn't possible to see my movies folder... and in this week voilá...."

Because amount of your files in this folder changes, but the problem still occurs. I cannot specify which files will disappear and which will be visible normally..

comment:14 Changed 9 years ago by streak

I'd love to see it corrected before R2, because it's terrible annoying while working with NTFS and files on it.

comment:15 in reply to:  7 Changed 9 years ago by bonefish

Priority: highnormal

Replying to colin:

When I'm trying to access my music folder it will not show any subdirectories and some music files, too.
Maybe this gives more insight:
I've enabled the debug output in the libntfs implementation. It revealed this error message:
"Failed to read index block: I/O Error"
Though as my insight into Haiku's filesystem in common and ntfs in concrete is very premature I can't tell what exactly this points to.
I've grepped the libntfs folder for this error message, which appears in two files only: attrib.c and dir.c. Though the I/O Error which it points to is pretty generic and is hardly usable for any further conclusions.
At least it seems the error appears when trying to read the ntfs index. I've run a complete ntfsck on the filesystem, which revealed no errors.
I also checked whether there are any compressed or encrypted files/folders in this music directory, which there aren't any.
Ubuntu 9.10 has no problems in accessing all files/folders, though.

Without looking at the code, missing directory entries and an error message that could originate from dir.c sounds very much like they could be related. From the lack of response so far no one seem to feel responsible. Assuming that you can reliably reproduce the problem, I recommend you track this bug down yourself. It shouldn't be too hard to find the exact point of failure by adding error output and maybe a panic() where the current error message is printed. For short turn-around times good choices are qemu under Linux or using userlandfs under Haiku. For the latter a userland add-on has to be built specifically. Have a look how that is e.g. done for packagefs. The advantage of userlandfs is that, when you enable debugging for the userlandfs kernel add-on, you automatically get tracing output for all FS interface calls. That should help you to quickly see which call fails without even touching the NTFS code.

BTW, it's always a good idea to check in the terminal what ll says. It is way more lenient than Tracker and prints error messages for problems it encounters.

comment:16 Changed 9 years ago by michaelvoliveira

streak,

please delete all your binutils and haiku trunk folder

start over again, http://www.haiku-os.org/guides

download sources, configure with --include-gpl-addons --include-3rdparty

build... and all works fine

I can see all my folders and files again

Tested with the latest hrev36814

comment:17 Changed 9 years ago by streak

tested with hrev37131 -> still the same problem [not all files / folders visible on NTFS partition].

BTW. Its releated with #4877 Tracker sometimes skips a files or reports that file not exist while copying from NTFS -> BFS. Its releated with this "not visible files on NTFS partitions.."

comment:18 Changed 9 years ago by streak

still present in hrev37693

comment:19 Changed 9 years ago by streak

Seems to be a TRACKER problem -> [finally i have proof :)] , check attachment

Changed 9 years ago by streak

Attachment: tracker_missing_files.png added

comment:20 Changed 9 years ago by streak

Genesis Commander detects all files in root of NTFS folder, but tracker not.. see tracker_missing_files.png

comment:21 Changed 9 years ago by streak

Component: File Systems/NTFSApplications/Tracker
Owner: changed from 3dEyes to axeld
Summary: haiku cant see all files / folders on NTFS partitionsTracker cant see all files / folders on NTFS partitions

comment:22 in reply to:  20 Changed 9 years ago by bonefish

Replying to streak:

Genesis Commander detects all files in root of NTFS folder, but tracker not.. see tracker_missing_files.png

That's all nice, but as I already wrote:

Replying to bonefish:

BTW, it's always a good idea to check in the terminal what ll says. It is way more lenient than Tracker and prints error messages for problems it encounters.

... so, please open a Terminal, type ll (or even better ls -lai) and report the output.

comment:23 Changed 9 years ago by aldeck

Cc: korli added

Could this have been fixed by hrev39067 ?

comment:24 Changed 9 years ago by streak

I cannot check it because i'm not using Haiku anymore.

comment:25 in reply to:  23 Changed 9 years ago by korli

Replying to aldeck:

Could this have been fixed by hrev39067 ?

For sure, not by this specific commit.

comment:26 Changed 9 years ago by michaelvoliveira

Checking now.. but I noticed that was fixed since 3905 ...

For me, should be closed

Thank you again Korli!! You're saving the day :D

comment:27 Changed 9 years ago by korli

I'll let streak test again, this bug should still be with us.

comment:28 Changed 8 years ago by Disreali

Cc: mdisreali@… added
Keywords: ntfs added

Experiencing on hrev41076-4h. Tracker and ll in Terminal show all the files and dirs in the root, but not in the directories below that.

comment:29 Changed 8 years ago by bonefish

Component: Applications/TrackerFile Systems/NTFS
Keywords: ntfs removed
Owner: changed from axeld to 3dEyes

comment:30 Changed 8 years ago by pulkomandy

Disraeli : still hapenning on more recent nightlies ? I can't see this problem here...

comment:31 Changed 8 years ago by scottmc

Milestone: R1/alpha3R1/beta1
Version: R1/Development

sliding this one to R1/beta while we wait for it to be checked further.

comment:32 in reply to:  30 Changed 8 years ago by Disreali

Replying to pulkomandy:

Disraeli : still happenning on more recent nightlies ? I can't see this problem here...

I haven't noticed this in the last month or so. However, I recall Alpha3 had the issue intermittently, so I don't know if the issue simply does not occur regularly.

comment:33 Changed 7 years ago by Disreali

Experienced again on hrev43475-2hn. I was traversing down the directories of an ntfs volume with right click navmenu, and when I got to the third level down, Tracker navmenu did not display anything in the directory and completely froze. I was not able to open anything else with tracker either. ProcessController showed the Tracker Popup thread was maxed out. I had to use PC to kill Tracker.

I attempting to view the same directory in terminal, but ls froze and spiked cpu and mem usage. I was able to close Terminal regularly.

Attaching KDL stack trace of tracker freeze.

comment:34 Changed 7 years ago by Disreali

I just noticed that while Terminal closed without issue, the ls thread still persisted to run. I tried to force unmount the ntfs volume, but it would not. I had to kill the ls thread before the ntfs volume could be unmounted.

comment:35 Changed 7 years ago by jstressman

Just an update to say it's still here in hrev44293.

I'm adding some shots of the Tracker, terminal, and a Windows CMD prompt all showing the same directory listing.

For the record, while the Japanese file name does cause some problems with Terminal (due to character widths I'd guess), this doesn't appear to be the problem with the listing, as the same thing happened before I changed the filename to that.

On top of leaving out several files and folders like the Tracker, ls in Terminal also doubles one of the folders in the listing, shows one of the files to be hundreds of petabytes in size, and lists the folder contents as a whole to be a staggering 5 zettabytes. (the actual partition size as a whole is only around 464GB... 499,118,182,400 bytes according to Windows) so the size listing is several orders of magnitude off)

I'd be happy to try suggestions to provide more information on this...

Last edited 7 years ago by jstressman (previous) (diff)

Changed 7 years ago by jstressman

Changed 7 years ago by jstressman

Changed 7 years ago by jstressman

using menu navigation appears to still work for the directory with the hugely misreported size

Changed 7 years ago by jstressman

Attachment: ntfs-bug-windows3-crop.png added

comment:36 Changed 7 years ago by luroh

Blocking: 9060 added

comment:37 Changed 7 years ago by 3dEyes

Resolution: fixed
Status: newclosed

Should be fixed in hrev44721

Note: See TracTickets for help on using tickets.