Tracker doesn't load all poses on folders with thousands of files.

[aldeck]

Tracker doesn't load all poses on folders with thousands of files. This problem was first mentioned in #2760's comments, but now it's more clear it's a different bug.

Investigating on #3011 i found some new infos:

I added some debug output to AddPosesTask and apparently GetNextDirents returns 0 although there are still more items in the dir. ​http://dev.haiku-os.org/browser/haiku/trunk/src/kits/tracker/PoseView.cpp#L1263 . It fails after a few thousand files but it's quite variable.

Interestingly, when i tried commenting "model->CloseNode()" here ​http://dev.haiku-os.org/browser/haiku/trunk/src/kits/tracker/PoseView.cpp#L1742 for #3011's investigation, i noticed the problem was worse. That is GetNextDirents would fail after a few hundred files.

Closing model nodes "model->CloseNode()" on each model in PoseView::AddPosesTask loop and reopening it in CreatePoses, made the problem disappear, although hitting performance a bit.

There must be something fishy with models not being all closed by seeing the old debug ifdefs in the related methods. Anyone has an idea why GetNextDirents would fail though?

setting tracker component, might be a kernel/fs bug.

[axeld]

What error does it return?

[aldeck]

It just returns 0 as if it had visited all entries.

[aldeck]

When creating lots of poses, something is going wrong with the way BNodes are opened/closed, further node opening fails with B_NO_MORE_FDS. I'll have a look at that.

Would that explain GetNextDirents giving up though?

[axeld]

There is a natural upper limit of the open file descriptors for every app - if Tracker hits that, it just means it keeps too many files open at once. I remember that has been a problem for file panels under BeOS as well (missing files).

Are you sure that GetNextDirents() give up early? Ie. it should always return all files, it's just that Tracker won't be able to create poses for them all if it hits the file descriptor limit.

At least there is nothing that should get GetNextDirents() to bail out early once the directory is up and running.

After all, a "ls" in the Terminal is using the same kernel call that powers it.

[aldeck]

Replying to axeld:

There is a natural upper limit of the open file descriptors for every app

Yes that's the problem, is it still limited to 256 like in beos? Couldn't find the limit in the sources.

Are you sure that GetNextDirents() give up early? Ie. it should always return all files

I'm pretty sure about that, unless i totally messed up my debug output. Ex: it would stop returning new entries (returning 0) at 8000 entries when there are 12000 in the directory. That's under the stress conditions i mentioned, after lots of NO_MORE_FDS errors. Note also that the file opening is done in a different thread than the (deffective) closing.

Fixing Tracker shouldn't be too hard. However i don't want to give up on that GetNextDirents mystery, i'll probably make a test app for that...

[aldeck]

Closing the node in the AddPoses thread and reopening it on reception works well. I added a small snooze in the AddPoses thread between two sent messages, it's a lot more fluid, the window thread (receiver) was stalling a lot. Doing some cleaning and testing, and i'm ready to commit.

[aldeck]

fixed in hrev28872