[app_server] Inefficiencies in font loading on startup

[waddlesplash]

Investigating #18669, I discovered that most of the time was spent waiting for app_server to start; and app_server spent most of its time in waiting for the GlobalFontManager lock (from DesktopSettings::_Load()), while the GlobalFontManager was on the initial fonts scan.

hrev57617 seems to be related. I did some basic debugging and found that pretty much the entire font list seems to be loaded twice: I added a debug_printf inside if-check for B_NAME_IN_USE in GlobalFontManager::_AddFont (and disabled FontManager::_AddFont calling FT_Done_Face so it would still be valid) and got this:

NAME IN USE: family 'Noto Sans' style 'Regular'
NAME IN USE: family 'Noto Sans' style 'Bold'
...

etc, for a total of 128 lines. Not only the Noto fonts but also the "DejaVu" fonts (which aren't loaded as part of the preloading of default fonts) show up this way, so, clearly there is something happening twice here when it shouldn't.

But in addition to this bug, I wonder if we could avoid actually loading all the fonts at once on startup? Beyond caching strategies (like fontconfig's cache), can we get away with the "preloading" behavior that we already use for FT_New_Face (passing the index as negative so FreeType only loads some basic metadata)? I'm not sure that the style name and family name are among the metadata loaded, but if they are (or we could otherwise get at them), the we could be "lazy" and avoid loading the full font on startup, but wait for its first real use. This would also save us a lot of memory as well as CPU cycles, especially when a large number of fonts are installed.

[waddlesplash]

Meanwhile, I ran the profiler on startup. It missed a bunch of ticks for GlobalFontManager, but of the ticks it did get (about three-quarters), 97% were spent in _user_map_file -> cache_prefetch_vnode -> packagefs I/O. Though it looks like we have more inefficiencies here, because 75% of the ticks were spent in nvme_disk_io, which should be much faster. packagefs may be using I/O rather inefficiently there, or perhaps the file cache reader is...

[waddlesplash]

Removed at least one lock of the font manager from the startup in hrev57944.

[diver]

Is this related to in any way? https://dev.haiku-os.org/ticket/17574#comment:21

[waddlesplash]

It's somewhat related but having the font manager unlocked during the initial scan would probably cause more issues. For example, applications started early won't see the whole font list, and the DesktopSettings view may randomly not find the actually set fonts from Appearance preferences.

[waddlesplash]

I'm not really sure the profiler results I am getting here are correct. I guess it's possible, especially seeing how other applications are all trying to read from the filesystem too, but the results seem odd and inconsistent between runs. Once I got one with memcpy() using over 50% I think, but now on this run it's 8%. The decompression routines seem consistently around 10% though. Some of these just look like flukes.

The high percentage in nvme_disk_io is pretty consistent, though, so I added some tracing to packagefs to show the read pattern. And we see a lot of things like this:

packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 0 size 10485760
packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 16551936 size 2068
packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 14573568 size 4096
packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 14577664 size 4096
packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 14602240 size 4096
packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 14606336 size 4096
packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 14647296 size 4096
packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 14610432 size 4096
packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 14614528 size 4096
packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 14618624 size 4096
packagefs: READ noto_sans_cjk_jp NotoSansCJKjp-Medium.otf offset 14622720 size 4096

The first line is mmap's precaching, the rest I guess is just the memory mapped area being accessed. This seems to happen for all the font files. I guess this is one area where the variable-width fonts would greatly improve things.

[madmax]

clearly there is something happening twice here when it shouldn't

Zapped with ​https://review.haiku-os.org/c/haiku/+/8001. Not a big difference, though, as the cache and whatever we don't do with duplicate fonts seems to be making wonders. In my font-ridden SSD system, the first pass of that directory takes 7.4 seconds while the second one takes 0.3.

[waddlesplash]

Loading hundreds of MB from packagefs should probably be faster than that (and I'll take a look at that this week) but ideally we wouldn't do it every boot anyway. But switching to using a cache may be a larger refactor than anyone has time for at the moment...

[waddlesplash]

I looked at what we are doing in packagefs here. It is already pretty efficient: we load a full compressed chunk (64KB uncompressed, so varying amounts) and then decompress it and add it to the cache. So really what we should do here is simply not read hundreds of megabytes (or more) of fonts on every app_server startup.

I haven't dug into the code to see what we could do here exactly (take better advantage of the "preloading" behavior, read more things lazily, ...), but I guess the problem has persisted long enough that there isn't a real hurry to solve this before beta5.

[waddlesplash]

(If we do go with a caching strategy, then it probably makes sense to just use fontconfig's cache. We already use fontconfig in app_server for other things, so adapting the font system to make more use of it only makes sense. Plus, a lot of ported apps like to query the cache too, so if app_server made sure to keep the cache up-to-date that would be helpful anyway.)

[nephele]

If we do use fontconfig's cache, app_server should update this on font installation time, in essence when installing a hpkg with a font, or when adding a font to the non-packaged dir by file watching.