VL-Gothic isn't usable in the Terminal with font hinting enabled

[jackburton]

When font hinting is enable, VL-Gothic isn't usable anymore in the Terminal. This font isn't a real "fixed" font, since it contains, like most Japanese fonts, glyphs with normal and "double" width. But the normal glyphs should have the same width, and in fact, it's considered monospaced for the practical uses. This also happen with Konatu Tohaba, although it should be a monospaced font too. The code which checks if the glyphs are fixed is in AppearPrefView::IsFontUsable().

[koki]

Replying to jackburton:

Since some time, Konatu Tohaba isn't usable anymore in the Terminal. This font isn't a real "fixed" font, since it contains, like most Japanese fonts, glyphs with normal and "double" width. But the normal glyphs should have the same width, and in fact, it's considered monospaced for the practical uses. I also tried the VL-Gothic font and it has the same problem, although it should be a monospaced font too.

You will have the same problem with any Japanese font, because even if it is monospaced, as you say, it contain single and double width glyph sets; this is a typographic requirement in the Japanese language and cannot be avoided or circumvented. This exception should taken into account to allow the system and/or applications to specify a Japanese fixed font, but unfortunately that is not the case in Haiku yet.

I tried to explain this problem quite some time ago (on a different bug report), but unfortunately the developers insisted on strictly sticking to the narrow-minded definition of monospaced. As a result, even today you cannot specify a Japanese monospaced font in the Fonts preferences applet (as it will not get listed), which needless to say is problematic.

[koki]

Here is the bug report that I was referring to in my previous comment (which, ironically, you marked as invalid):

​http://dev.haiku-os.org/ticket/947

Hopefully, this problem will be addressed this time.

[axeld]

Replying to koki:

I tried to explain this problem quite some time ago (on a different bug report), but unfortunately the developers insisted on strictly sticking to the narrow-minded definition of monospaced.

That's not narrow minded, that's just sane. Monospaced is monospaced, and Konatu and other Japanese fonts aren't monospaced. Those fonts are "half and full width" fonts, and to support them properly, we need to propagate that attribute, and evaluate it from applications like Terminal.

Applications just cannot regard Japanese fonts as monospaced fonts, as not all characters have the same width, and therefore, might seriously mess up your font rendering if you don't support it. You always need special support for these fonts to make sure they are rendered correctly. Incorrectly categorizing them is just not a very good solution.

[koki]

@axel

The definition of monospaced font that you so strictly stick to is narrow-minded, because it was obviously created by people who do not understand typography beyond the needs of western languages.

In proportional fonts, each character has a specific width such that the combination of characters when forming a word looks more balanced and thus improves readability. On the other hand, in monospaced fonts all characters in a single glyph set have the same width; this is the case with monospaced Japanese fonts too, the only difference being that all Japanese fonts include multiple glyph sets to meet the typographic requirements of their language. But that does not make them proportional fonts.

All other operating systems and Japanese language-aware applications in them recognize and accommodate this difference and expose monospaced Japanese fonts as such. I hope Haiku will do the same. Excluding Japanese fonts from monospaced font selection menus does not solve the problem, it simply hides it.

[jackburton]

Didn't want to start the flame again. I'm perfecly conscious of the problem, since I worked on Terminal, and I was the one who opened ticket 947. It was correctly closed as invalid, as an application has to support a half/double width font to be able to display it correctly, thus it has to rely on BFont::IsFixed() to return false in this case. There is BFont::IsFullAndHalfFixed() for this kind of fonts. The problem is the And this is a different problem than the one discussed in 947. Terminal is checking only a small range of fonts to tell if a font is usable or not, and Konatu Tohaba (but also VL Gothic mono) don't pass this check, because "!" and the space character don't have the same width.

[axeld]

To Koki:

Konatu is not a proportional font (and I don't claim it would be), but it is not a monospaced font either, as not all of its characters have the same width - what's so hard to understand about this?

That it could be exposed in the UI as monospaced font is a completely different matter; for the user, the difference between monospaced vs. half-and-full width font surely doesn't matter at all.

However, on the application level, it has to support this specifically, as it simply cannot hanlde Konatu as a monospaced font. Therefore, it doesn't make any sense to break our perfectly fine API, and replace it with something ambiguous and less powerful that would also break all apps that don't properly support fonts like Konatu.

Calling this differentiation narrow minded just doesn't make any sense.

[koki]

Replying to jackburton:

...and Konatu Tohaba (but also VL Gothic mono) don't pass this check, because "!" and the space character don't have the same width.

That's strange, because both in VL-Gothic and Konatu Tohaba, all characters within any given glyph set should have the same width and in fact they do; to verify, I just opened VL-Gothic with a font editor, and the space and exclamation mark have the same width. Don't have Konatu Tohaba handy, but I am pretty sure it is the same too.

[jackburton]

Replying to koki:

Replying to jackburton:

...and Konatu Tohaba (but also VL Gothic mono) don't pass this check, because "!" and the space character don't have the same width.

That's strange, because both in VL-Gothic and Konatu Tohaba, all characters within any given glyph set should have the same width and in fact they do; to verify, I just opened VL-Gothic with a font editor, and the space and exclamation mark have the same width. Don't have Konatu Tohaba handy, but I am pretty sure it is the same too.

Yeah, I'm pretty sure it should be like this, since, as I said, Konatu Tohaba and VL Gothic (as opposed as VL PGothic, which is proportional) are considered non-proportional/monospaced fonts.

[koki]

Replying to axeld:

To Koki:

Konatu is not a proportional font (and I don't claim it would be), but it is not a monospaced font either, as not all of its characters have the same width - what's so hard to understand about this?

Look, Axel, I am not stupid and I understand what you are saying. But in the real world, a font is either proportional or monospaced, and when a user installs a monospaced font, he/she expects it to appear in the GUI as such. That's all that matters in the end. Period.

If you want to create an internal categorization to support some form of algorithm to make this happen, that's all good; but exposing such categorization to the end user serves no purpose (other than perhaps confusing them). Or do you actually think that you can convince Japanese/Chinese/Korean users that the monospaced fonts they have been using for so many years are actually not monospaced fonts?

However, on the application level, it has to support this specifically, as it simply cannot hanlde Konatu as a monospaced font. Therefore, it doesn't make any sense to break our perfectly fine API, and replace it with something ambiguous and less powerful that would also break all apps that don't properly support fonts like Konatu.

What would exactly happen if Konatsu were handled as a monospaced font? How exactly would that break the API?

Calling this differentiation narrow minded just doesn't make any sense.

It is the definition of monospaced that I am calling narrow-minded, because it is western-centric and does not take into account the typographic needs of several important languages. What's so hard to understand about this?

[koki]

Replying to jackburton:

Yeah, I'm pretty sure it should be like this, since, as I said, Konatu Tohaba and VL Gothic (as opposed as VL PGothic, which is proportional) are considered non-proportional/monospaced fonts.

Why would it not pass the test then?

[jackburton]

Replying to koki:

Replying to jackburton:

Yeah, I'm pretty sure it should be like this, since, as I said, Konatu Tohaba and VL Gothic (as opposed as VL PGothic, which is proportional) are considered non-proportional/monospaced fonts.

Why would it not pass the test then?

I haven't investigated, but I guess there's something wrong with the way we calculate the width in this particular case. That's why I opened this ticket in the first place :-)

[jackburton]

Replying to koki:

Look, Axel, I am not stupid and I understand what you are saying. But in the real world, a font is either proportional or monospaced, and when a user installs a monospaced font, he/she expects it to appear in the GUI as such. That's all that matters in the end. Period.

I can understand, but still, if you eliminate this distinction, apps WILL break. Because they don't support the double width japanese fonts. Of course, we will have to find a way to let the user USE these fonts, without breaking every app.

What would exactly happen if Konatsu were handled as a monospaced font? How exactly would that break the API?

There are apps which can only use monospaced fonts, for various reasons. If you use a font which has some glyphs which are double width, and some single width, the output of those apps will become messed up. Just like when you use a proportional font with an application which doesn't support it. Terminal is written to support such kind of fonts (in theory) so it will still work. But not many apps are like that.

[koki]

Replying to jackburton:

I can understand, but still, if you eliminate this distinction, apps WILL break. Because they don't support the double width japanese fonts. Of course, we will have to find a way to let the user USE these fonts, without breaking every app.

Do you know this for real or is it an assumption based on mere theory?

What would exactly happen if Konatsu were handled as a monospaced font? How exactly would that break the API?

There are apps which can only use monospaced fonts, for various reasons. If you use a font which has some glyphs which are double width, and some single width, the output of those apps will become messed up. Just like when you use a proportional font with an application which doesn't support it. Terminal is written to support such kind of fonts (in theory) so it will still work. But not many apps are like that.

Forgive my ignorance, but if a text view is capable of displaying proportional fonts with various characters widths (which they are), then why should displaying font sets with only 2 character widths be a problem?

[anevilyak]

Replying to koki:

Forgive my ignorance, but if a text view is capable of displaying proportional fonts with various characters widths (which they are), then why should displaying font sets with only 2 character widths be a problem?

Because not every textview is. The one used in things like StyledEdit, etc. (aka BTextView) does, but things like editors and terminals tend to use a significantly simplified variant that only knows how to handle a fixed width font for a number of reasons, mostly pertaining to performance. Handling proportional fonts, especially with line wrapping comes at a relatively significant computational cost, so that overhead is avoided where not necessary. In such a case, the assumption that every character is the same width is used is relied upon to be able to quickly calculate all kinds of things, including which character is under the cursor, what character to line break at, etc. Even having one alternative width in the font will completely break these assumptions and consequently all the functionality that relies on them.

[koki]

@anevilyak

I appreciate the detailed explanation, but there still is two fundamental flaws in the way this bug is being approached.

One is that is the devs are sticking to a definition of monospaced font that for all practical purposes is inaccurate.

The other is that what the devs have been doing so far is hide the cause so that the malfunction does not surface; this will not lead to a solution. You have to expose the problem so that you can actually know where and how things break.

That's why I have been saying all along, in this and the other related bug report, that fonts like Konatu Tohaba should be seen as monospaced at the system level; only then can you know what and how will break, in order to determine how it has to be fixed.

[anevilyak]

Replying to koki:

One is that is the devs are sticking to a definition of monospaced font that for all practical purposes is inaccurate.

No, it is not inaccurate, which is the whole reason I gave that detailed explanation, to try and make that clear. Your arbitrary choice of semantics does not change the fact that not every glyph in the font is the same width and as such it is by definition not a monospaced font. End of story. As Stefano was saying, as a consequence the apps basically have to be aware that such a font is not in fact truly monospaced and work around it to pretend it is one, but the fact remains it is not.

[koki]

Replying to anevilyak:

Replying to koki:

One is that is the devs are sticking to a definition of monospaced font that for all practical purposes is inaccurate.

No, it is not inaccurate, which is the whole reason I gave that detailed explanation...

What part of "for all practical purposes" did you miss?

2 apples + 2 apples equals 4 apples, but that does not mean that 2 small apples + 2 big apples are the same as 4 small apples.

[anevilyak]

Replying to koki:

What part of "for all practical purposes" did you miss?

2 apples + 2 apples equals 4 apples, but that does not mean that 2 small apples + 2 big apples are the same as 4 small apples.

For future reference, being condescending is not the way to go if you want to motivate people to fix a problem. The fact of the matter is that the CJK fonts, and *only* the CJK fonts use a broken definition of monospace that requires hacks to use them in an environment expecting a font to actually adhere to that definition. End of story.

[koki]

Replying to anevilyak:

Replying to koki:

What part of "for all practical purposes" did you miss?

2 apples + 2 apples equals 4 apples, but that does not mean that 2 small apples + 2 big apples are the same as 4 small apples.

For future reference, being condescending is not the way to go if you want to motivate people to fix a problem.

I am not trying to be condescending. I am simply emphasising the key point of my argument, which you conveniently seem to have missed or chose to ignore.

The fact of the matter is that the CJK fonts, and *only* the CJK fonts use a broken definition of monospace that requires hacks to use them in an environment expecting a font to actually adhere to that definition. End of story.

You have a very western-centric view of the problem. This is not uncommon: people who do not fully understand the needs of other cultures or languages come up with this half-assed rules and pretend to impose them universally, which is exactly what's happening here.

If you are going to profess a universal typographic concept and apply it universally as you and the other Haiku devs are doing here, then you need to make sure that it is universally viable. The existing definition of monospaced fonts is not universally viable, and that is exactly where it totally fails. It's the definition that is broke, not the CJK fonts.

[jackburton]

Replying to koki:

If you are going to profess a universal typographic concept and apply it universally as you and the other Haiku devs are doing here, then you need to make sure that it is universally viable. The existing definition of monospaced fonts is not universally viable, and that is exactly where it totally fails. It's the definition that is broke, not the CJK fonts.

But that's exactly the reason why we have a BFont::IsFullAndHalfFixed(), which means the font contains different sets of glyphs with different widths (double, actually), but fixed within the same set. The api is there already (just not implemented, since I don't know how to do this with freetype), there is no need to argue over the definition of monospaced font. I really want people to be able to use Konatu Tohaba (or VL Gothic, since it seems to be a better choice) as monospaced font (and that's why I opened this ticked, and that's why I'm trying to work on this. But changing the meaning of BFont::IsFixed() is not the way to go. Now, please, let's not continue to argue over this within Trac.

[jackburton]

Replying to koki:

Replying to jackburton:

Yeah, I'm pretty sure it should be like this, since, as I said, Konatu Tohaba and VL Gothic (as opposed as VL PGothic, which is proportional) are considered non-proportional/monospaced fonts.

Why would it not pass the test then?

I found the problem being in the glyph hinting. Disabling it in the Appearance preflet lets Konatu (and VL Gothic) pass the test. Anyone has an explanation for this?

[anevilyak]

Replying to koki:

I am not trying to be condescending. I am simply emphasising the key point of my argument, which you conveniently seem to have missed or chose to ignore.

You were in fact being condescending, whether you want to believe it or not, by basically implying I can't do simple math. I wasn't ignoring your key point, just pointing out how from a technical perspective it completely breaks any app that assumes a fixed font is what it claims to be, but since that seems to have gone in one ear and out the other I guess I won't bother trying next time since you'll just throw it back in my face again.

You have a very western-centric view of the problem. This is not uncommon: people

Please don't make that assumption either. I happen to read and speak that language to an extent, and as a consequence I'm well aware of the pitfalls as far as fonts, input and various other things go, I was just trying to point out the simple reality of how it is in our code, right now, today, and why that makes this a non-trivial request, but thanks for taking that and again assuming I have no idea what I'm talking about and throwing it back in my face like you always do.

[axeld]

I guess the technical details have been conveyed, and that's all this bug tracker should care about. So if you need to continue this discussion, as entertaining it might be, please do so personally. Thanks for your time :-)

[axeld]

Replying to jackburton:

I found the problem being in the glyph hinting. Disabling it in the Appearance preflet lets Konatu (and VL Gothic) pass the test. Anyone has an explanation for this?

I would guess it's either because we're using FreeType incorrectly, and/or have bitmap spacing only? Maybe the hinting differentiates between monospaced and proportional fonts as well in some way?

[jackburton]

Replying to axeld:

Replying to jackburton:

I found the problem being in the glyph hinting. Disabling it in the Appearance preflet lets Konatu (and VL Gothic) pass the test. Anyone has an explanation for this?

I would guess it's either because we're using FreeType incorrectly, and/or have bitmap spacing only? Maybe the hinting differentiates between monospaced and proportional fonts as well in some way?

I had a look at GlyphLayoutEngine::LayoutGlyphs() (in the app_server) and there's a nice

// TODO: Implement spacing modes

So I guess that the problem is right there.

[axeld]

On second thought, hinting shouldn't influence the width of monospaced fonts - and also not of those that are half-and-full. This sounds like we need to specifically support this server side (and differentiate between proportional fonts, monospaced fonts, and half-and-full fonts).