UTF-8 multibyte characters not handled by UTF8Char::IsSpace()

[scgtrp]

Terminal's UTF8Char::IsSpace() does not properly handle multibyte characters. It was suggested on IRC that I open a ticket here and attach my patch to it.

[yourpalal]

Hi scgtrp, nice patch :) I'm not going to comment on the unicode portions of it (the majority of the patch, I know), because I'm not knowledgeable in that area. Disregarding the functional aspects, this patch looks good, it follows the style guide fairly well, although I did notice style violations on lines 83 and 94 of UTF8Char.h; the '}' and the "else {"/"else if {" should be on the same line.

[scgtrp]

Same patch as before, fixed formatting.

[korli]

What's your source for the ranges you used?

I looked at ​FreeBSD And here are the space class elements: 0x0085 0x00a0 0x1680 0x2000-0x200b 0x2028 0x2029 0x202f 0x205f 0x3000

[scgtrp]

I can't actually remember where I got that list - I want to say Wikipedia but I can't find the page I'm thinking of. Yours seems to be more complete. Will update the patch now.

[scgtrp]

Now with even more spaces!

[bonefish]

Besides that you screwed with the coding style (removing and adding blank lines where you shouldn't have), there's a public BUnicodeChar class in the locale kit that already provides character classification and that should be used here.

[scgtrp]

bonefish: The style guidelines say to "Put exactly two blank lines between functions"; I was advised on IRC to adjust other style problems in files I'm working on. Should I not have done that?

I was unaware of BUnicodeChar's existence - was it not yet implemented when this was written or something?

[bonefish]

Replying to scgtrp:

bonefish: The style guidelines say to "Put exactly two blank lines between functions"; I was advised on IRC to adjust other style problems in files I'm working on. Should I not have done that?

The coding guidelines don't explicitly cover methods implemented inline in class definitions. In those cases only one blank line should be used and there's also no line break before the method name.

I was unaware of BUnicodeChar's existence - was it not yet implemented when this was written or something?

Indeed, the code predates the locale kit.

[mmlr]

I'm removing the flag as the patch shouldn't be applied in this state. As per Ingo's comment the BUnicodeChar should be used for the classification instead.

[siarzhuk]

Fixed in hrev45379. Please check. Thank you for the pointing out the problem and possible solution.