Console backspace doesn't properly handle non-ascii unicode

[Adek336]

A testcase

mkdir xyz
cd xyz
# fine
cd ..
cd Ź<backspace>xyz
# sh: cd: yz: No such file or directory

The examplary 'Ź' character does not have to be typed-in from keyboard, it may very well be copied-pasted.

hrev37286, gcc4+2 hybrid, VBox 3.2

[HaikuBot]

Yes. I have same when forgot switch keymap from russian and type something in terminal and then try use backspace key and get something like Adek336 have.

[bonefish]

This is a duplicate of #5775, but it has a better summary and description. So closing that one.

[bonefish]

(In #5775) Closing as duplicate of #6276, which has the better description and summary.

[heto]

If you try to set LC_CTYPE or any of the other locale variables, bash will warn that setlocale fails. This is probably because Haiku doesn’t ship with any glibc locale files, so applications won’t even know what character encoding is associated with a locale. On Linux (Debian squeeze), these files seem to be installed under /usr/share/i18n.

This also means that any application that relies on C/POSIX locale support for finding out the character encoding will fail, including applications such as svn. And because the locale files are not installed, setting LC_CTYPE will not mean anything.

[anevilyak]

Replying to heto:

If you try to set LC_CTYPE or any of the other locale variables, bash will warn that setlocale fails.

Actually, it's because Haiku doesn't currently implement any of the POSIX locale stuff properly. There is a work-in-progress branch to rectify this, but until it's complete, none of the LC_* stuff will work correctly, regardless of the presence of those files.

[bonefish]

(In #6836) Same bug as #6276. Some readline or bash issue I'd say.

[bonefish]

After a first glance it looks very much like a bash/readline problem. Looking a bit closer...

[bonefish]

Passing on to Oliver. This is a POSIX locale related issue. Haiku's <stdlib.h> defines MB_CUR_MAX to 1. It is also noteworthy that <limits.h> doesn't define MB_LEN_MAX, so in gcc's fixed header it is defined to 1 as well.

[bonefish]

(In #7244) Duplicate of #6276.

[pulkomandy]

Is setting both MB_LEN_MAX and MB_CUR_MAX to 6 a correct fix (seems to be the value needed for utf-8) ? Or is there something more involved ?

MB_LEN_MAX must be a constant so we should use 6 for it, MB_CUR_MAX may be variable depending on the current locale but I'm not sure that's of any use.

[bonefish]

Replying to pulkomandy:

Is setting both MB_LEN_MAX and MB_CUR_MAX to 6 a correct fix (seems to be the value needed for utf-8) ? Or is there something more involved ?

It is probably more involved. Possibly also including changes to the compiler.

To represent any Unicode code point 4 byte UTF-8 suffices, BTW. glibc seems to define MB_LEN_MAX to 16 (sixteen).

[zooey]

With hrev43310, the behaviour has improved with respect to the handling of backspace, but entering the multibyte characters manually still causes problems - the character shows only after pressing space for instance). Pasting multibyte characters and deleting them via backspace works ok, though.

I'll investigate why editing doesn't work ..

[zooey]

Fixed in hrev45307.