mbtowc() is unusable

[schily]

mbtowc() returns 0 for ASCII char input. This is only allowed in case that the input string points to a null character, but it happens for any ASCII char.

There seem to be other bugs in mwtomb() and in wcwidth(), but I did not yet inverstigate in them.

[zooey]

I have tried, but failed to reproduce any error. The testmb.c test program of glibc invokes mbtowc() and that seems to work just fine.

Could you please post an invocation of mbtowc() that fails for you?

[schily]

Where is this program?

Could you tell me why this test program works for you?

The program that does work is the command line editor from bsh

See ​ftp://ftp.berlios.de/pub/schily/schily-dist-pre2.tar.bz2

the code is in the file bsh/inputc.c and this code was written according to the POSIX standard and works on any other OS I was able to test until now: Solaris, Linux, HP-UX, Mac OS X......

The behavior seen with the mbtowc() from Haiku (returning 0 although the string input points to a valid 7 bit ASCII character) is not permitted.

[zooey]

glibc test file that invokes mbtowc()

[zooey]

I have attached testmb.c to this ticket.

Which revision have you used when you noticed the problem? I am asking because we have changed our wchar_t type in hrev31443 (from short to int). If you haven't updated beyond that, please do so (and make sure to update your buildtools, too).

[schily]

Thank you for forwarding the test program!

The test program is not correct:

1) calling mbtowc(%wc, &c, MB_CUR_MAX) is illegal as there is no valid string

in the second parameter.

2) The test program does not set up a locale. This does never apply to

correct software that uses wide chars. You need to first call something like:

if (setlocale(LC_ALL, "") == NULL)

error("Bad locale in inital environment.\n");

[zooey]

Replying to schily:

Thank you for forwarding the test program!

The test program is not correct:

1) calling mbtowc(%wc, &c, MB_CUR_MAX) is illegal as there is no valid string

in the second parameter.

POSIX doesn't say anything about that parameter being a valid c-string, but I adjusted it anyway ...

2) The test program does not set up a locale. This does never apply to

correct software that uses wide chars. You need to first call something like:

if (setlocale(LC_ALL, "") == NULL)

error("Bad locale in inital environment.\n");

... adding that doesn't change anything either: testmb.c runs through without failure for me.

So,

• which revision of haiku are you using?
• which revision of haiku's buildtools are you using?
• does testmb.c work for you?

[schily]

I just tested the program test,b.c and as expected, it works on Solaris and it fails on Haiku:

mbstowcs FAILED! wcstombs FAILED! mblen 1 FAILED! conversion to wide failed, result: 0 Test FAILED!

BTW: am using the latest version which is hrev31644

[anevilyak]

Did you update your build tools before building that version? It sounds like you're using a cross compiler toolchain that hasn't been updated with the wchar_t changes.

[schily]

Do you like to tell me that you deliver a non-working GCC together with Haiku?

Anyway, the autoconf results tell me that sizeof(L'a') is 4 as well as sizeof(wchar_t). In addition, the specific problem cannot be a rsult of such size problems as the function mbtowc() cannot know about this and the incorrect return value is a result from interpreting the input string that is an array of 8 bit chars.

As mentioned before, I asume that the internal tables used by glibc are set up incorrectly.

Another problem is that Haiku does not set any if the LC_* or LANG environment variables and an application thus would in a POSIX compliant anvironment asume the "C" locale which could be either "ISO-646" or "ISO-8859-1" coding.

As the problem is easy to reproduce even with your small test program, it should be easy to fix....

[anevilyak]

Replying to schily:

Do you like to tell me that you deliver a non-working GCC together with Haiku?

This isn't about the version that comes a development package in Haiku. It's about the one used to build the system itself. As you haven't specified where your copy of Haiku comes from, I assume you're cross compiling it yourself. If so, you would have needed to svn update your copy of the buildtools/trunk subdir and rerun ./configure --build-cross-tools(-gcc4), as otherwise the libs would be out of sync with the headers and/or compiler builtins.

Anyway, the autoconf results tell me that sizeof(L'a') is 4 as well as sizeof(wchar_t). In addition, the specific problem

Which is irrelevant since within Haiku it's getting that from the compiler optional package which has nothing to do with which one you used to build the system.

As the problem is easy to reproduce even with your small test program, it should be easy to fix....

Except we can't reproduce it, otherwise these questions wouldn't be getting asked.

[schily]

Do you like to tell me that you cannot reproduce the problem using:

haiku-pre-alpha-hrev31644-raw.zip

For me it is reproducable with two different installations.

[bga]

Could it be that the build tools in the build factory were not updated after the change to wchar? schilly, any chance you could try compiling Haiku yourself to test this? I can't reproduce this bug either on my instalarion (hrev31711).

[schily]

I did compile and run testmb.c on an installation that is based on the HD image from:

haiku-pre-alpha-hrev31644-raw.zip

If you do the same, you must get the same error as I get.

If it does not fail for you, please tell me what specific things you do in order to prevent the failure.

[anevilyak]

Replying to schily:

I did compile and run testmb.c on an installation that is based on the HD image from:

haiku-pre-alpha-hrev31644-raw.zip

That would be the missing piece of the puzzle, and indicates that the build factory was never updated with the newer toolchain. Most of us are using Haiku installations that were built directly on our own systems with all the needed updates. Have sent an email to the maintainer of that build factory. Until he gets around to updating that, the only way you'll work around this is by building Haiku yourself.

In any case, on a correctly built system:

~> uname -a
Haiku leviathan 1 r31713 Jul 23 2009 07:36:06 BePC Haiku
~> gcc --version
gcc (GCC) 4.3.3
Copyright (C) 2008 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

~> ./testmb 
wide value: 0x0078
Test succeeded.

[umccullough]

Has anyone notified Phil yet? AFAIK, he doesn't follow the bug list.

[bonefish]

(In #4116) I'd guess that this is a dup of #4113 and will disappear as soon as the build factory uses the correct build tools.

[anevilyak]

Replying to umccullough:

Has anyone notified Phil yet? AFAIK, he doesn't follow the bug list.

Yes, I notified Phil, I was under the impression he was working on updating them, but he never gave me confirmation that he was finished with that yet. Will see what's going on (or you can if you see him before me).

[anevilyak]

Please try again with ​http://haiku-files.org/raw/haiku-pre-alpha-gcc4-r31728-raw.zip .

[anevilyak]

Confirmed fixed.

[schily]

OK, it now works with the gcc4 version

[bbjimmy]

fixes the gensyscalls issue using gcc2.