Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#6599 closed bug (fixed)

fs_shell parse error during BuildHaikuImage

Reported by: Disreali Owned by: zooey
Priority: normal Milestone: R1
Component: System/libroot.so Version: R1/Development
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description (last modified by anevilyak)

Since hrev38571, building haiku on a haiku-gcc2hybrid host fails during the BuildHaikuImage section with the following error:

Error: Failed to open source path `/HDev/haiku/haiku/generated.x86gcc2/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark as': Bad file descriptor

full error text attached

Attachments (3)

BuildHaikuImage_error.txt (1.6 KB ) - added by Disreali 9 years ago.
kdl_bfs_shell-backtrace.txt (10.8 KB ) - added by Disreali 9 years ago.
bfs_shell_kdebugger.txt (3.2 KB ) - added by aldeck 9 years ago.

Download all attachments as: .zip

Change History (24)

by Disreali, 9 years ago

Attachment: BuildHaikuImage_error.txt added

comment:1 by anevilyak, 9 years ago

Description: modified (diff)

comment:2 by Disreali, 9 years ago

I just discovered that ProcessController list three instances of bfs_shell, one for each jam attempt that I've done on this boot.

Attaching bt from kdl/syslog.

by Disreali, 9 years ago

Attachment: kdl_bfs_shell-backtrace.txt added

comment:3 by phoudoin, 9 years ago

As 'Mark as<B_UTF8_ELLIPSIS>' is quite an unique case in all files copied by fs_shell tool, maybe it's due to fs_shell's ArgsVector::Parse() method don't handle UTF8 chars as one would expect.

in reply to:  3 comment:4 by bonefish, 9 years ago

Replying to phoudoin:

As 'Mark as<B_UTF8_ELLIPSIS>' is quite an unique case in all files copied by fs_shell tool, maybe it's due to fs_shell's ArgsVector::Parse() method don't handle UTF8 chars as one would expect.

There's nothing one can do wrong by not handling UTF-8 characters specially (which the code surely doesn't (that is unless someone messed with it since it wrote it)).

I haven't tested the build under Haiku lately, but unless I'm much mistaken the add-on has been part of the image for ages and I've never seen such an issue. Can anyone else reproduce it? Disreali, do you have any non-default locale settings (e.g. a different char encoding)?

comment:5 by anevilyak, 9 years ago

Tried it with the latest revision as of last night and I'm unable to reproduce that issue, a full @alpha-raw build went through without a problem.

comment:6 by aldeck, 9 years ago

I stumbled on this bug yesterday, trying to build hrev38656 on haiku host hrev38635, fresh svn checkout.

The error msg is slightly different here though:

Error: Failed to open source path `generated/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark as': Bad file descriptor

notice the missing ellipsis

Last edited 9 years ago by aldeck (previous) (diff)

in reply to:  6 comment:7 by bonefish, 9 years ago

Replying to aldeck:

I stumbled on this bug yesterday, trying to build hrev38656 on haiku host hrev38635, fresh svn checkout.

The error msg is slightly different here though:

Error: Failed to open source path `generated/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark as': Bad file descriptor

notice the missing ellipsis

Yep, it's the same in the ticket description.

Are you able to reliably reproduce the problem, or has this been a one-time encounter?

comment:8 by aldeck, 9 years ago

This is a permanent failure, unless i manually rename the object "Mark as..." to "Mark as" so that bfs_shell finds what it is looking for.

I could possibly bisect the problematic revision but i'd need a smaller test for that (suggestions welcome). Though i can't guess if it's an issue on the host or the source.

comment:9 by aldeck, 9 years ago

I don't know if that helps, but i Just noticed that it leaves a stray bfs_shell process.

Attached is the output of 'thread' and 'bt' on it in the kernel debugger.

EDIT: sorry it's late, i didn't see the top of the ticket i.e nothing new :)

EDIT2: except maybe the top line KERN: bfs: bfs_open_dir:1594: Not a directory

Last edited 9 years ago by aldeck (previous) (diff)

by aldeck, 9 years ago

Attachment: bfs_shell_kdebugger.txt added

in reply to:  8 comment:10 by bonefish, 9 years ago

Replying to aldeck:

This is a permanent failure, unless i manually rename the object "Mark as..." to "Mark as" so that bfs_shell finds what it is looking for.

I could possibly bisect the problematic revision but i'd need a smaller test for that (suggestions welcome). Though i can't guess if it's an issue on the host or the source.

Does jam -q @image update "Mark as..." reproduce the problem?

Replying to aldeck:

I don't know if that helps, but i Just noticed that it leaves a stray bfs_shell process.

That's a known "feature". Since we enabled set -o errexit in build_haiku_image an error causes an immediate exit of the script, so it never tells the bfs_shell process to terminate. I believe I described the situation in some ticket already. There's also #5498 which I believe has this cause.

comment:11 by aldeck, 9 years ago

Yes that reproduces the problem.

Output (with recompilation of the addon):

jam -q @image update "Mark as…"
[...]
Link generated/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark as… 
XRes1 generated/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark as… 
SetType1 generated/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark as… 
MimeSet1 generated/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark as… 
SetVersion1 generated/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark as… 
Chmod1 generated/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark as… 
AppendToContainerCopyFilesScript <HaikuImage>haiku.image-copy-files-dummy-system/add-ons/Tracker 
InitScript1 generated/haiku.image-extract-files 
BuildHaikuImage1 generated/haiku.image 

Populating image ...
Error: Failed to open source path `generated/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark as': Bad file descriptor
Error: Command failed: Bad file descriptor
Error: Command was:
  cp -f :generated/objects/haiku/x86/release/add-ons/tracker/mark_as/Mark\ as… /myfs/system/add-ons/Tracker

comment:12 by bonefish, 9 years ago

I just tried on an hrev38606 gcc4 Haiku and things work fine.

Instead of binary searching the revision to blame I'd recommend to simply track the problem down. The last line of output stems from the fs_shell_command and apparently at this point the file name is still correct. So it either gets lost in the communication to the bfs_shell or in the bfs_shell itself. The control flow (all within src/tools/fs_shell/) is the following:

  1. fs_shell_command (fs_shell_command_beos.cpp) sends the whole command to the bfs_shell.
  2. It arrives in FSShell::get_external_command() (external_commands_beos.cpp).
  3. Which is called by input_loop() (fssh.cpp). The command line is parsed and the respective command function is invoked.
  4. In this case that is command_cp() in (command_cp.cpp), which does the actual copying.

A bit of debug output along the path should allow to track the point of failure down quickly.

comment:13 by aldeck, 9 years ago

Thanks for the details. It fails in ArgVec::Parse, isspace() misbehaves on the ellipsis. Locale issue? I'm in english/english

as…
char 'a'
char 's'
isspace (char = 0xffffffe2)
isspace (char = 0xffffff80)
isspace (char = 0xffffffa6)

which looks like the raw utf-8 inverted for the ellipsis: 0xE2 0x80 0xA6

Last edited 9 years ago by aldeck (previous) (diff)

comment:14 by bonefish, 9 years ago

Component: - GeneralSystem/libroot.so
Owner: changed from nobody to zooey
Status: newassigned

Yep, the character type function implementations have been changed not too long ago. Since it seems to work on gcc 4 Haiku (at least for me), it might a gcc 2 only issue.

in reply to:  13 ; comment:15 by bonefish, 9 years ago

Replying to aldeck:

char 'a'
char 's'
isspace (char = 0xffffffe2)
isspace (char = 0xffffff80)
isspace (char = 0xffffffa6)

which looks like the raw utf-8 inverted for the ellipsis: 0xE2 0x80 0xA6

I'd say the char is sign-extended to int.

in reply to:  15 comment:16 by zooey, 9 years ago

Replying to bonefish:

Replying to aldeck:

char 'a'
char 's'
isspace (char = 0xffffffe2)
isspace (char = 0xffffff80)
isspace (char = 0xffffffa6)

which looks like the raw utf-8 inverted for the ellipsis: 0xE2 0x80 0xA6

I'd say the char is sign-extended to int.

Yup, and I think the culprit is the isctype macro in ctype.h, which casts the given value to 'int' - it should cast the value to unsigned char instead, as otherwise any negative signed value will be extended inappropriately. Since the value is then used for an out-of-bounds access to ctype_b[], the outcome is undpredictable. The latter could explain different results for gcc4 and gcc2, additionally it could be that gcc2 handles char types differently (maybe makes them signed by default?).

If no-one beats me to it, I can fix this tomorrow.

comment:17 by aldeck, 9 years ago

I was observing a new bug in Tracker and it turns out that it might be totally related. The thread gets stuck in NaturalCompare() (from tracker/Utilities.h). It uses isspace() and other char functions. It ends in an infinite loop, making some tracker queries hang on me.

I was initialy suspecting Axel's recent changes there but now that i looked closer it definitely looks like the same cause :)

comment:18 by zooey, 9 years ago

Status: assignedin-progress

comment:19 by zooey, 9 years ago

Should be fixed in hrev38708.

The invocation of any ctype-function/macro with signed char is basically incorrect, but having to cast every value to unsigned char before calling any such function isn't very nice - so I left the fssh-code as it is (and all the other thousands of isXXX() invocations spread all over the place ;-).

comment:20 by zooey, 9 years ago

Resolution: fixed
Status: in-progressclosed

comment:21 by aldeck, 9 years ago

I confirm the fix. No more issues building, no tracker deadlock here on hrev38711 gcc2. Thanks!

Note: See TracTickets for help on using tickets.