new gcc indirect linking policy changes cause build failures / missing symbols.

[kallisti5]

Haiku gcc4 build fails on Ubuntu 11.10.

gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) 

hrev42873

attached is full build output.

generated/objects/linux/x86/release/tools/unzip/libunzip.a(beos.o): In function `setBeOSexfield':
beos.c:(.text+0x43e): undefined reference to `_kern_open'
beos.c:(.text+0x494): undefined reference to `__swap_int32'
beos.c:(.text+0x4a5): undefined reference to `__swap_int64'
beos.c:(.text+0x599): undefined reference to `fs_write_attr'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_open_dir_entry_ref'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_haiku_build_strerror'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_unlink'
generated/objects/linux/lib/libbe_build.so: undefined reference to `strlcpy'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_read_stat'
generated/objects/linux/lib/libbe_build.so: undefined reference to `atomic_get'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_open_attr_dir'
generated/objects/linux/lib/libbe_build.so: undefined reference to `__gUmask'
generated/objects/linux/lib/libbe_build.so: undefined reference to `create_sem'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessage::~KMessage()'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_haiku_build_errno'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessage::GetNextField(BPrivate::KMessageField*) const'
generated/objects/linux/lib/libbe_build.so: undefined reference to `fs_stat_attr'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_remove_attr'
generated/objects/linux/lib/libbe_build.so: undefined reference to `find_thread'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_create_dir'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessageField::TypeCode() const'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_read'
generated/objects/linux/lib/libbe_build.so: undefined reference to `delete_sem'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_write'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessage::ReplyToken() const'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessage::ReplyPort() const'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_unlock_node'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_entry_ref_to_path'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessage::KMessage()'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessage::TargetToken() const'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_rename_attr'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessageField::ElementAt(int, int*) const'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_rewind_dir'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_read_link'
generated/objects/linux/lib/libbe_build.so: undefined reference to `__swap_int16'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessageField::Name() const'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_open_dir'
generated/objects/linux/lib/libbe_build.so: undefined reference to `atomic_add'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_open_parent_dir'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_read_dir'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessageField::KMessageField()'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_rename'
generated/objects/linux/lib/libbe_build.so: undefined reference to `release_sem'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessage::SetTo(void const*, int, unsigned int)'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessage::What() const'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_close'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessage::kMessageHeaderMagic'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_lock_node'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_seek'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_open_entry_ref'
generated/objects/linux/lib/libbe_build.so: undefined reference to `debugger'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessageField::HasFixedElementSize() const'
generated/objects/linux/lib/libbe_build.so: undefined reference to `fs_read_attr'
generated/objects/linux/lib/libbe_build.so: undefined reference to `acquire_sem_etc'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_fsync'
generated/objects/linux/lib/libbe_build.so: undefined reference to `BPrivate::KMessageField::CountElements() const'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_create_symlink'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_write_stat'
generated/objects/linux/lib/libbe_build.so: undefined reference to `_kern_dup'

[kallisti5]

full build output

[bonefish]

On a quick read through the gcc 4.6 release changes I didn't see anything related. What's the binutils version? Please also list the symbols of libroot_build (readelf --syms ...).

[kallisti5]

kallisti5@houvonglucka23:~$ ar -V
GNU ar (GNU Binutils for Ubuntu) 2.21.53.20110810
Copyright 2011 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

readelf output attached

[bonefish]

The link line lists libroot_build.so and the library defines and exports the symbols in question, so I don't see why the linker would complain. I didn't spot anything related in the binutils changelog either.

You could experiment a bit with the link line -- e.g. with the library order -- and see if that makes a difference. Also run with -v so we see the tools invoked.

[mt]

Hi, I built gcc4/gcc2 hybrid on Ubuntu 11.10 X86-32. I encountered same unzip problem, and adding $(HOST_LIBROOT) in unzip/Jamfile solved problem (other part except unzip can build well).

diff --git a/src/tools/unzip/Jamfile b/src/tools/unzip/Jamfile
index 6ca3b0a..79b9db4 100644
--- a/src/tools/unzip/Jamfile
+++ b/src/tools/unzip/Jamfile
@@ -31,6 +31,6 @@ BuildPlatformMain unzip :
unzip.c
unreduce.c
unshrink.c
- : libunzip.a $(HOST_LIBBE) $(HOST_LIBSUPC++)
+ : libunzip.a $(HOST_LIBBE) $(HOST_LIBROOT) $(HOST_LIBSUPC++)
;

Another problem: gcc2 build tools could not build with gcc 4.6.1 in Ubuntu 11.10, so I need to install gcc-4.5 package and confirured with CC=gcc-4.5 .

[bonefish]

Replying to mt:

Hi, I built gcc4/gcc2 hybrid on Ubuntu 11.10 X86-32. I encountered same unzip problem, and adding $(HOST_LIBROOT) in unzip/Jamfile solved problem (other part except unzip can build well).

Can you show me the resulting link line (near the end of the jam -na '<build>unzip' output)? As we see in kallisti5's output, libroot_build.so is already linked against. Adding $(HOST_LIBROOT) would just add it a second time. So I wonder, if the link order is somehow relevant, although it shouldn't be for shared libraries. This smells a bit like a linker (ld or gold, whichever is used) bug.

Another problem: gcc2 build tools could not build with gcc 4.6.1 in Ubuntu 11.10, so I need to install gcc-4.5 package and confirured with CC=gcc-4.5 .

The gcc 2 sources are rather old and we've repeatedly seen issues with newer compiler versions. They usually could be fixed without too much work. Patches welcome, of course. :-)

[kallisti5]

confirmed what bonefish said. reversing the two shared libraries in the linking fixed the build...

[kallisti5]

#gcc:

09:40 < Zordan> if you are using e.g. newer gold linker
     (and perhaps newer regular ld as well?) then indirect shared
     library dependencies are no longer followed (e.g. shared
     libraries have to list their own direct dependencies)

[kallisti5]

#gcc:

09:44 < Zordan> kallisti5: maybe similar in change to what fedora made
      the default as well:
      https://fedoraproject.org/wiki/UnderstandingDSOLinkChange

09:48 < Zordan> apparently also in ubuntu since natty:
      https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition

[bonefish]

Replying to kallisti5:

#gcc:

09:40 < Zordan> if you are using e.g. newer gold linker
     (and perhaps newer regular ld as well?) then indirect shared
     library dependencies are no longer followed (e.g. shared
     libraries have to list their own direct dependencies)

The first part of this statement seems to refer to what is explained in the "UnderstandingDSOLinkChange" wiki page. It basically says that, if your app uses a library function, it has to be linked directly to that library and cannot rely on another library pulling in the first library. This makes sense. (BTW, it is something that was mandatory on BeOS as well, as the runtime loader did support resolving symbols only in direct dependencies.)

The part in the last parentheses is a bit ambiguous. It might either mean when a library is linked, it has to be linked against all its dependencies or that when linking against a library all its dependencies have to be listed as well. The latter I would find rather ridiculous.

Anyway, we do list libroot_build.so, both when linking libbe_build.so as well as when linking the app, so either way any condition the statement might refer to is fulfilled.

What apparently seems to be the problem is that the linker (the verbose output doesn't show which one is actually invoked by collect2 -- the error message says "ld returned 1 exit status", but that might not refer to the actual name of the linker) even requires the libraries to be listed in dependency order. That I would consider a bug or at least a seriously missing feature.

[kallisti5]

swapping the shared library orders in collect2 also resolves the issue.

Verbose failure, original sorting (collect2-verbose.txt) 
Verbose success, modified shared library sorting (collect2-verbose-success.txt)

[mmlr]

Replying to bonefish:

What apparently seems to be the problem is that the linker (...) even requires the libraries to be listed in dependency order. That I would consider a bug or at least a seriously missing feature.

It's by design. The second link has it in there:

Also in Natty, ld runs with the --as-needed option enabled by default.
...
The --as-needed option also makes the linker sensitive to the
ordering of libraries on the command-line. You may need to
move some libraries later in the command-line, so they come
after other libraries or files that require symbols from them.

Basically --as-needed links in only actually required libraries. Since that lookup obviously only goes in one direction it's pretty much the same as for static libraries now.

[kallisti5]

mmlr: confirmed. removing --as-needed from collect made it link properly. Adding '--no-as-needed' also made collect link properly.

So maybe setting a linker flag of '--no-as-needed' in the build... or are we just covering a bug in our build process?

[jljusten]

Replying to kallisti5:

mmlr: confirmed. removing --as-needed from collect made it link properly. Adding '--no-as-needed' also made collect link properly.

Did this require changes to the jam build files? If so, for the jam newbies out there (ie, me), could you post a diff? Thanks.

[leavengood]

If mt's change from #comment:5 fixes the build, why don't we just do that? Assuming of course it doesn't cause issues on Haiku or other build platforms. I'll probably be setting up Ubuntu soon on a new computer and would like to be able to build Haiku on it.

[kallisti5]

leavengood: yeah, we can do that... however it's a hack. $(HOST_LIBROOT) is already linked.. but in the wrong order...

then again.. having --no-as-needed --as-needed --no-as-needed --as-needed --no-as-needed in the final compile statement is kinda a hack as well :)

I'll add it.

[bonefish]

Replying to kallisti5:

leavengood: yeah, we can do that... however it's a hack. $(HOST_LIBROOT) is already linked.. but in the wrong order...

then again.. having --no-as-needed --as-needed --no-as-needed --as-needed --no-as-needed in the final compile statement is kinda a hack as well :)

Not really. It would just counter a flag that really shouldn't be there in the first place. I agree with the general intend of avoiding building in library dependencies that aren't necessary, but --as-needed implements it in a way that makes using the linker unnecessarily inconvenient. I find it rather stupid to list libraries multiple times just to appease such a linker.

[kallisti5]

bonefish: libroot was *just* added in hrev43186 to unzip build, confirmed working on gcc4 Ubuntu 11.10

alternate solutions? I couldn't find any flags that would reliably work. really we need to remove the new DSO flags... but thats hard to do as they are compiled in.

Maybe a gcc spec file?

[bonefish]

Replying to kallisti5:

alternate solutions? I couldn't find any flags that would reliably work.

Huh? In comment:13 you write that adding --no-as-needed works. I don't see why adding -Xlinker --no-as-needed to HOST_LINKFLAGS wouldn't work. If it doesn't, please post the error and command line.

really we need to remove the new DSO flags... but thats hard to do as they are compiled in.

There's a very simple solution: Build ld respectively gcc without this flag enabled. I haven't found in either release notes that the flag has been enabled by default, so I suspect that's just a "brilliant" move of the Ubuntu makers. Complaining to them might help.

[kallisti5]

bonefish: yeah. I think I was mistaken on the --no-as-needed change due to the unzip lib already being built.

Let me do some testing minus hrev43186 real quick and post here.

[kallisti5]

verified that I was incorrect before...

pre hrev43186:

• stock: build failure
• add --no-as-needed: build failure
• add --as-needed: build failure
• add --no-add-needed: build failure
• add --add-needed: build failure

the stock collect command where the issue is:

/usr/lib/gcc/x86_64-linux-gnu/4.6.1/collect2 --build-id --no-add-needed --as-needed --eh-frame-hdr -m elf_x86_64 --hash-style=gnu -dynamic-linker /lib64/ld-linux-x86-64.so.2 -z relro -o ../../../generated/objects/linux/x86/release/tools/unzip/unzip /usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.6.1/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/4.6.1 -L/usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../.. ../../../generated/objects/linux/x86/release/tools/unzip/unzip.o ../../../generated/objects/linux/x86/release/tools/unzip/unreduce.o ../../../generated/objects/linux/x86/release/tools/unzip/unshrink.o ../../../generated/objects/linux/lib/libroot_build.so ../../../generated/objects/linux/x86/release/tools/unzip/libunzip.a ../../../generated/objects/linux/lib/libbe_build.so -lsupc++ -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.6.1/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../x86_64-linux-gnu/crtn.o

notice the multiple -needed flags. Ubuntu and Fedora seem to do this now with Debian considering it.

• Removing the first --as-needed causes the collect to succeed... adding more flags doesn't seem to disable it.

[kallisti5]

I see people dealing with this by writing out a modified gcc spec file... seems like a pain to automate though:

​http://permalink.gmane.org/gmane.comp.clustering.open-mpi.user/14707

[bonefish]

Can you please show me the link line produced by our build system (I mean the cc ...; the resulting collect2 invocation might be interesting, too) when is -Xlinker --no-as-needed is added to HOST_LINKFLAGS.

[leavengood]

Could we just keep the somewhat "hacky" fix from hrev43186, even if it isn't totally correct, since this only seems to affect one component? I'm not sure the time investment is worth it now due to one build platform's bad decision.

[leavengood]

Well I guess I spoke up too late, given hrev43191. Still it might be good to look further into why this was set up on Ubunutu versus just turning off this option globally in our build system. Though it does seem like a stupidly implemented GCC option.

[kallisti5]

sigh.

seems you were right bonefish, gcc is smart enough and pick up the removal through Xlinker and remove the excess arguments when calling collect.

fixed *properly* in hrev43191. I added the HOST_LINKFLAGS addition below HOST_GCC_VERSION in the BuildSetup file in case we need to check the host gcc version at some point. (i have no idea when --no-as-needed was introduced)

resolved.