Support debug info in separate file

[bonefish]

GDB supports debug info in a separate file. Debugger should support that eventually, too. Once package management is in place that allows to have separate debug info packages, that only people interested in those could install.

Some more info ​in the GDB docs.

[anevilyak]

Will look into this one. Out of interest, would we want to support both the build-id and debuglink variants described in the gdb docs, or just one or the other?

[bonefish]

I think it is sufficient to support only one of the two options. IIRC one seemed to make more sense, though I don't recall which one. Please research.

[anevilyak]

The debuglink variant looks more sensible to me since, in addition to explicitly identifying the file, it also supplies a CRC for the linked debug file for validation.

[anevilyak]

Would there be any issue with regard to using the checksum code found at the above link directly? I assume it comes from gdb or binutils source, which would imply GPL, but it's published on that page without any explicit license indicator.

[bonefish]

Since there isn't any copyright or license notice the safest thing to do is not to use it at all. Given that it's a common algorithm I'd suspect it should be easy to find code with a liberal license on the internet. zlib also exports a crc32() function which can possibly be used.

[anevilyak]

Replying to bonefish:

Since there isn't any copyright or license notice the safest thing to do is not to use it at all. Given that it's a common algorithm I'd suspect it should be easy to find code with a liberal license on the internet. zlib also exports a crc32() function which can possibly be used.

Didn't realize the CRC algorithm in question was a generic one, thanks for the tip. Looks like zlib's will fit the bill indeed. The only other remaining question is one of search paths: gdb looks in the image's directory, if not found there, subdir debug/, and after that whatever the user has additionally configured as debug information search locations. The first of those seems sensible to me, but is there any preference or particular location we're going to be placing them in package management world?

[bonefish]

Alongside the executable seems obvious at first, but from an aesthetical point of view it is not so nice. It certainly doesn't harm to look there, but I'd go with a separate directory in all case. That is with the following layout for self-contained app directories:

Foo-O-Matic
add-ons/
  bar-extension
debug/
  bar-extension-debug-info
  Foo-O-Matic-debug-info
  libfoo.so-debug-info
lib/
  libfoo.so

That would require searching "<image path>/debug" and "<image path>/../debug" (respectively only the one corresponding to the image type).

Similarly, for executables/libraries/add-ons installed globally in one of the installation locations (system, common, or home) I'd go with a single directory in the installation location. I'm unsure which of the following three I'd prefer, though: "debug", "data/debug", or "develop/debug". "debug" would be at the same level as "bin", "lib", and "add-ons", but it would add another top-level directory. "data/debug" seems logical, since it actually is debug data that would be stored there. "develop/debug" also sounds kind of good, since the directory content is development related.

[anevilyak]

Replying to bonefish:

Attached a patch which implements those search heuristics, going with the develop/debug subdir for the global libs. This appears to work with the exception of one problem: currently, the commands specified for generating the separate debug information file using objcopy aren't entirely working correctly ; all the .debug_* sections get copied correctly, and the file has a .eh_frame section that's the same size as that of the original, but the data in the new file's .eh_frame is bogus (confirmed with objdump itself which states that it's simply full of null sections while the original has valid CIE/FDEs), so we can't actually unwind any call frames, though we can build up all the types and source information. Bug in our version of binutils perhaps?

[anevilyak]

I checked and our build of binutils appears to be current ; I also tried the alternative methodology of copying the full executable then using strip to remove everything but the debug info from the latter, but that unfortunately seems to yield the same result as objcopy --only-keep-debug.

What does work is having the file that .gnu_debuglink points to be the full executable rather than just a file containing the extracted debug sections since that way the frame section doesn't get mangled, but that's obviously less than ideal, though in most cases the additional data of the executable image itself will be quite small compared to the debugging information. Incidentally I also noticed objcopy has the ability to apply zlib compression to the dwarf sections. Is that something that can be done directly doing compilation instead of being a separate step, and if so would it be worth filing an enhancement ticket for us to support that?

[bonefish]

Yeah, might be a binutils bug or a Haiku bug. To narrow it down you could try FreeBSD's or Linux's objcopy. If that works, Haiku's objcopy for cross-compilation on FreeBSD/Linux could be tried.

That aside, doesn't the .eh_frame section remain in the stripped object? Or is it reduced to what is actually needed for exception handling? If remains unchanged, it could still be used.

I skimmed the patch. There are several cast operator instances using the wrong pointer style and the find_directory() thing could be done in a loop. Moreover, handling the self-contained app case last, you might actually find the debug info file for a globally installed library before even trying the "debug" subdirectory of the app directory. At the latest with package management, applications (including self-contained ones) can be installed in the installation locations (e.g. under /boot/common/apps). I would first try the relative paths, then try the "develop/debug" in the respective installation location.

[bonefish]

Compressing the debug info would be nice. I don't know whether ld can do that. But there's a manual for it. :-)

[anevilyak]

Replying to bonefish:

That aside, doesn't the .eh_frame section remain in the stripped object? Or is it reduced to what is actually needed for exception handling? If remains unchanged, it could still be used.

Just checked, it looks like it stays unchanged. Only the one in the copy seems to be bogus. That'd certainly be doable, though going with that approach would mean DwarfFile would have to keep ElfFiles around for both the original file and the debug info instance, which would make management of the sections a little bit messier. That begs another question though: since, in pkg world, the original executable would most likely have been built in release mode, does that .eh_frame section actually contain sufficient information for us? I notice it appears to contain entries that objdump --dwarf=frames can output useful information about but that's not necessarily a guarantee that it has all we need.

I skimmed the patch. There are several cast operator instances using the wrong pointer style and the find_directory() thing could be done in a loop. Moreover, handling the self-contained app case last, you might actually find the debug info file for a globally installed library before even trying the "debug" subdirectory of the app directory. At the latest with package management, applications (including self-contained ones) can be installed in the installation locations (e.g. under /boot/common/apps). I would first try the relative paths, then try the "develop/debug" in the respective installation location.

Will fix those issues, thanks for the review!

[anevilyak]

Replying to bonefish:

Yeah, might be a binutils bug or a Haiku bug. To narrow it down you could try FreeBSD's or Linux's objcopy. If that works, Haiku's objcopy for cross-compilation on FreeBSD/Linux could be tried.

On FreeBSD the result is consistent with Haiku as far as .eh_frame goes, and that one's using binutils 2.15, so this certainly doesn't seem to be new behavior. OTOH, .debug_frame gets copied correctly (FreeBSD still uses gcc 4.1.2 which generates both sections). I guess in that case I'll always try to grab .eh_frame from the primary binary and retrieve everything else from the separate debug info file, since if present we'll prefer .debug_frame anyways.

[anevilyak]

Updated patch attached. Seems to work as best I can tell, with a caveat: if the original executable is a release executable and simply paired with/linked to the debug info file copied out of its debug equivalent, things don't appear to entirely work as planned ; some symbols aren't resolvable as source references (for instance, in my tests with Deskbar, main() and a few other random symbols off BarApp and other classes aren't resolved and simply show disassembly instead). Furthermore, breakpoints don't seem to work as intended in the above scenario either, they claim to be set correctly but wind up never being hit. None of these problems show up if the primary executable was originally built with debug and then stripped of its debug information afterwards.

Also note, another testing caveat - objcopy --add-gnu-debuglink destroys Haiku resources while adding its section to the file so those need to be re-added via xres after adding the link.

[bonefish]

The objcopy bug might be worth a bug report. :-)

Regarding your first caveat, if you used Haiku's build system to build the release and the debug version of the file, problems are to be expected, since one is built with and the other without optimization. In the Linux world usually the release versions are built with both optimization and debug info (and where possible, the debug info is split into another file/package). The result is not optimal for debugging -- due to the optimization inlining and reordering code -- but after all it is a release build.

Regarding the resources, the Haiku build system could automate the step, respectively add the resources after extracting the debug info in the first place.

Regarding your patch:

• Not important, but I would swap the _GetDebugInfoPath() parameters. Output parameters usually come last (though there are exceptions when there are other parameters with default values).
• I don't know why you made the dirLocations array int16. It should be directory_which. Also the -1 element is not needed. The number of elements is sizeof(dirLocations) / sizeof(dirLocations[0]). Moreover the array could be const.

[anevilyak]

Replying to bonefish:

The objcopy bug might be worth a bug report. :-)

Will do.

Regarding your first caveat, if you used Haiku's build system to build the release and the debug version of the file, problems are to be expected, since one is built with and the other without optimization. In the Linux world usually the release versions are built with both optimization and debug info (and where possible, the debug info is split into another file/package). The result is not optimal for debugging -- due to the optimization inlining and reordering code -- but after all it is a release build.

That was it indeed, my mistake.

Regarding your patch:

• Not important, but I would swap the _GetDebugInfoPath() parameters. Output parameters usually come last (though there are exceptions when there are other parameters with default values).
• I don't know why you made the dirLocations array int16. It should be directory_which. Also the -1 element is not needed. The number of elements is sizeof(dirLocations) / sizeof(dirLocations[0]). Moreover the array could be const.

Adjustments made and checked in in hrev44428. Thanks!

[anevilyak]

Replying to bonefish:

The objcopy bug might be worth a bug report. :-)

For reference, ​http://sourceware.org/bugzilla/show_bug.cgi?id=14416 filed.