jam -j fails

[Anarchos]

With parallel build, the download of packages is not safe : build_packages/libpng-xxx is empty and prevents the build to succeed.

[bonefish]

As is, the report is not very useful. Generally the use of downloaded build feature packages works just fine. Occasionally a dependency to a build feature is not specified correctly, in which case multi-job (and even single-job) builds may fail.

Please reopen, when you can provide the necessary information: build platform, build configuration, revision, build log.

[anevilyak]

I've run into something similar before, and I believe the behavior being described essentially boils down to: multi-job build is downloading a package with the intent of extracting it. Meanwhile, another parallel build step fails (i.e. a source file with a compilation error due to an incorrect commit).

As such, the package gets downloaded but not extracted, but seemingly, the destination folder for it has been created. The latter's existence appears to be the dependency the build system relies on to know whether to download/extract the package, and consequently that step is skipped entirely without extracting the package at all, leading to a later build failure due to missing headers or whatnot from said build package.

[bonefish]

I believe a regular build error does not result in this state. Jam doesn't interrupt ongoing actions but rather waits for them to finish. So unless the package extraction itself fails, the directory either should not have been created or shouldn't remain empty.

Manually interrupting the build process (Ctrl-C) does, however, interrupt ongoing actions. I.e. if you interrupt the actions between creating the directory and extracting the package, you may indeed get an empty directory.

Unfortunately jam doesn't remove directory targets when the actions creating them fail or are interrupted. So there's little that can be done (unless anyone wants to hack jam).

[jessicah]

Actually, I've just run into this myself. The output is pretty conclusive that the jam rules aren't quite correct.

2014-10-21 23:15:55 (65.7 KB/s) - ‘.../generated/download/libwebp_devel-0.4.1-1-x86_gcc2.hpkg’ saved [731348/731348]

C++ .../generated/objects/haiku/x86_gcc2/release/add-ons/translators/webp/WebPTranslator.o
C++ .../generated/objects/haiku/x86_gcc2/release/add-ons/translators/webp/ConfigView.o
.../src/add-ons/translators/webp/WebPTranslator.cpp:21: webp/encode.h: No such file or directory

Checking the file system, it had indeed downloaded the webp_devel package, but in the build_packages directory, the webp_devel package hadn't even been extracted (and the quite apparent lack of the ExtractPackage1 action in the log).

This is probably exacerbated by my changes to jam in btrev43095.

[jessicah]

I've finally been able to reliably reproduce the issue without resorting to parallel jam. I think we might need an extra unique target for build feature attributes, perhaps?

jam -aq -d4 '<extracted-x86_gcc2:libwebp-devel>develop/headers/webp'

Chmod1 /home/jessica/develop/gcc2h/generated/objects/linux/x86_64/release/tools/package/package 
ExtractHPKGArchive1 /home/jessica/develop/gcc2h/generated/build_packages/libwebp-0.4.1-1-x86_gcc2 
ExtractHPKGArchive1 /home/jessica/develop/gcc2h/generated/build_packages/libwebp-0.4.1-1-x86_gcc2 
...updated 260 target(s)...

This creates the build_packages/libwebp directory and extracts a bunch of build features. Now let's simulate parallel jam by removing the develop headers sub-directory and jamming again.

rm -rf generated/build_packages/libwebp*/develop jam -aq -d4 '<extracted-x86_gcc2:libwebp-devel>develop/headers/webp'

Chmod1 /home/jessica/develop/gcc2h/generated/objects/linux/x86_64/release/tools/package/package 
...updated 258 target(s)...

Notice how with this invocation, that ExtractHPKGArchive1 is never invoked, and that generated/build_packages/libwebp*/develop doesn't exist.

Attached are the before and after logs from jam using -d4 for debugging.

[bonefish]

Manually removing something in a subdirectory of generated/build_package is not simulating a parallel build. For a build feature a single target is used for the extraction of all packages associated with that build feature. All targets (like the headers directory) that are associated with attributes of the build feature depend on that main target. So all users of the build feature can depend on any of the targets and the extraction of all packages is guaranteed. Unless you manually manipulate the extraction directory so that it looks like the main target doesn't have to be made. Or you somehow manage to interrupt jam between extracting the two packages (a regular build failure shouldn't cause it).

That being said, I wonder if your jam fix is correct. I'm not particularly familiar with the jam sources, but the patch(es) suggested on the jamming mailing list were rather different. The first one I saw (cf. ​this mail was proposed by Chuck Homic in June 2010. The thread continues (with a break) through December 2010 (it isn't that long) with a few more analysis and patch suggestions. Also note that boost jam is a fork of a rather old version of the mainline jam.

[waddlesplash]

kallisti5 ran a bunch of tests with -j16 and none of them failed with dependency problems. So it seems this one is fixed.

[nielx]

Assign tickets with status=closed and resolution=fixed within the R1/beta2 development window to the R1/beta2 Milestone