Higher job count parallel builds (jam -j 4 for example) don't work

[meianoite]

When doing a "jam -j4" to build the image from scratch I receive a lot of missing targets, and end up having to repeat the build command 2+ times more to have it finally complete. And I'm not really sure the final image is sane, as there are errors regarding the copy of some files, and there's no sure-fire way of knowing it anyway, but I'll try comparing md5sums of the generated objects from one build and the other and see if I can filter out changes that are only related to hardcoding timestamps.

Building with "jam -j2" works, though, as plenty of devs are known to use this.

I guess the order of building and the dependencies between targets isn't fully fleshed out on our jamfiles.

I'm under the strong impression that this is known already, but I've searched Trac all over and haven't found a single ticket related to this; anyway, it's not anything critical, so I'm filing this one as "enhancement". Still, doing such parallel build under Haiku itself is fantastic to test the IO subsystem, the scheduler, and to demo how Haiku will be able to decode 8+ streams of video, build itself, and not drop a single frame in the process ;)

Build logs attached; same build, each an additional run of "jam -q -j4" (5 runs in total)

[umccullough]

Has been issues suspected with even -j2 as noted in the following tickets: #1116, #1976, #2572

[umccullough]

Replying to umccullough:

Has been issues suspected with even -j2 as noted in the following tickets: #1116, #1976, #2572

Whoops, that last one should have been #2776

[meianoite]

Kind of hard to spot those when they contain no mention to "jam" on the summaries :)

Many thanks for your diligent search!

[bonefish]

Please don't attach zipped text files. That's just annoying to browse.

[meianoite]

As you wish! I had thought that the 488k limit was global, not per-file, sorry. Plain text files attached now.

Now, I can't remove the gzipped files; if someone else can, please, do it on my behalf.

[mmadia]

deleted most .gz attachments.

[meianoite]

Holy Cow. For some reason Trac decided to re-pack the original gzipped text files, and I blindly downloaded and reattached them uncompressed without checking. Turns out that Trac gave me gzipped Trac-annotated pages containing an embedded gzipped file totally corrupted, and even the raw-attachment's are gzipped *twice*. That totally tripped me, and since that was over a month ago and only looking at the small (non-)decompressed size I was lead to believe it was unnecessary. But it was not.

The thing is, those logs are indeed bigger than 488k, build_jam_j4_1.txt being 1.382KB big and build_jam_j4_2.txt 1.073KB big. That's why I compressed them all.

So, I'm very sorry for the annoyance, but it's hardly my fault that Trac has a size limit, and it's *totally* not my fault that it acted up and went nuts as to how those attachments were stored internally.

Recapitulating: you have to download the raw-attachment's, gunzip them, *and* gunzip the resulting files to get the plaintext ones.

[meianoite]

First run (proper)

[meianoite]

4th run, has to be terminated with ctrl-c as bfs_shell gets stuck (proper)

[meianoite]

5th and final run, which I suspect isn't 100% complete either… (proper)

[mmadia]

Replying to meianoite:

Holy Cow. For some reason Trac decided to re-pack the original gzipped text files, and I blindly downloaded and reattached them uncompressed without checking.

No, i believe this is an error in Haiku's gunzip : ​http://dev.haiku-os.org/ticket/4054 (this is why i left a single .txt.gz file attached on this ticket)

[bonefish]

The problem in the 0th log is fixed in hrev31415. The problem in the 4th log is a known problem.

[meianoite]

Second run (proper, gzipped, over 1MB uncompressed )

[meianoite]

Third run, image gets built, build system complains about stuff (proper, gzipped, over 1MB uncompressed)

[bonefish]

Log 3 is the same problem as 4. The logs 1 and 2 are garbage now.

[meianoite]

Replying to mmadia:

No, i believe this is an error in Haiku's gunzip : ​http://dev.haiku-os.org/ticket/4054 (this is why i left a single .txt.gz file attached on this ticket)

I don't think so. I'm not running Haiku right now, but Windows, and the gzipped files were unpacked by 7-zip. This makes me quite sure it's is a Trac issue.

[meianoite]

Replying to bonefish:

Log 3 is the same problem as 4. The logs 1 and 2 are garbage now.

I've reattached them back in their original gzipped form, please download them once more.

[mmadia]

Replying to meianoite:

I don't think so. I'm not running Haiku right now, but Windows, and the gzipped files were unpacked by 7-zip. This makes me quite sure it's is a Trac issue.

Ok, you're right. In Haiku, I was using Trac to download the file. In FreeBSD, i used wget http://dev.haiku-os.org/raw-attachment/ticket/4005/build_jam_j4_4.txt.gz, which worked as expected. Trying that command in Haiku confirmed the issue being in Trac.

[meianoite]

mmadia, while you are at it, could you please remove the attachments not marked as (proper)? Thanks :)

[bonefish]

The problem in log 1 is fixed in hrev31416 (unless there are more dependencies between timezone files).

[bonefish]

Log 2 shows the same problem as 3 and 4, which is the same as in #1976.

[bonefish]

The script corruption issues are fixed in hrev31419. In case you find a new problem, please open a new ticket.