Opened 10 years ago

Closed 10 years ago

#4005 closed enhancement (fixed)

Higher job count parallel builds (jam -j 4 for example) don't work

Reported by: meianoite Owned by: bonefish
Priority: normal Milestone: Unscheduled
Component: Build System Version: R1/pre-alpha1
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description

When doing a "jam -j4" to build the image from scratch I receive a lot of missing targets, and end up having to repeat the build command 2+ times more to have it finally complete. And I'm not really sure the final image is sane, as there are errors regarding the copy of some files, and there's no sure-fire way of knowing it anyway, but I'll try comparing md5sums of the generated objects from one build and the other and see if I can filter out changes that are only related to hardcoding timestamps.

Building with "jam -j2" works, though, as plenty of devs are known to use this.

I guess the order of building and the dependencies between targets isn't fully fleshed out on our jamfiles.

I'm under the strong impression that this is known already, but I've searched Trac all over and haven't found a single ticket related to this; anyway, it's not anything critical, so I'm filing this one as "enhancement". Still, doing such parallel build under Haiku itself is fantastic to test the IO subsystem, the scheduler, and to demo how Haiku will be able to decode 8+ streams of video, build itself, and not drop a single frame in the process ;)

Build logs attached; same build, each an additional run of "jam -q -j4" (5 runs in total)

Attachments (5)

build_jam_j4_0.txt (230.1 KB ) - added by meianoite 10 years ago.
First run (proper)
build_jam_j4_3.txt (47.1 KB ) - added by meianoite 10 years ago.
4th run, has to be terminated with ctrl-c as bfs_shell gets stuck (proper)
build_jam_j4_4.txt (53.2 KB ) - added by meianoite 10 years ago.
5th and final run, which I suspect isn't 100% complete either… (proper)
build_jam_j4_1.txt.gz (82.4 KB ) - added by meianoite 10 years ago.
Second run (proper, gzipped, over 1MB uncompressed )
build_jam_j4_2.txt.gz (62.7 KB ) - added by meianoite 10 years ago.
Third run, image gets built, build system complains about stuff (proper, gzipped, over 1MB uncompressed)

Download all attachments as: .zip

Change History (22)

comment:1 by umccullough, 10 years ago

Has been issues suspected with even -j2 as noted in the following tickets: #1116, #1976, #2572

in reply to:  1 comment:2 by umccullough, 10 years ago

Replying to umccullough:

Has been issues suspected with even -j2 as noted in the following tickets: #1116, #1976, #2572

Whoops, that last one should have been #2776

comment:3 by meianoite, 10 years ago

Kind of hard to spot those when they contain no mention to "jam" on the summaries :)

Many thanks for your diligent search!

comment:4 by bonefish, 10 years ago

Please don't attach zipped text files. That's just annoying to browse.

comment:5 by meianoite, 10 years ago

As you wish! I had thought that the 488k limit was global, not per-file, sorry. Plain text files attached now.

Now, I can't remove the gzipped files; if someone else can, please, do it on my behalf.

comment:6 by mmadia, 10 years ago

deleted most .gz attachments.

comment:7 by meianoite, 10 years ago

Holy Cow. For some reason Trac decided to re-pack the original gzipped text files, and I blindly downloaded and reattached them uncompressed without checking. Turns out that Trac gave me gzipped Trac-annotated pages containing an embedded gzipped file totally corrupted, and even the raw-attachment's are gzipped *twice*. That totally tripped me, and since that was over a month ago and only looking at the small (non-)decompressed size I was lead to believe it was unnecessary. But it was not.

The thing is, those logs are indeed bigger than 488k, build_jam_j4_1.txt being 1.382KB big and build_jam_j4_2.txt 1.073KB big. That's why I compressed them all.

So, I'm very sorry for the annoyance, but it's hardly my fault that Trac has a size limit, and it's *totally* not my fault that it acted up and went nuts as to how those attachments were stored internally.

Recapitulating: you have to download the raw-attachment's, gunzip them, *and* gunzip the resulting files to get the plaintext ones.

by meianoite, 10 years ago

Attachment: build_jam_j4_0.txt added

First run (proper)

by meianoite, 10 years ago

Attachment: build_jam_j4_3.txt added

4th run, has to be terminated with ctrl-c as bfs_shell gets stuck (proper)

by meianoite, 10 years ago

Attachment: build_jam_j4_4.txt added

5th and final run, which I suspect isn't 100% complete either… (proper)

in reply to:  7 comment:8 by mmadia, 10 years ago

Replying to meianoite:

Holy Cow. For some reason Trac decided to re-pack the original gzipped text files, and I blindly downloaded and reattached them uncompressed without checking.

No, i believe this is an error in Haiku's gunzip : http://dev.haiku-os.org/ticket/4054 (this is why i left a single .txt.gz file attached on this ticket)

comment:9 by bonefish, 10 years ago

The problem in the 0th log is fixed in hrev31415. The problem in the 4th log is a known problem.

by meianoite, 10 years ago

Attachment: build_jam_j4_1.txt.gz added

Second run (proper, gzipped, over 1MB uncompressed )

by meianoite, 10 years ago

Attachment: build_jam_j4_2.txt.gz added

Third run, image gets built, build system complains about stuff (proper, gzipped, over 1MB uncompressed)

comment:10 by bonefish, 10 years ago

Log 3 is the same problem as 4. The logs 1 and 2 are garbage now.

comment:11 by meianoite, 10 years ago

Replying to mmadia:

No, i believe this is an error in Haiku's gunzip : http://dev.haiku-os.org/ticket/4054 (this is why i left a single .txt.gz file attached on this ticket)

I don't think so. I'm not running Haiku right now, but Windows, and the gzipped files were unpacked by 7-zip. This makes me quite sure it's is a Trac issue.

in reply to:  10 comment:12 by meianoite, 10 years ago

Replying to bonefish:

Log 3 is the same problem as 4. The logs 1 and 2 are garbage now.

I've reattached them back in their original gzipped form, please download them once more.

in reply to:  11 comment:13 by mmadia, 10 years ago

Replying to meianoite:

I don't think so. I'm not running Haiku right now, but Windows, and the gzipped files were unpacked by 7-zip. This makes me quite sure it's is a Trac issue.

Ok, you're right. In Haiku, I was using Trac to download the file. In FreeBSD, i used wget http://dev.haiku-os.org/raw-attachment/ticket/4005/build_jam_j4_4.txt.gz, which worked as expected. Trying that command in Haiku confirmed the issue being in Trac.

comment:14 by meianoite, 10 years ago

mmadia, while you are at it, could you please remove the attachments not marked as (proper)? Thanks :)

comment:15 by bonefish, 10 years ago

The problem in log 1 is fixed in hrev31416 (unless there are more dependencies between timezone files).

comment:16 by bonefish, 10 years ago

Log 2 shows the same problem as 3 and 4, which is the same as in #1976.

comment:17 by bonefish, 10 years ago

Resolution: fixed
Status: newclosed

The script corruption issues are fixed in hrev31419. In case you find a new problem, please open a new ticket.

Note: See TracTickets for help on using tickets.