Enable wget timestamping for file download jamrules

[meianoite]

Trivial patch to enable timestamping so the optional packages aren't re-downloaded unnecessarily depending on the jam invocation.

[meianoite]

Add -N option

[meianoite]

Okay, that doesn't work.

According to the wget manual,

"Use of -O is not intended to mean simply "use the name file instead of the one in the URL;" rather, it is analogous to shell redirect- ion: wget -O file ​http://foo is intended to work like wget -O - ​http://foo > file; file will be truncated immediately, and all downloaded content will be written there.

For this reason, -N (for timestamp-checking) is not supported in combination with -O: since file is always newly created, it will always have a very new timestamp. A warning will be issued if this combination is used."

I'm not at all familiar with the intricacies of Jam, but iteems like parenthesis aren't being used for subshell invocation like in sh, but for variable substitution. I hope backticks still work in this situation. Would it be valid if we did something like:

actions DownloadFile1 {

pushd pwd cd dirname $(1) wget - $(URL) popd

}

?

[meianoite]

Argh. WikiFormatting.

actions DownloadFile1 {

pushd `pwd`

cd `dirname $(1)`

wget -N $(URL)

popd

}

[bonefish]

I don't get the point of this ticket. Please elaborate.

Regarding formatting, just enclose code blocks in tripple braces ({{{ ... }}}).

[meianoite]

Replying to bonefish:

I don't get the point of this ticket. Please elaborate.

First and foremost, it's an enhancement ticket, not a bug report. Maybe what I find to be "enhanced" behaviour is actually considered undesirable by you or other developers, but my rationale is as follows:

If you invoje jam with the -a option, it will re-download packages unconditionally. Since I had to do it quite a number of times today because of what became #4070 (to check that I wasn't messing up somewhere else), I thought it would be better for both the servers' bandwidth, and my monthly download limit, if those files weren't redownloaded in case they were identical.

Ideally the build system would invoke a port of the to-become Haiku packaging system to check for new versions of packages along with a more robust check of md5/sha1 signatures, and download remote packages only when they differ from the local copy, but I figured that for the time being the timestamping option of wget would suffice.

[bonefish]

Replying to meianoite:

Replying to bonefish:

I don't get the point of this ticket. Please elaborate.

First and foremost, it's an enhancement ticket, not a bug report. Maybe what I find to be "enhanced" behaviour is actually considered undesirable by you or other developers, but my rationale is as follows:

If you invoje jam with the -a option, it will re-download packages unconditionally. Since I had to do it quite a number of times today because of what became #4070 (to check that I wasn't messing up somewhere else), I thought it would be better for both the servers' bandwidth, and my monthly download limit, if those files weren't redownloaded in case they were identical.

Mmh, I don't think I've ever built an image with "-a". I usually just remove the "objects" directory and built normally, which also has the advantage of removing all the files that the build system doesn't know about anymore.

Ideally the build system would invoke a port of the to-become Haiku packaging system to check for new versions of packages along with a more robust check of md5/sha1 signatures, and download remote packages only when they differ from the local copy, but I figured that for the time being the timestamping option of wget would suffice.

Actually the files at the URLs are not intended to change. If a new version of a package is built, it gets a new (unique) name and OptionalPackages is adjusted accordingly. The point is that the build system is supposed to define the the contents of the image, not "third parties" swapping optional package files behind the scenes. So IMO the timestamp change is superfluous. Either we decide not to let the build system update downloads ever -- then the files could simply be marked NoUpdate -- or we are fine with "-a" updating them (which is consistent behavior) and tell people not to build jam with "-a", if they don't want that to happen.

Regarding your patch. pushd/popd are superfluous, since actions are run in separate shells anyway.

[umccullough]

FWIW, you could also modify your UserBuildConfig such that optional packages are only included when using a build profile

jam -aq

followed by your:

jam -q @disk

where @disk is defined with the optional packages in a conditional block based on the profilename.

Furthermore, you could define a build profile that intentionally excludes them.

I don't know, it seems like it's not that big of a deal, and easily worked around if desired.

[meianoite]

Replying to bonefish:

Mmh, I don't think I've ever built an image with "-a". I usually just remove the "objects" directory and built normally, which also has the advantage of removing all the files that the build system doesn't know about anymore.

Hmm... Good point. What about the build/*cache files? Does it matter if I leave them there or erase them? I figure jam -a recreates those.

Actually the files at the URLs are not intended to change. If a new version of a package is built, it gets a new (unique) name and OptionalPackages is adjusted accordingly.

Well, yeah, but has the side-effect of leaving old packages behind on the download directory.

The point is that the build system is supposed to define the the contents of the image, not "third parties" swapping optional package files behind the scenes. So IMO the timestamp change is superfluous. Either we decide not to let the build system update downloads ever -- then the files could simply be marked NoUpdate -- or we are fine with "-a" updating them (which is consistent behavior) and tell people not to build jam with "-a", if they don't want that to happen.

I think I like the NoUpdate option better. We're then consistent with rebuilding everything the build system *generates*, while leaving what it *imports* alone.

Regarding your patch. pushd/popd are superfluous, since actions are run in separate shells anyway.

I wouldn't count that as a patch, actually :) Just pseudocode. I have no idea if invoking a subshell via backticks would work. Anyway, it's still necessary to cd into the downloads directory when invoking wget with -N, but I figure it's more robust to cd into $(HAIKU_DOWNLOAD_DIR) anyway.

Replying to umccullough:

FWIW, you could also modify your UserBuildConfig such that optional packages are only included when using a build profile where @disk is defined with the optional packages in a conditional block based on the profilename.

Furthermore, you could define a build profile that intentionally excludes them.

I don't know, it seems like it's not that big of a deal, and easily worked around if desired.

Hey, this is just brilliant! Thanks! :D

Closing as invalid since I too find this to be the best solution to my problem.

[bonefish]

Replying to meianoite:

Replying to bonefish:

Mmh, I don't think I've ever built an image with "-a". I usually just remove the "objects" directory and built normally, which also has the advantage of removing all the files that the build system doesn't know about anymore.

Hmm... Good point. What about the build/*cache files? Does it matter if I leave them there or erase them? I figure jam -a recreates those.

They contain information obtained from header scanning and from reading the jamfiles. Unless you've messed with timestamps (or your system time), there's no need to remove them.