Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

#4068 closed enhancement (invalid)

Enable wget timestamping for file download jamrules

Reported by: meianoite Owned by: bonefish
Priority: normal Milestone: R1
Component: Build System Version: R1/pre-alpha1
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description

Trivial patch to enable timestamping so the optional packages aren't re-downloaded unnecessarily depending on the jam invocation.

Attachments (1)

timestamping.patch (268 bytes) - added by meianoite 10 years ago.
Add -N option

Download all attachments as: .zip

Change History (9)

Changed 10 years ago by meianoite

Attachment: timestamping.patch added

Add -N option

comment:1 Changed 10 years ago by meianoite

Okay, that doesn't work.

According to the wget manual,

"Use of -O is not intended to mean simply "use the name file instead of the one in the URL;" rather, it is analogous to shell redirect- ion: wget -O file http://foo is intended to work like wget -O - http://foo > file; file will be truncated immediately, and all downloaded content will be written there.

For this reason, -N (for timestamp-checking) is not supported in combination with -O: since file is always newly created, it will always have a very new timestamp. A warning will be issued if this combination is used."

I'm not at all familiar with the intricacies of Jam, but iteems like parenthesis aren't being used for subshell invocation like in sh, but for variable substitution. I hope backticks still work in this situation. Would it be valid if we did something like:

actions DownloadFile1 {

pushd pwd cd dirname $(1) wget - $(URL) popd

}

?

comment:2 Changed 10 years ago by meianoite

Argh. WikiFormatting.

actions DownloadFile1 {

pushd `pwd`

cd `dirname $(1)`

wget -N $(URL)

popd

}

comment:3 Changed 10 years ago by bonefish

I don't get the point of this ticket. Please elaborate.

Regarding formatting, just enclose code blocks in tripple braces ({{{ ... }}}).

comment:4 in reply to:  3 ; Changed 10 years ago by meianoite

Replying to bonefish:

I don't get the point of this ticket. Please elaborate.

First and foremost, it's an enhancement ticket, not a bug report. Maybe what I find to be "enhanced" behaviour is actually considered undesirable by you or other developers, but my rationale is as follows:

If you invoje jam with the -a option, it will re-download packages unconditionally. Since I had to do it quite a number of times today because of what became #4070 (to check that I wasn't messing up somewhere else), I thought it would be better for both the servers' bandwidth, and my monthly download limit, if those files weren't redownloaded in case they were identical.

Ideally the build system would invoke a port of the to-become Haiku packaging system to check for new versions of packages along with a more robust check of md5/sha1 signatures, and download remote packages only when they differ from the local copy, but I figured that for the time being the timestamping option of wget would suffice.

comment:5 in reply to:  4 ; Changed 10 years ago by bonefish

Replying to meianoite:

Replying to bonefish:

I don't get the point of this ticket. Please elaborate.

First and foremost, it's an enhancement ticket, not a bug report. Maybe what I find to be "enhanced" behaviour is actually considered undesirable by you or other developers, but my rationale is as follows:

If you invoje jam with the -a option, it will re-download packages unconditionally. Since I had to do it quite a number of times today because of what became #4070 (to check that I wasn't messing up somewhere else), I thought it would be better for both the servers' bandwidth, and my monthly download limit, if those files weren't redownloaded in case they were identical.

Mmh, I don't think I've ever built an image with "-a". I usually just remove the "objects" directory and built normally, which also has the advantage of removing all the files that the build system doesn't know about anymore.

Ideally the build system would invoke a port of the to-become Haiku packaging system to check for new versions of packages along with a more robust check of md5/sha1 signatures, and download remote packages only when they differ from the local copy, but I figured that for the time being the timestamping option of wget would suffice.

Actually the files at the URLs are not intended to change. If a new version of a package is built, it gets a new (unique) name and OptionalPackages is adjusted accordingly. The point is that the build system is supposed to define the the contents of the image, not "third parties" swapping optional package files behind the scenes. So IMO the timestamp change is superfluous. Either we decide not to let the build system update downloads ever -- then the files could simply be marked NoUpdate -- or we are fine with "-a" updating them (which is consistent behavior) and tell people not to build jam with "-a", if they don't want that to happen.

Regarding your patch. pushd/popd are superfluous, since actions are run in separate shells anyway.

comment:6 Changed 10 years ago by umccullough

FWIW, you could also modify your UserBuildConfig such that optional packages are only included when using a build profile

jam -aq

followed by your:

jam -q @disk

where @disk is defined with the optional packages in a conditional block based on the profilename.

Furthermore, you could define a build profile that intentionally excludes them.

I don't know, it seems like it's not that big of a deal, and easily worked around if desired.

comment:7 in reply to:  5 ; Changed 10 years ago by meianoite

Resolution: invalid
Status: newclosed

Replying to bonefish:

Mmh, I don't think I've ever built an image with "-a". I usually just remove the "objects" directory and built normally, which also has the advantage of removing all the files that the build system doesn't know about anymore.

Hmm... Good point. What about the build/*cache files? Does it matter if I leave them there or erase them? I figure jam -a recreates those.

Actually the files at the URLs are not intended to change. If a new version of a package is built, it gets a new (unique) name and OptionalPackages is adjusted accordingly.

Well, yeah, but has the side-effect of leaving old packages behind on the download directory.

The point is that the build system is supposed to define the the contents of the image, not "third parties" swapping optional package files behind the scenes. So IMO the timestamp change is superfluous. Either we decide not to let the build system update downloads ever -- then the files could simply be marked NoUpdate -- or we are fine with "-a" updating them (which is consistent behavior) and tell people not to build jam with "-a", if they don't want that to happen.

I think I like the NoUpdate option better. We're then consistent with rebuilding everything the build system *generates*, while leaving what it *imports* alone.

Regarding your patch. pushd/popd are superfluous, since actions are run in separate shells anyway.

I wouldn't count that as a patch, actually :) Just pseudocode. I have no idea if invoking a subshell via backticks would work. Anyway, it's still necessary to cd into the downloads directory when invoking wget with -N, but I figure it's more robust to cd into $(HAIKU_DOWNLOAD_DIR) anyway.

Replying to umccullough:

FWIW, you could also modify your UserBuildConfig such that optional packages are only included when using a build profile where @disk is defined with the optional packages in a conditional block based on the profilename.

Furthermore, you could define a build profile that intentionally excludes them.

I don't know, it seems like it's not that big of a deal, and easily worked around if desired.

Hey, this is just brilliant! Thanks! :D

Closing as invalid since I too find this to be the best solution to my problem.

comment:8 in reply to:  7 Changed 10 years ago by bonefish

Replying to meianoite:

Replying to bonefish:

Mmh, I don't think I've ever built an image with "-a". I usually just remove the "objects" directory and built normally, which also has the advantage of removing all the files that the build system doesn't know about anymore.

Hmm... Good point. What about the build/*cache files? Does it matter if I leave them there or erase them? I figure jam -a recreates those.

They contain information obtained from header scanning and from reading the jamfiles. Unless you've messed with timestamps (or your system time), there's no need to remove them.

Note: See TracTickets for help on using tickets.