HaikuDepot is perpetually rebuilt

[waddlesplash]

Building Haiku on Haiku nighly, x86_64.

After the refactor to generate the data load files at build-time, HaikuDepot is always rebuilt. Just run "jam HaikuDepot" twice in a row; it will fully rebuild HaikuDepot both times.

[apl-haiku]

Hello @waddlesplash; I am running the build jam -q HaikuDepot two times on my linux x86_64 host and likewise on a Haiku x86_64 v.host and I am unable to see this problem at my end. HaikuDepot does not rebuild the second time. As a diagnostic measure, does this happen on a linux build slave for you?

You will also see a generated source file .../objects/haiku/x86_64/common/apps/haikudepot/dumpexportpkgmodel/dummy.dat (for example). Is this being written with a correct modify-timestamp on the host where you see the problem?

[waddlesplash]

That file exists, and it is not rebuilt each time, only the C++ files are.

[apl-haiku]

I take that to mean that the file dummy.dat does not get the modify-timestamp updated at each build, but the C++ files do have updated modify-timestamps because they are re-generated. Can you please run the following jam -dx -q HaikuDepot and see if there is a line such as...

touch objects/haiku/x86_64/common/apps/haikudepot/dumpexportrepositorymodel/dummy.dat

...appearing in the log trace.

[waddlesplash]

Full output attached.

[apl-haiku]

Strangely from that trace in out.txt, the builder does not appear to be generating the DTO sources using the python script nor is it compiling any part of that area. For some more diagnostics as to the actual cause, are you able to run this jam -dc -q HaikuDepot and attach the output of that. Hopefully this will show what the trigger of the unneeded build is.

[apl-haiku]

After discussing with @waddlesplash on IRC we see that a difference between my system (where I do not see the problem) and his system is that I am building from a "generated" directory and he is building from the source root. In a fresh directory I setup a new build from the source root and it also does not seem to repeat-build.

[waddlesplash]

Attached.

[apl-haiku]

Hello @waddlesplash; Could I please ask you to get the last modified date on the following object;

generated/objects/haiku/x86_64/common/apps/haikudepot/dumpexportrepositorymodel/dummy.dat

[waddlesplash]

ls -al generated/objects/haiku/x86_64/common/apps/haikudepot/dumpexportrepositorymodel/dummy.dat
-rw-r--r-- 1 user root 0 Jan 26 16:15 generated/objects/haiku/x86_64/common/apps/haikudepot/dumpexportrepositorymodel/dummy.dat

[waddlesplash]

Fixed in hrev52901. Thanks!

[waddlesplash]

Seems this causes build failures: ​https://build.haiku-os.org/buildbot/#builders/56/builds/220

[korli]

Still with us: ​https://build.haiku-os.org/buildbot/#/builders/46/builds/236

[apl-haiku]

Missed comment 11; will try look into the problem soon.

[apl-haiku]

In case it expires; here is the trace...

---

The failed run looks like;

...
MkDir1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener 
warning: using independent target objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener/dummy.dat 
HdsSchemaGenBulkParser1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener/dummy.dat 
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener/DumpExportRepositoryJsonListener.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener/DumpExportRepositoryJsonListener.h 
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgbulklistener/DumpExportPkgJsonListener.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgbulklistener/DumpExportPkgJsonListener.h 
MkDir1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel 
warning: using independent target objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/dummy.dat 
MkDir1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel 
warning: using independent target objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/dummy.dat 
HdsSchemaGenModel1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/dummy.dat 
touch: cannot touch 'objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/dummy.dat': No such file or directory
touch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/dummy.dat

...failed HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/dummy.dat ...

HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkg.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgCategory.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgScreenshot.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgVersion.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkg.h objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgCategory.h objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgScreenshot.h objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgVersion.h 
BUILD FAILURE:

...failed updating 1 target(s)...

A successful run looks like;

HdsSchemaGenBulkParser1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgbulklistener/dummy.dat 
SetType1 objects/haiku/x86_gcc2/release/libs/glut/libglut.so 
MimeSet1 objects/haiku/x86_gcc2/release/libs/glut/libglut.so 
SetVersion1 objects/haiku/x86_gcc2/release/libs/glut/libglut.so 
Chmod1 objects/haiku/x86_gcc2/release/libs/glut/libglut.so 
Link objects/haiku/x86_gcc2/release/tests/kits/opengl/glinfo/GLInfo 
XRes1 objects/haiku/x86_gcc2/release/tests/kits/opengl/glinfo/GLInfo 
SetType1 objects/haiku/x86_gcc2/release/tests/kits/opengl/glinfo/GLInfo 
MimeSet1 objects/haiku/x86_gcc2/release/tests/kits/opengl/glinfo/GLInfo 
SetVersion1 objects/haiku/x86_gcc2/release/tests/kits/opengl/glinfo/GLInfo 
Chmod1 objects/haiku/x86_gcc2/release/tests/kits/opengl/glinfo/GLInfo 
CreateAppMimeDBEntries1 objects/haiku/x86_gcc2/release/tests/kits/opengl/glinfo/GLInfo_mimedb 
AddDirectoryToContainerCopyFilesScript <hpkg_-haiku.hpkg>data/mime_db/-/<mimedb-app->GLInfo_mimedb 
MkDir1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener 
warning: using independent target objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener/dummy.dat 
HdsSchemaGenBulkParser1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener/dummy.dat 
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgbulklistener/DumpExportPkgJsonListener.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgbulklistener/DumpExportPkgJsonListener.h 
MkDir1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel 
warning: using independent target objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/dummy.dat 
HdsSchemaGenModel1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/dummy.dat 
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener/DumpExportRepositoryJsonListener.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorybulklistener/DumpExportRepositoryJsonListener.h 
MkDir1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel 
warning: using independent target objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/dummy.dat 
HdsSchemaGenModel1 objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/dummy.dat 
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/DumpExportRepository.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/DumpExportRepositorySource.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/DumpExportRepositorySourceMirror.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/DumpExportRepository.h objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/DumpExportRepositorySource.h objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportrepositorymodel/DumpExportRepositorySourceMirror.h 
C++ objects/haiku/x86_gcc2/release/apps/haikudepot/CompoundEdit.o 
HdsSchemaGenTouch objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkg.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgCategory.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgScreenshot.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgVersion.cpp objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkg.h objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgCategory.h objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgScreenshot.h objects/haiku/x86_gcc2/common/apps/haikudepot/dumpexportpkgmodel/DumpExportPkgVersion.h 

[apl-haiku]

I think this was to do with concurrently; a race condition between touching the "dummy file" and creating the directory in which the file was located. I have made a change such that the creation of the directory, running the python and touching the "dummy file" happen as a series of script commands rather than being Jam actions.