Update to OpenEXR 3

[pulkomandy]

The EXR translator uses OpenEXR 2.x but HaikuWebKit uses 3.x.

To avoid shipping the two versions in Haiku images we should migrate the EXR translator to OpenEXR 3.

Is it possible to have it work with both? OpenEXR 2 may stil lbe needed for the gcc2 version of the translator.

[waddlesplash]

It appears the version of OpenEXR 3.x isn't the most recent either. (Also the package is called "openexr30", that's confusing and should probably be fixed.)

[KitsunePrefecture]

On Haiku 32 bit I found 3 versions of Open EXR - installed :

~> cd /boot/system/packages/; ls -l *exr* -rw-r--r-- 1 user root 1311456 nov. 14 2022 openexr_x86-2.4.1-1-x86_gcc2.hpkg -rw-r--r-- 1 user root 1870024 nov. 14 2022 openexr-2.2.1-3-x86_gcc2.hpkg -rw-r--r-- 1 user root 1281733 márc. 17 2023 openexr30_x86-3.0.5-1-x86_gcc2.hpkg /boot/system/packages>

I found these several versions at at an earlier case I checked HPKG(s) of Open EXR. (#18560)

[cocobean]

Technically, openexr_x86-2.4.1-1-x86_gcc2.hpkg is the dependency for haiku_datatranslators.

OpenEXR 2.2.1 for GCC2.

OpenEXR 3.0.5 - all else (haikuports, haikuwebkit, etc)

Latest: OpenEXR v3.2.1.

[cocobean]

Updated and fixed packaging of openexr for OpenEXR 3.2.1. Tested builds of dependent packages of OpenEXR within Haikuports. No known build or migration issues so far.

[KitsunePrefecture]

Does it mean ..

soon one OpenEXR package will update both packages on x86_64 Haiku ?

~> ls -l ../system/packages/*exr*
-rw-r--r-- 1 user root 1381944 nov.   26 23:10 ../system/packages/openexr-2.4.1-2-x86_64.hpkg
-rw-r--r-- 1 user root 1267222 nov.   26 23:11 ../system/packages/openexr30-3.0.5-1-x86_64.hpkg
~> pkgman update openexr30
100% repochecksum-1 [65 bájt]
Ellenőrzőkód érvényesítése a tárolóhoz (BeSly Software Solutions)...done.
100% repochecksum-1 [65 bájt]
Ellenőrzőkód érvényesítése a tárolóhoz (FatElk_64)...done.
100% repochecksum-1 [65 bájt]
Ellenőrzőkód érvényesítése a tárolóhoz (Haiku)...done.
100% repocache-2 [2.38 KiB]
Ellenőrzőkód érvényesítése a tárolóhoz (Haiku)...done.
100% repochecksum-1 [64 bájt]
Ellenőrzőkód érvényesítése a tárolóhoz (HaikuPorts)...done.
100% repochecksum-1 [71 bájt]
Ellenőrzőkód érvényesítése a tárolóhoz (LOTE)...done.
Nothing to do.
~> pkgman update openexr
100% repochecksum-1 [65 bájt]
Ellenőrzőkód érvényesítése a tárolóhoz (BeSly Software Solutions)...done.
100% repochecksum-1 [65 bájt]
Ellenőrzőkód érvényesítése a tárolóhoz (FatElk_64)...done.
100% repochecksum-1 [65 bájt]
Ellenőrzőkód érvényesítése a tárolóhoz (Haiku)...done.
100% repochecksum-1 [64 bájt]
Ellenőrzőkód érvényesítése a tárolóhoz (HaikuPorts)...done.
100% repochecksum-1 [71 bájt]
Ellenőrzőkód érvényesítése a tárolóhoz (LOTE)...done.
Nothing to do.
~> 
{{{

Kind regards,
}}}

[cocobean]

Yes. OpenEXR 3.2.1 for x86 and x86_64 (GCC 13+). OpenEXR 2.2.1 for x86_gcc2 (GCC 2).

There are small issues with the CI build system with the translator"

Seems like it downloads (x86_64):

openexr30-3.0.5-1-x86_64.hpkg

openexr-2.4.1-1-x86_64.hpkg

​https://ci.haiku-os.org/teams/nightly/pipelines/master-x86_64/jobs/compile-master-x86_64/builds/373

...on 15200th target...
AppendToContainerCopyFilesScript <hpkg_-haiku_datatranslators.hpkg>haiku.package-copy-files-dummy-data/locale/catalogs/x-vnd.Haiku-BMPTranslator 
MkDir1 objects/catalogs/add-ons/translators/exr 
ExtractCatalogEntries1 objects/catalogs/add-ons/translators/exr/en.catkeys 
../haiku-git/src/add-ons/translators/exr/ConfigView.cpp:17:10: fatal error: OpenEXRConfig.h: No such file or directory
   17 | #include <OpenEXRConfig.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
In file included from ../haiku-git/src/add-ons/translators/exr/EXRTranslator.cpp:11:
../haiku-git/src/add-ons/translators/exr/EXRGamma.h:9:10: fatal error: halfFunction.h: No such file or directory
    9 | #include "halfFunction.h"
      |          ^~~~~~~~~~~~~~~~
compilation terminated.

[pulkomandy]

Moving to beta5 as this is listed in #18565

[Begasus]

Checking with readelf -d I see both translators in system and in the generated directory still use openEXR 2.4

Looking into the "download" directory in the generated directory I only the devel package for 2.4 also in there.

Builds on 64bit R1B4 so far never failed on me, anything I could do to check here?

[pulkomandy]

If you want to test OpenEXR 3, you have to add the needed packages manually in the download directory. Then edit the repo package list (see for example ​https://review.haiku-os.org/c/haiku/+/7718/1) to include it there.

Then you have to run jam using -sHAIKU_NO_DOWNLOADS=1 to force Haiku building without downloading packages from the repos (it will use the ones that are in the download directory only, and fail if something is missing).

This way you can try to make a build using openexr 3. Adjust other packages in the package list file if needed (and add them to the download directory) until all dependencies can be solved and the Haiku build works.

[korli]

as the package name changed, it also has to be changed: ​https://github.com/haiku/haiku/blob/6a0e78e576600beac1fd36a15598dfc5b945fd58/build/jam/BuildFeatures#L569

[Begasus]

Thanks both, got a bit further and when the build starts I'm not getting a message for the missing openexr like build-feature packages unavailable on x86_64: openexr

I still can't see any exr in the build translator add-ons directory, is it possible to run a specific jam command only for the translator?

[Begasus]

Thanks to zard on IRC I found out how to target the EXR translator. The build fails however with:

../src/add-ons/translators/exr/ConfigView.cpp:17:10: fatal error: OpenEXRConfig.h: No such file or directory
   17 | #include <OpenEXRConfig.h>
      |          ^~~~~~~~~~~~~~~~~


../src/add-ons/translators/exr/EXRGamma.h:9:10: fatal error: halfFunction.h: No such file or directory
    9 | #include "halfFunction.h"
      |          ^~~~~~~~~~~~~~~~

I changed in build/jam/repositories/haikuports/x86_64 to

	openexr-2.4.1-1
	openexr30-3.0.5-1
	openexr30_devel-3.0.5-1

And in build/jam/BuildFeatures:

# openexr
if [ IsPackageAvailable openexr30_devel ] {
	ExtractBuildFeatureArchives openexr30 :
		file: base openexr30
			runtime: lib
		file: devel openexr30_devel
			depends: base
			libraries:
				$(developLibDir)/libIex.so
				$(developLibDir)/libIlmThread.so
				$(developLibDir)/libImath.so
				$(developLibDir)/libOpenEXR.so
				$(developLibDir)/libOpenEXRUtil.so
			headers: $(developHeadersDir) $(developHeadersDir)/OpenEXR $(developHeadersDir)/Imath
		;

	EnableBuildFeatures openexr30 ;
} else {
	unavailableBuildFeatures += openexr30 ;
}

One header is in the OpenEXR headers dir and the other in Imath headers dir.

[Begasus]

First draft diff file to get the build started

[Begasus]

Build errors after changed files for openexr30

[korli]

It seems to me that the EXR translator doesn't build with OpenEXR 3.0.

[Begasus]

With the help of madmax I was able to build the translator. :)

[Begasus]

Builds OK on 64bit R1B4 with this patch, credits to madmax

[korli]

a new PR with ​https://review.haiku-os.org/c/haiku/+/7718 as parent would be nice :)

[Begasus]

Whoops, I've never done anything like that at gerrit (never done anything "with" gerrit for that matter. I'm fine if someone else could check and add it. :)

fwiw, it's not included in the booted image (yet) :)

[Begasus]

Slimmed down some parts (partially with madmad advice)

[Begasus]

Forget patch rev3, missed the update in there. :)

[Begasus]

This should be the correct (slimmed down) patch, that worked here.

[KitsunePrefecture]

Does it mean,...

... a new one nail has beeing hammered in again

that had blocked R1/Beta5 ?

[korli]

HAIKU_BUILD_FEATURE_%HAIKU_PACKAGING_ARCH%_openexr30_ENABLED doesn't look correct as the build feature name stays openexr

[Begasus]

Good call, seems I forgot to revert that one. Just for checks, I'm trying with the new (local) openexr-3.2.4, so far jxl needed a revbump for it, but getting an error now on brotli while Resolving package dependencies ...

Encountered problems:
problem 1: nothing provides lib:libbrotlicommon>=1.1.0 needed by libjxl-0.6.1-5
  solution 1:
    - allow deinstallation of webpositive-r1~beta4_hrev57768_dirty-1

I changed refs for both in the x86_64 file and cp'd the files into the download dir.

EDIT: that tip fixed the problem for brotli.

[Begasus]

OK, not saying to switch to openexr-3.2.4, but got it to build :)

Is there a "quick" way to see if it "actually" works?

[Begasus]

Screenshot with openexr-3.2.4

[korli]

Test images: ​https://openexr.com/en/latest/test_images/index.html#test-images

Maybe it would be better to move EXRTranslator to HaikuArchives like ​https://github.com/HaikuArchives/HPGSTranslator, and provide a recipe. This way it can easierly be upgraded.

[Begasus]

Looks to be working fine, I can open the exr images fine with ShowImage, on the move to HaikuArchives, that's up to you folks around here. :) PS, there is another one at haikuports for jxl hosted here: ​https://github.com/wattoc/JXLTranslator

[Begasus]

Screenshot with example images

[korli]

x86_gcc2 should still use openexr, I think.

[Begasus]

This this should do it? Attatching a diff for openexr30 and one for openexr3.2

For 32bit we should still fallback to older openexr, no chance on getting the 3.* version build with gcc2, so not sure where to go from here.

[Begasus]

Diff for openexr30

[Begasus]

Diff for openexr3.2

[Begasus]

Took the liberty to try out a seperate repository for this, used the one mentioned earlier here on HaikuArchives as an example. I can build (make) the translator in Terminal for both architectures, the master branch contains the changes needed for openexr3.*, the 32bit one still falls back on the original source.

​https://github.com/Begasus/EXRTranslator

Maybe coould be used as an example to move the changes to HaikuArchives? (PS, locales not included there)

[korli]

Cool, you could keep the old version for openexr 2.x in a branch.

[Begasus]

Well first run with installed addon produced a crash with ShowImage :)

Created draft recipe to build/install it.

[Begasus]

Got it! (64bit, 32bit will be a check for tomorrow). EXRTranslator build, packaged with haikuporter and installed.

[Begasus]

Installed 64bit addon with openexr 3.2.4

[Begasus]

OK, I got it working for both arch's now, I'm still facing an issue with SONAME, created an issue at: ​https://github.com/Begasus/EXRTranslator/issues/2 Probably should be taken there, anyone with some insight on what's happening is welcome. :)

[Begasus]

Drafter a recipe at: ​https://github.com/Begasus/haikuports/tree/exrtranslator

If this could be used maybe the code could be moved to HaikuArchives? (PS, haven't fiddled with locales yet).

[waddlesplash]

EXR images are mostly used by professionals, right? I wouldn't be opposed to outsourcing the EXR translator and removing it from the default install.

[waddlesplash]

HaikuPorts now has an EXRTranslator recipe, so we should be able to drop it from the main Haiku repository.

[waddlesplash]

EXRTranslator removed in hrev57820.

[jmairboeck]

Some of the translations for the OpenEXR translator were added back with the latest pootle sync in ​https://cgit.haiku-os.org/haiku/commit/?id=4aa39172c0a6dac49c9c262dbf67077ba6b4212b, this time directly in data/catalogs. It appears that there is something wrong still with pootle.