JPEG Translator not read large files.

[Premislaus]

​http://www.edtruthan.com/mars/Sol177-MAHLI-MSL-Self-Portrait-22000x17000px.zip - I have a problem with this file! JPEG Translator is not able to read it.

[pdziepak]

The size of the picture after decompression is 22000 * 17000 * 4 which is more than 1.4 GB. On x86 process heap starts at 384 MB. Lower addresses are partially occupied by elf images and addresses near the top of address space available for process are occupied by stacks. Any attempt to allocate anything big (including temporary buffers for decompression) will fail.

[jscipione]

Another example: ​http://www.nasa.gov/images/content/618486main_earth_full.jpg

[jscipione]

WebPositive trying to open large jpeg

[jscipione]

To my tremendous surprise, ShowImage opened the file linked above without issue (once I downloaded it via wget).

[jscipione]

8000x8000 jpg open in ShowImage

[Alexco]

For me this is not an error in the JPEG translator. The translator only allocates memory for 2 scan lines. The output buffer is allocated by the client application, e.g. Showimage. This can be tested with the command line tool translate:

translate --verbose image.jpg output.bin bits

The image is translated into a B_TRANSLATOR_BITMAP without issue. For the example linked above (Sol177...) this results in a ~1.4GB file, which is too large to be handled by a simple memory buffer used by Showimage:

From ImageCache.cpp:

	// Translate image data and create a new ShowImage window

	BBitmapStream outstream;

	status = roster->Identify(&file, &ioExtension, &info, 0, NULL,
		B_TRANSLATOR_BITMAP);
	if (status == B_OK) {
		status = roster->Translate(&file, &info, &ioExtension, &outstream,
			B_TRANSLATOR_BITMAP);
	}
	if (status != B_OK)
		return status;

	BBitmap* bitmap;
	if (outstream.DetachBitmap(&bitmap) != B_OK)
		return B_ERROR;
	.
	.
	.

Possible solution would be to change Showimage and use some kind of memory mapped file, but there seems no support in BFile and/or BBitmapStrem/BBitmap for this(?).

[pulkomandy]

Indeed, BBitmapStream itself does not hande this because its goal is to create a BBitmap, which is an in-memory structure.

An on-disk buffer does not seem like the best thing to do, however (do we really want to store gigabytes of decoded data on disk for this?)

I think the right strategy would be to have the translator accept extra parameters in the BMessage* ioExtension passed to it, so that ShowImage (or other apps) could request a bitmap with a reduced size, or only a small part of the complete image (if zoomed in).

[Alexco]

I think this needs some more thinking.

Scenario 1 "partial request"

The application would request the translated buffer from a translator. Since the translator is (most likely) deleted after the operation, it would request a new translation everytime the user scrolls up/down/left/right and the end of the image is reached. For me this sounds strange, since it most probably will result in sluggish behaviour of the application. If you use the example here, translate needs 5-7 seconds on my i7 here.

Scenario 2 "reduced size"

For viewing okay, but what about editing? I mean you want to edit the original and not just some kind of "low quality copy". Editing the original picture would still be impossible

Scenario 3 "memory mapped file"

Application would not create a BBitmapStream, but would create a temporary BBitmap on disk. This temp file would be memory mapped and be used to create an in-memory BBitmap or BBitmapStream so that the application can decide what to show and how to handle. Comming from Objective-C, I would create a NSData object from the file with memory mapped flag set and feed this into whatever consumer I have...

Questions:

• How does x64 Haiku behave? Showimage should have no problems allocating enough memory
• How is this handled on other 32-Bit OS?
• Any other ideas?

[pulkomandy]

Scenario 1: translate takes a long time because it decodes the whole image. It should be faster to decode just one part of the picture (in any case, much faster than a memory mapped file that goes into the hard disk). This relies on the translator being able to decode just one part of the picture, however, which may not be possible on all platforms.

The "display" tool from ImageMagick can do things like this, so maybe we should look at their code.

For editing, the matter is different. The editor would have its own storage, and if large files should be supported, then it would indeed use an on-disk backing store (probably cut in small tiles to allow fast scrolling). This is up to the editor to do, but it will still need a way to populate the tiles from the translators, so it would decode piece-wise.

[Alexco]

Looking at this ​http://www.imagemagick.org/Usage/files/#massive gives also some ideas, but I still doubt that all that should be handled by a translator. Cropping could be done, though (even if it is rather painful for JPG).

BTW #7740 gives some infos about x64 and large Jpegs, too.