Opened 6 years ago

Last modified 12 months ago

#9496 new bug

JPEG Translator not read large files.

Reported by: Premislaus Owned by: nobody
Priority: normal Milestone: R1
Component: Add-Ons/Translators/JPEG Version: R1/Development
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: x86

Description

http://www.edtruthan.com/mars/Sol177-MAHLI-MSL-Self-Portrait-22000x17000px.zip - I have a problem with this file! JPEG Translator is not able to read it.

Attachments (4)

ShowImage.png (114.9 KB) - added by Premislaus 6 years ago.
WonderBrush.png (74.1 KB) - added by Premislaus 6 years ago.
WebPositive-848-debug-28-12-2013-03-43-08.report.txt (45.5 KB) - added by jscipione 5 years ago.
WebPositive trying to open large jpeg
main_earth_full_showimage.png (522.0 KB) - added by jscipione 5 years ago.
8000x8000 jpg open in ShowImage

Download all attachments as: .zip

Change History (12)

Changed 6 years ago by Premislaus

Attachment: ShowImage.png added

Changed 6 years ago by Premislaus

Attachment: WonderBrush.png added

comment:1 Changed 6 years ago by pdziepak

The size of the picture after decompression is 22000 * 17000 * 4 which is more than 1.4 GB. On x86 process heap starts at 384 MB. Lower addresses are partially occupied by elf images and addresses near the top of address space available for process are occupied by stacks. Any attempt to allocate anything big (including temporary buffers for decompression) will fail.

Last edited 6 years ago by pdziepak (previous) (diff)

Changed 5 years ago by jscipione

WebPositive trying to open large jpeg

comment:3 Changed 5 years ago by jscipione

To my tremendous surprise, ShowImage opened the file linked above without issue (once I downloaded it via wget).

Changed 5 years ago by jscipione

8000x8000 jpg open in ShowImage

comment:4 Changed 12 months ago by Alexco

For me this is not an error in the JPEG translator. The translator only allocates memory for 2 scan lines. The output buffer is allocated by the client application, e.g. Showimage. This can be tested with the command line tool translate:

translate --verbose image.jpg output.bin bits

The image is translated into a B_TRANSLATOR_BITMAP without issue. For the example linked above (Sol177...) this results in a ~1.4GB file, which is too large to be handled by a simple memory buffer used by Showimage:

From ImageCache.cpp:

	// Translate image data and create a new ShowImage window

	BBitmapStream outstream;

	status = roster->Identify(&file, &ioExtension, &info, 0, NULL,
		B_TRANSLATOR_BITMAP);
	if (status == B_OK) {
		status = roster->Translate(&file, &info, &ioExtension, &outstream,
			B_TRANSLATOR_BITMAP);
	}
	if (status != B_OK)
		return status;

	BBitmap* bitmap;
	if (outstream.DetachBitmap(&bitmap) != B_OK)
		return B_ERROR;
	.
	.
	.

Possible solution would be to change Showimage and use some kind of memory mapped file, but there seems no support in BFile and/or BBitmapStream/BBitmap for this(?).

Last edited 12 months ago by Alexco (previous) (diff)

comment:5 Changed 12 months ago by pulkomandy

Indeed, BBitmapStream itself does not hande this because its goal is to create a BBitmap, which is an in-memory structure.

An on-disk buffer does not seem like the best thing to do, however (do we really want to store gigabytes of decoded data on disk for this?)

I think the right strategy would be to have the translator accept extra parameters in the BMessage* ioExtension passed to it, so that ShowImage (or other apps) could request a bitmap with a reduced size, or only a small part of the complete image (if zoomed in).

comment:6 Changed 12 months ago by Alexco

I think this needs some more thinking.

Scenario 1 "partial request"

The application would request the translated buffer from a translator. Since the translator is (most likely) deleted after the operation, it would request a new translation everytime the user scrolls up/down/left/right and the end of the image is reached. For me this sounds strange, since it most probably will result in sluggish behaviour of the application. If you use the example here, translate needs 5-7 seconds on my i7 here.

Scenario 2 "reduced size"

For viewing okay, but what about editing? I mean you want to edit the original and not just some kind of "low quality copy". Editing the original picture would still be impossible

Scenario 3 "memory mapped file"

Application would not create a BBitmapStream, but would create a temporary BBitmap on disk. This temp file would be memory mapped and be used to create an in-memory BBitmap or BBitmapStream so that the application can decide what to show and how to handle. Comming from Objective-C, I would create a NSData object from the file with memory mapped flag set and feed this into whatever consumer I have...

Questions:

  • How does x64 Haiku behave? Showimage should have no problems allocating enough memory
  • How is this handled on other 32-Bit OS?
  • Any other ideas?

comment:7 Changed 12 months ago by pulkomandy

Scenario 1: translate takes a long time because it decodes the whole image. It should be faster to decode just one part of the picture (in any case, much faster than a memory mapped file that goes into the hard disk). This relies on the translator being able to decode just one part of the picture, however, which may not be possible on all platforms.

The "display" tool from ImageMagick can do things like this, so maybe we should look at their code.

For editing, the matter is different. The editor would have its own storage, and if large files should be supported, then it would indeed use an on-disk backing store (probably cut in small tiles to allow fast scrolling). This is up to the editor to do, but it will still need a way to populate the tiles from the translators, so it would decode piece-wise.

comment:8 Changed 12 months ago by Alexco

Looking at this http://www.imagemagick.org/Usage/files/#massive gives also some ideas, but I still doubt that all that should be handled by a translator. Cropping could be done, though (even if it is rather painful for JPG).

BTW #7740 gives some infos about x64 and large Jpegs, too.

Note: See TracTickets for help on using tickets.