Opened 12 years ago

Last modified 7 years ago

#9496 new bug

JPEG Translator not read large files.

Reported by: Premislaus Owned by: nobody
Priority: normal Milestone: R1
Component: Add-Ons/Translators/JPEG Version: R1/Development
Keywords: Cc:
Blocked By: Blocking:
Platform: x86

Description

http://www.edtruthan.com/mars/Sol177-MAHLI-MSL-Self-Portrait-22000x17000px.zip - I have a problem with this file! JPEG Translator is not able to read it.

Attachments (4)

ShowImage.png (114.9 KB ) - added by Premislaus 12 years ago.
WonderBrush.png (74.1 KB ) - added by Premislaus 12 years ago.
WebPositive-848-debug-28-12-2013-03-43-08.report.txt (45.5 KB ) - added by jscipione 11 years ago.
WebPositive trying to open large jpeg
main_earth_full_showimage.png (522.0 KB ) - added by jscipione 11 years ago.
8000x8000 jpg open in ShowImage

Download all attachments as: .zip

Change History (12)

by Premislaus, 12 years ago

Attachment: ShowImage.png added

by Premislaus, 12 years ago

Attachment: WonderBrush.png added

comment:1 by pdziepak, 12 years ago

The size of the picture after decompression is 22000 * 17000 * 4 which is more than 1.4 GB. On x86 process heap starts at 384 MB. Lower addresses are partially occupied by elf images and addresses near the top of address space available for process are occupied by stacks. Any attempt to allocate anything big (including temporary buffers for decompression) will fail.

Last edited 12 years ago by pdziepak (previous) (diff)

by jscipione, 11 years ago

WebPositive trying to open large jpeg

comment:3 by jscipione, 11 years ago

To my tremendous surprise, ShowImage opened the file linked above without issue (once I downloaded it via wget).

by jscipione, 11 years ago

8000x8000 jpg open in ShowImage

comment:4 by Alexco, 7 years ago

For me this is not an error in the JPEG translator. The translator only allocates memory for 2 scan lines. The output buffer is allocated by the client application, e.g. Showimage. This can be tested with the command line tool translate:

translate --verbose image.jpg output.bin bits

The image is translated into a B_TRANSLATOR_BITMAP without issue. For the example linked above (Sol177...) this results in a ~1.4GB file, which is too large to be handled by a simple memory buffer used by Showimage:

From ImageCache.cpp:

	// Translate image data and create a new ShowImage window

	BBitmapStream outstream;

	status = roster->Identify(&file, &ioExtension, &info, 0, NULL,
		B_TRANSLATOR_BITMAP);
	if (status == B_OK) {
		status = roster->Translate(&file, &info, &ioExtension, &outstream,
			B_TRANSLATOR_BITMAP);
	}
	if (status != B_OK)
		return status;

	BBitmap* bitmap;
	if (outstream.DetachBitmap(&bitmap) != B_OK)
		return B_ERROR;
	.
	.
	.

Possible solution would be to change Showimage and use some kind of memory mapped file, but there seems no support in BFile and/or BBitmapStream/BBitmap for this(?).

Last edited 7 years ago by Alexco (previous) (diff)

comment:5 by pulkomandy, 7 years ago

Indeed, BBitmapStream itself does not hande this because its goal is to create a BBitmap, which is an in-memory structure.

An on-disk buffer does not seem like the best thing to do, however (do we really want to store gigabytes of decoded data on disk for this?)

I think the right strategy would be to have the translator accept extra parameters in the BMessage* ioExtension passed to it, so that ShowImage (or other apps) could request a bitmap with a reduced size, or only a small part of the complete image (if zoomed in).

comment:6 by Alexco, 7 years ago

I think this needs some more thinking.

Scenario 1 "partial request"

The application would request the translated buffer from a translator. Since the translator is (most likely) deleted after the operation, it would request a new translation everytime the user scrolls up/down/left/right and the end of the image is reached. For me this sounds strange, since it most probably will result in sluggish behaviour of the application. If you use the example here, translate needs 5-7 seconds on my i7 here.

Scenario 2 "reduced size"

For viewing okay, but what about editing? I mean you want to edit the original and not just some kind of "low quality copy". Editing the original picture would still be impossible

Scenario 3 "memory mapped file"

Application would not create a BBitmapStream, but would create a temporary BBitmap on disk. This temp file would be memory mapped and be used to create an in-memory BBitmap or BBitmapStream so that the application can decide what to show and how to handle. Comming from Objective-C, I would create a NSData object from the file with memory mapped flag set and feed this into whatever consumer I have...

Questions:

  • How does x64 Haiku behave? Showimage should have no problems allocating enough memory
  • How is this handled on other 32-Bit OS?
  • Any other ideas?

comment:7 by pulkomandy, 7 years ago

Scenario 1: translate takes a long time because it decodes the whole image. It should be faster to decode just one part of the picture (in any case, much faster than a memory mapped file that goes into the hard disk). This relies on the translator being able to decode just one part of the picture, however, which may not be possible on all platforms.

The "display" tool from ImageMagick can do things like this, so maybe we should look at their code.

For editing, the matter is different. The editor would have its own storage, and if large files should be supported, then it would indeed use an on-disk backing store (probably cut in small tiles to allow fast scrolling). This is up to the editor to do, but it will still need a way to populate the tiles from the translators, so it would decode piece-wise.

comment:8 by Alexco, 7 years ago

Looking at this http://www.imagemagick.org/Usage/files/#massive gives also some ideas, but I still doubt that all that should be handled by a translator. Cropping could be done, though (even if it is rather painful for JPG).

BTW #7740 gives some infos about x64 and large Jpegs, too.

Note: See TracTickets for help on using tickets.