Very large JPEG crashes WebPositive

[jstressman]

When attempting to load a very large image, in this case the full size JPEG image from the link at the bottom of this page:   ​http://photojournal.jpl.nasa.gov/catalog/PIA15689 (not the link at the top, which is considerably smaller, even though it says full size at the top as well)

The image is 23096 x 7981 pixels, 13.15MB

WebPositive crashes every time.

[jstressman]

backtrace of the crash.

[jstressman]

syslog from after the crash (shows me crashing it a couple times)

[jstressman]

There's an old closed ticket #7740 that I thought might be related to this, and seems to point to the same memory issue.

I have 12GB of mem in this system, so I would think it should either load it anyway, or at least handle it a bit more gracefully.

[pulkomandy]

Still crashing as of hrev46516. Complete backtrace saved with Debugger.

[pulkomandy]

So, what happens is app_server does not want to allocate a bitmap that big. I added a check for this in Web+ and I can avoid the crash, rendering a white page instead.

The image needs about 700MB of memory when decoded. ShowImage manages to get this going, but it seems WebKit is keeping two copies of the image, which gets dangerously close to the limit of 2GB of user space memory. This may be because of the incremental decoding feature, where intermediate versions of the image are rendered while it's still loading.

Finally, when you navigate away from the page, it seems the memory is not freed immediately. This will prevent allocating even very small bitmaps in other parts of the code, with a similar crash (I got an allocation failure on an 1x20 pixels bitmap this way...)

I could try to reduce the memory use for this case, but it would be possible to get the same problem by loading a bigger image anyway.

[jstressman]

Wasn't there a discussion somewhere about this that included talk of breaking up the image in order to handle it better?

Like either only loading into memory the part that would actually be displayed (if full size), or only a scaled version if viewed scaled or something.

"Tiling" or something. (Sorry for my non-technical explanation.)

Is this, or would this be, included on the OS side rather than on an application specific basis?

[pulkomandy]

jstressman: while the image can be tiled once decoded, it's not really possible to decode only a part of a JPEG file. So this would need saving the decoded bitmap to disk as it is decoded, and then reading tiles from that decoded "cache".

While we could implement this OS-wide in the translator API, it wouldn't solve the problem for Web+, which uses libjpeg directly.

Closing this issue since we don't crash anymore. I don't think there is a reasonable way to handle this, and ShowImage can be used without problems.