app_server: crash when zooming picture

[X512]

This is hrev54888.

Steps to reproduce:

1. Open attached picture in ​PictureView.
2. Zoom picture with mouse wheel.
3. app_server will crash. Also global freezes will occur.

Can be reproduced in test_app_server. When building PictureView, be should be replaced with be_test in Makefile.

[X512]

Crash report for debug build of test_app_server

[waddlesplash]

Looks plausibly related to the guarded_heap crash in #16489.

[X512]

It can be uint32 overflow in ServerBitmap. When analyzing one of crashes, Alpha mask buffer bitmap size was 46848 x 27648 and color space was B_RGBA32. 46848*27648*4 = 0x1337e8000 > UINT32_MAX.

[waddlesplash]

Why are the alpha mask buffers so large? Is the picture zoomed that big? And, aren't alpha masks 1-byte?

[X512]

Is the picture zoomed that big?

Yes. But some kind of clipping probably can be introduced.

And, aren't alpha masks 1-byte?

No, because Painter don't support buffer formats except B_RGBA32: ​https://git.haiku-os.org/haiku/tree/src/servers/app/drawing/Painter/Painter.cpp#n230.

After adding proper overflow check, it now crash in different place.

[X512]

Crash after adding overflow check in ServerBitmap

[X512]

Alpha mask is a good motivation for implementing 8 bit rendering in Painter.

[X512]

Second crash seems to be NULL dereference to bitmap buffer that was failed to allocate.

[waddlesplash]

If you zoom in too far, it won't redraw anymore, and if you zoom at all the text disappears, but those seem like separate issues, and the crash itself is gone.

[waddlesplash]

CC'ing madmax, as he may want to investigate the "text disappears when zooming picture" issue (or perhaps he already knows of it, and this might be an easier way to test.)

[pulkomandy]

And, aren't alpha masks 1-byte?

No, you can have a subpixel precise alpha mask if you want to. For example this allows you to render some text in the alpha mask, and then fill it with a gradient or some other fun things, and preserve the subpixel antialiasing of the text still.

[X512]

AGG support RLE-like compressed masks.