Infinite recursion in agg rasterizer

[madmax]

app_server crash while running webkit test ​fast/css/border-image-scale-crash.html.

It seems the crash is ultimately due to infinite recursion in ​agg::rasterizer_cells_aa<Cell>::line. dx, dy, cx and cy can overflow. In particular, this one comes from a call with (0, 0, -2147483648, 0). After some splitting you get x1=-715827883 and x2=-2147483648, where cx becomes 715827882 instead of -1431655765 and the next split for cx..x2 is again at -715827883.

The -2147483648 value seems to come from another overflow in ​upscaling 128441500.0.

[waddlesplash]

We really should collaborate with some of the other agg 2.4 forks that are out there. I have an item on my TODO list to create a "haiku-agg" repository at GitHub and merge all our patches into it, and then try to merge some from the various other places agg has been forked.

[X512]

I got this on riscv64.

STrap(interrupt sTimer)
  sstatus: (ie: {}, pie: {s}, spp: u, sum: 0)
  sie: {sSoft, sTimer, sExtern}
  sip: {sTimer}
  stval: 0x0
  tp: 0xffffffc13ff1f900(w:1697:ThreadTests2.log)
Stack:
FP: 0xffffffc12ba58e00
FP: 0xffffffc12ba58f00, PC: 0xffffffc00215d7ed <kernel_riscv64> STrap + 1185
FP: 0xffffffc12ba59000, PC: 0xffffffc00215b29d <kernel_riscv64> SVecU + 109
FP: 0x37d0dd3080, PC: 0x112f78ab5b <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 5
FP: 0x37d0dd3130, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd31e0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3290, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3340, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd33f0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd34a0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3550, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3600, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd36b0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3760, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3810, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd38c0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3970, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3a20, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3ad0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3b80, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3c30, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3ce0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3d90, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3e40, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3ef0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd3fa0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd4050, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd4100, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd41b0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd4260, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd4310, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd43c0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd4470, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd4520, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd45d0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd4680, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd4730, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797
FP: 0x37d0dd47e0, PC: 0x112f78ae73 <_APP_> _ZN3agg19rasterizer_cells_aaINS_7cell_aaEE4lineEiiii + 797

[X512]

Maybe this can be fixed by doing bounds check before passing coordinates to AGG?

[pulkomandy]

I have an item on my TODO list to create a "haiku-agg" repository at GitHub and merge all our patches into it, and then try to merge some from the various other places agg has been forked.

It seems very strange to me that the answer to "there are too many agg forks out there" is "let's create another agg fork". Can't we just use ​https://sourceforge.net/projects/agg/ ? As far as I know this is the upstream for agg (they had two new releases, even, 2.6.0 and 2.7.0), it is still active (not a lot, but this is a mature library with no ongoing development and only bugfixes) and we could just submit all the patches to there.

[waddlesplash]

They have no activity in a year, and there are a number of patches waiting to be merged there that nobody has dealt with, e.g.: ​https://sourceforge.net/p/agg/patches/2/ So I think that upstream is basically dead.

My idea with creating a new GitHub repository is that we could potentially invite in all of those other forks to collaborate, or at least to keep us apprised of their changes. Technically we already have a fork in our own tree, this would just make it into its own repository.

[marcoapc]

In the forum there are conversations with a lot of interest and results involving AGG:

​https://discuss.haiku-os.org/t/gsoc-2023-progress-on-perspective-transformation-haiku-project/13594/33

​https://discuss.haiku-os.org/t/agg-anti-grain-geometry-library/13913