static (hidden) thread_local values do not work in some cases

[waddlesplash]

gnutls makes use of this, and it had to be disabled to fix crashes: ​https://github.com/haikuports/haikuports/commit/cda76a62712b91647892ba01f1a5d0dcb38cadc5

[waddlesplash]

CC korli, perhaps you have some idea about what to do here?

[waddlesplash]

Hmm, I guess this counts as "static TLS", which would be #13294. But we should get an error and the program shouldn't load at all in that case...

[waddlesplash]

Ah, right, since this is a shared object, it does not count as "static TLS", I don't think, so this should in theory work. But it appears at least in this test application that putting the "_Thread_local" variable outside the function and deleting the "static" doesn't make it work either, in fact in that case I get all 5 having the same pointer value!

[waddlesplash]

A bit more testing shows that __tls_get_addr in libroot is indeed returning bad values:

__tls_get_addr: thread 2224, module 0, offset (nil) -> 0x48bc539990
__tls_get_addr: thread 2221, module 0, offset (nil) -> 0x48bc5398f0
__tls_get_addr: thread 2225, module 0, offset (nil) -> 0x48bc5398f0
__tls_get_addr: thread 2222, module 0, offset (nil) -> 0x48bc5398f0
__tls_get_addr: thread 2223, module 0, offset (nil) -> 0x48bc5398f0

That pushes the problem up into runtime_loader indeed.

[waddlesplash]

This does not seem to be a race condition. Adding snooze(100000) into both for loops actually has the effect that we always seem to get the same pointer for all threads!

[waddlesplash]

Got it: the TLS_DYNAMIC_THREAD_VECTOR has different pointers for each thread (good, that would mean base TLS is broken otherwise) but they all point to the same place. Now how did that happen...

[waddlesplash]

Ah, so what is happening is that the threads are all exiting before the next one even starts, and so the TLS memory gets freed when the thread exits and then reused when the next thread starts. Adding snooze(10000); to the end of the thread function and removing the snoozes from elsewhere shows all variables have different values indeed.

I then readded the static and things still worked just fine. So I guess I am not sure what the problem with gnutls is that it works when the thread-local variables are non-static...

[waddlesplash]

<X512> I found that after writing to TLS variable it read as zero.
<X512> Removing static fixed that.
<X512> I tried to make mores simple case, but it worked fine...
<X512> Write value to static TLS variable and immediately after read it. It will become zero.
<X512> I suspect that GCC use some optimization for static TLS variable that may use TLS model unsupported in Haiku.

[waddlesplash]

More likely it seems to me is that somehow the generation-accounting logic in runtime_loader's elf_tls.cpp is not correct, and blocks are getting spuriously deleted and recreated. Comparing it to FreeBSD I note a bunch of differences, but based on commit messages some of these may be intentional.

[waddlesplash]

Here's a difference of the gcc -S output when compiling random.c from GNUTLS with static vs no static on the variable declarations. Note the .value 0x6666: what's that about?

[waddlesplash]

I also did find what I am pretty sure is an issue in the TLS DTV handling, but I think it's not necessarily related to this: ​https://review.haiku-os.org/c/haiku/+/5604