mail_daemon slowdown fetching mail from gmail

[humdinger]

As mentioned in this ​thread, mail_daemon recently became very slow when fetching mail from my gmail account. Via binary search, I narrowed it down to between releases hrev37437 (fast) and hrev37453 (slow). ​Timeline.

[jackburton]

The only related thing seems hrev37443, although the version of regex.c is the same.

[stippi]

You would have to do at most four more binary searching builds to obtain the exact revision when it broke. ;-)

[humdinger]

I took the lazy road and downloaded nightly builds and there aren't any between those revisions. I've hoped someone would spot the culprit in that relatively narrow range. I'd rather not down- and incrementally upgrade my workcopy and compile it all if I don't have to. Do I have to? :)

[bga]

There are 2 contention points:

1 - Parsing the email. This uses a BPositioIO object backed up by a BFile. Any changes related to IO or to memory accesses could trigger this.
2 - Saving the email to disk. This is taking a *LONG* time. Again changes to IO in general could be the culprit.

I will take a look at this ASAP.

[humdinger]

While I'd welcome any additional speedup, BGA, it's not a general slowness reported in this ticket.

Through binary search I have now definitely pinpointed the extreme slowdown to hrev37443. Up to that revision my GMail is fetched (pop3, ssl) with normal speed. From hrev37443 on it's slow as molasses.

[korli]

Replying to humdinger:

Through binary search I have now definitely pinpointed the extreme slowdown to hrev37443. Up to that revision my GMail is fetched (pop3, ssl) with normal speed. From hrev37443 on it's slow as molasses.

The libroot regex implementation seems to be older and buggier than the one which existed in mail kit. The switch could very well be the reason for the slowness problem.

[jackburton]

Replying to korli:

The libroot regex implementation seems to be older and buggier than the one which existed in mail kit. The switch could very well be the reason for the slowness problem.

That's weird, though, since the copyright of the libroot implementation seems to imply it's a newer version.

[jackburton]

Anyway, the quick fix would be revert hrev37443. Obviously would be nice if our libroot implementation would be updated instead.

[anevilyak]

Actually may not be that simple, at least from what I can see the regex.c referenced by that Jamfile isn't in the tree any more.

[korli]

Replying to anevilyak:

Actually may not be that simple, at least from what I can see the regex.c referenced by that Jamfile isn't in the tree any more.

A bit tired? It would come back as part of hrev37443 :)

[anevilyak]

Ah oops, didn't look carefully enough, the changeset browser made it look as if the only modification was regex.c being removed from the Jamfile. Never mind then :)

[korli]

see also hrev23792

[korli]

A proposition I implemented locally: update glibc regex to 2.11, headers/posix/regex.h included. This imposes some changes in some of our sources using the regex.h. Mostly bin tools and mail kit. The use of __USE_GNU macro is needed for instance when the expected regex implementation is the GNU one.

Can I commit this and let people validate ? I can imagine there are side effects but I don't see another way to be up to date.

[jackburton]

Replying to korli:

A proposition I implemented locally: update glibc regex to 2.11, headers/posix/regex.h included. This imposes some changes in some of our sources using the regex.h. Mostly bin tools and mail kit. The use of __USE_GNU macro is needed for instance when the expected regex implementation is the GNU one.

Can I commit this and let people validate ? I can imagine there are side effects but I don't see another way to be up to date.

I'd say go for it.

[axeld]

Does it impose a binary compatibility issue? If not, there is no reason not to proceed. But does it actually fix the problem, too? :-)

[korli]

Did so in hrev38031. I'm not sure about binary compatibility. We could define __USE_GNU by default in regex.h. It just defines things not required by POSIX.

[humdinger]

I'd love to try out the patch, but current revisions refuse to get me online and the patch doesn't apply cleanly to older revisions...

[axeld]

Then please file a separate bug report for that - AFAIK, the network is working fine.

[humdinger]

Figuring that hrev38078 would help I tried a current revision and lo! at least with static IP using ethernet it looked OK. I could ping my router but "installoptionalpackage" wouldn't download Web+ and just stuck on downloading the zip. After a reboot, the wifi was gone from the network panel, leaving the rtl81xx ethernet. Unfortunately, I now run into #6446...

[humdinger]

Since one of stippi's first contracting-commits fixed my sporadic app_server crashes with newer revisions and Axel finally ( :P ) fixed the networking issues (at least for static IPs for now) with my rtl81xx (thanks berregon for bughunting!), I could finally test a current revision (hrev38200). Yay! Thanks all[[BR]] I'm pleased to say that fetching mail from GMail is again as fast as it once was. Not sure which commit did it...

BGA is "in-progress" on it, so I'm not sure we can just pull it from under him...:)
So, BGA, close at your convenience.

[humdinger]

Since BGA is MIA, I guess I close the ticket as fixed. Since it is for some time now