Opened 9 years ago

Closed 9 years ago

#6386 closed bug (fixed)

mail_daemon slowdown fetching mail from gmail

Reported by: humdinger Owned by: bga
Priority: normal Milestone: R1
Component: Servers/mail_daemon Version: R1/Development
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description

As mentioned in this thread, mail_daemon recently became very slow when fetching mail from my gmail account. Via binary search, I narrowed it down to between releases hrev37437 (fast) and hrev37453 (slow). Timeline.

Change History (21)

comment:1 by jackburton, 9 years ago

The only related thing seems hrev37443, although the version of regex.c is the same.

comment:2 by stippi, 9 years ago

You would have to do at most four more binary searching builds to obtain the exact revision when it broke. ;-)

comment:3 by humdinger, 9 years ago

I took the lazy road and downloaded nightly builds and there aren't any between those revisions. I've hoped someone would spot the culprit in that relatively narrow range. I'd rather not down- and incrementally upgrade my workcopy and compile it all if I don't have to. Do I have to? :)

comment:4 by bga, 9 years ago

Status: newin-progress

There are 2 contention points:

1 - Parsing the email. This uses a BPositioIO object backed up by a BFile. Any changes related to IO or to memory accesses could trigger this.
2 - Saving the email to disk. This is taking a *LONG* time. Again changes to IO in general could be the culprit.

I will take a look at this ASAP.

comment:5 by humdinger, 9 years ago

While I'd welcome any additional speedup, BGA, it's not a general slowness reported in this ticket.

Through binary search I have now definitely pinpointed the extreme slowdown to hrev37443. Up to that revision my GMail is fetched (pop3, ssl) with normal speed. From hrev37443 on it's slow as molasses.

in reply to:  5 ; comment:6 by korli, 9 years ago

Replying to humdinger:

Through binary search I have now definitely pinpointed the extreme slowdown to hrev37443. Up to that revision my GMail is fetched (pop3, ssl) with normal speed. From hrev37443 on it's slow as molasses.

The libroot regex implementation seems to be older and buggier than the one which existed in mail kit. The switch could very well be the reason for the slowness problem.

in reply to:  6 comment:7 by jackburton, 9 years ago

Replying to korli:

The libroot regex implementation seems to be older and buggier than the one which existed in mail kit. The switch could very well be the reason for the slowness problem.

That's weird, though, since the copyright of the libroot implementation seems to imply it's a newer version.

comment:8 by jackburton, 9 years ago

Anyway, the quick fix would be revert hrev37443. Obviously would be nice if our libroot implementation would be updated instead.

comment:9 by anevilyak, 9 years ago

Actually may not be that simple, at least from what I can see the regex.c referenced by that Jamfile isn't in the tree any more.

in reply to:  9 comment:10 by korli, 9 years ago

Replying to anevilyak:

Actually may not be that simple, at least from what I can see the regex.c referenced by that Jamfile isn't in the tree any more.

A bit tired? It would come back as part of hrev37443 :)

comment:11 by anevilyak, 9 years ago

Ah oops, didn't look carefully enough, the changeset browser made it look as if the only modification was regex.c being removed from the Jamfile. Never mind then :)

comment:12 by korli, 9 years ago

see also hrev23792

comment:13 by korli, 9 years ago

A proposition I implemented locally: update glibc regex to 2.11, headers/posix/regex.h included. This imposes some changes in some of our sources using the regex.h. Mostly bin tools and mail kit. The use of __USE_GNU macro is needed for instance when the expected regex implementation is the GNU one.

Can I commit this and let people validate ? I can imagine there are side effects but I don't see another way to be up to date.

in reply to:  13 comment:14 by jackburton, 9 years ago

Replying to korli:

A proposition I implemented locally: update glibc regex to 2.11, headers/posix/regex.h included. This imposes some changes in some of our sources using the regex.h. Mostly bin tools and mail kit. The use of __USE_GNU macro is needed for instance when the expected regex implementation is the GNU one.

Can I commit this and let people validate ? I can imagine there are side effects but I don't see another way to be up to date.

I'd say go for it.

comment:15 by axeld, 9 years ago

Does it impose a binary compatibility issue? If not, there is no reason not to proceed. But does it actually fix the problem, too? :-)

comment:16 by korli, 9 years ago

Did so in hrev38031. I'm not sure about binary compatibility. We could define __USE_GNU by default in regex.h. It just defines things not required by POSIX.

comment:17 by humdinger, 9 years ago

I'd love to try out the patch, but current revisions refuse to get me online and the patch doesn't apply cleanly to older revisions...

comment:18 by axeld, 9 years ago

Then please file a separate bug report for that - AFAIK, the network is working fine.

comment:19 by humdinger, 9 years ago

Figuring that hrev38078 would help I tried a current revision and lo! at least with static IP using ethernet it looked OK. I could ping my router but "installoptionalpackage" wouldn't download Web+ and just stuck on downloading the zip. After a reboot, the wifi was gone from the network panel, leaving the rtl81xx ethernet. Unfortunately, I now run into #6446...

comment:20 by humdinger, 9 years ago

Since one of stippi's first contracting-commits fixed my sporadic app_server crashes with newer revisions and Axel finally ( :P ) fixed the networking issues (at least for static IPs for now) with my rtl81xx (thanks berregon for bughunting!), I could finally test a current revision (hrev38200). Yay! Thanks all[[BR]] I'm pleased to say that fetching mail from GMail is again as fast as it once was. Not sure which commit did it...

BGA is "in-progress" on it, so I'm not sure we can just pull it from under him...:)
So, BGA, close at your convenience.

comment:21 by humdinger, 9 years ago

Resolution: fixed
Status: in-progressclosed

Since BGA is MIA, I guess I close the ticket as fixed. Since it is for some time now

Note: See TracTickets for help on using tickets.