New scheduler: substantial performance drop

[stippi]

I've already made Pawel aware of it, but wanted it logged, that the new scheduler comes with a substantial drop in performance for certain work loads. I've timed the Haiku revision before the merge building HaikuDepot (after jam clean). These are my timings:

jam -q -j4 HaikuDepot

Haiku old scheduler:
real    0m58.340s
user    2m25.305s
sys     0m52.408s

real    0m57.608s
user    2m25.175s
sys     0m52.455s


Haiku, new scheduler:
real    1m40.426s
user    2m23.923s
sys     0m42.496s

real    1m40.283s
user    2m23.842s
sys     0m42.488s


Linux 12.04:
real	0m24.603s
user	1m5.592s
sys	0m5.376s

real	0m24.620s
user	1m5.720s
sys	0m5.268s

The system time actually improved, but the overall time for the concurrent jobs is much worse than before. Looing at ActivityMonitor, the thread migration between cores seems to be the problem, because each core peaks only for short times, while with the old scheduler, the cores stay peaked.