review.haiku-os.org: Slow 'git clone' transfer speeds

[cocobean]

'git clone' procedure seems 5x slower compared to Github:

git clone https://review.haiku-os.org/haiku

NOTE: No packet loss. Procedure completed successfully.

[kallisti5]

This is purely the doing of Gerrit. Performance has always been pretty horrid even though Gerrit gets the most resources on the server.

A few developers *strongly* want Gerrit... so I'm kind of stuck. I've been doing upgrades to Gerrit as time permits to grasp at some "potential performance boost" but have yet to see it.

I have a feeling the slow speeds are because Gerrit is insisting on using "NoteDB" which stores *everything* in git. git performance != database performance

[cocobean]

Gerrit 2.16.13 mentioned JGIT/NoteDB improvement. Worthwhile to bump?

[pulkomandy]

I don't see how notedb would have any impact on checking out the sources, which is like, the only place where you will be accessing git data no matter what, and the notedb thing would not be touched at all. Performance of the web interface is another matter.

[kallisti5]

Everything uses the same Traefik Ingress. Gerrit is the only app that seems to be having troubles. Maybe it's just the nature of the git checkout workload.

Given our history of Gerrit and its painful-to-administer issues though, I tend to distrust it on pretty much all occasions first.

I'm open to tuning suggestions, but not sure there's much we can do with it.

[gerrit]
	basePath = git
	serverId = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
	canonicalWebUrl = https://review.haiku-os.org/
	canonicalGitUrl = git://git.haiku-os.org
[noteDb "changes"]
        autoMigrate = false
        trial = false
        write = true
        read = true
        sequence = true
        primaryStorage = note db
        disableReviewDb = true
[index]
	type = LUCENE
[auth]
	type = OAUTH
	gitBasicAuthPolicy = HTTP
[receive]
	enableSignedPush = false
[sendemail]
	smtpServer = smtp
	smtpServerPort = 25
	enable = true
	html = true
	from = Gerrit <review@review.haiku-os.org>
	connectTimeout = 30
[container]
	user = gerrit2
	javaHome = /usr/lib/jvm/java-1.8-openjdk/jre
	heapLimit = 1536m
	javaOptions = "-Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance"
	javaOptions = "-Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance"
[sshd]
	listenAddress = *:29418
	advertisedAddress = git.haiku-os.org:22
[httpd]
	listenUrl = proxy-https://*:8080
[cache]
	directory = cache
[hooks]
	path = /var/gerrit/review_site/hooks
[oauth]
	allowRegisterNewEmail = true
	allowEditFullName = true
[plugin "gerrit-oauth-provider-github-oauth"]
	client-id = XXXXXX
	client-secret = XXXXXX
	fix-legacy-user-id = false
[plugins]
	allowRemoteAdmin = true
[plugin "events-log"]
	storeUrl = jdbc:h2:/var/gerrit/review_site/db/ChangeEvents
[gitweb]
	type = cgit
	url = http://cgit.haiku-os.org/
[plugin "gerrit-oauth-provider-google-oauth"]
	fix-legacy-user-id = false
[plugin "gerrit-oauth-provider-bitbucket-oauth"]
	fix-legacy-user-id = false
[plugin "gerrit-oauth-provider-cas-oauth"]
	fix-legacy-user-id = false

[cocobean]

@kallisti5 - Downloaded the repo within 13 minutes and using '--depth 10' could download the repo within 4-5 minutes as of today (2020/11/04).

I think the key thing was to get no packet loss and copy repo successfully. No problems seen as of today.

Please close this ticket.

[kallisti5]

thanks! Gerrit does indeed start "slowing down" to the point of not serving any requests within a reasonable timeframe.

Not really sure of the cause, but we're slowly bumping up the version to see if it's been fixed. The workaround in the past has been to just restart Gerrit.

Keep me posted if it starts happening again.