0

I'm using gitlab 11.1.4 , installed with omnibus package, with apache2

Everything is working fine, except git lfs seems to timeout.

On the client side I can see 

bar/foo/lb/bar.cast (99%)
Fatal error: Server error: https://gitlab.example.com/foo/whatever.git/gitlab-lfs/objects/777ed934b03ee88e4d6f25b16e79c1409030ece50d8ff751818f5f064149226c/203686721 from HTTP 502                        
Uploading LFS objects:   0% (0/2), 0 B | 346 KB/s, done
Fatal error: Server error: https://gitlab.example.com/foo/whatever.git/gitlab-lfs/objects/c119f668d98e8f67acefeade50d7d89f50cfe0654bbe6775ad971d229f99612b/233214051 from HTTP 502
error: failed to push some refs to 'ssh://git@gitlab.example.com:8090/foo/whatever'

and in the apache2 log I see 

[Thu Aug 02 10:48:09.195315 2018] [proxy:error] [pid 23394] (104)Connection reset by peer: [client 195.68.4.138:57292] AH01084: pass request body failed to 127.0.0.1:8181 (127.0.0.1)
[Thu Aug 02 10:48:09.195340 2018] [proxy_http:error] [pid 23394] [client 195.68.4.138:57292] AH01097: pass request body failed to 127.0.0.1:8181 (127.0.0.1) from 195.68.4.138 ()

As the file is 40MB large, I suspect a timeout-ish issue. (uploading a smaller file get me a 422 instead of 502, but I think it's because of this bug: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20923)

I tried to change the following in the apache2 conf

Timeout 2400 ProxyTimeout 2400 ProxyBadHeader Ignore

and In the gitlab.rb I put the following 

gitlab_workhorse['proxy_headers_timeout'] = "20m0s"
unicorn['worker_timeout'] = 3600

and gitlabctl restart

but still 502.

Am I missing something ?

allan.simon
  • 3,886
  • 6
  • 35
  • 60

1 Answers1

1

I ran across an issue which appears similar. There is a fixed timeout in lib/gitlab/lfs_token.rb set to 1800 seconds (30 minutes).

I raised a ticket https://gitlab.com/gitlab-org/gitlab-ce/issues/57353 on this.

Gitlab have elected to implement the 'expires_in" attribute in the lfs-authenticate response, so lfs knows to request interim authentication before the original expires. I understand the fix is targeted at v11.9. Check the merge requests with the ticket for details.

In the mean time, I work around the problem by setting the DEFAULT_EXPIRE_TIME to 18000 in lib/gitlab/lfs_token.rb, after each upgrade.

grahamjamesaddis
  • 43
  • 6