httpx async http2 errorhandling slow down and retry on error

Question

When server gets overloaded it stats dropping, but my program then drops everything left in jobs. 2-300 lines of the below error is common (same last_stream_id):

20230428143422.761|ERROR|gitlab.py:293|gitlab failed us: https://xxx/api/v4/projects/yyy/pipelines/zzz/test_report - <ConnectionTerminated error_code:0, last_stream_id:1999, additional_data:None>

Program just continue but do not show testreport info. Most of the time it works excellent, (at 80req/sec) but we are 40+ persons using this gitlab-server.

How can I slow down and retry without just dropping?

Make it a bit more error prone for load fluctuations.

Reconnecting the global session socket?

It is a testreport board showing test history so it is pretty pointless without testreport.

Worked fine with requests but taking 7times longer :-)

fastapi route:

tasks = [gitlab.testreport(v['project_id'], v['pipeline_id'], summary=True) for j in jobs]
for ta in asyncio.as_completed(tasks):
  testreports = await asyncio.gather(ta)
  for tr in testreports:
    <update job structure with testreports>

calling gitlab.testreport connector:

s = httpx.AsyncClient(http1=False, http2=True)

async def testreport(id:int, pipid:int, summary:bool=False) -> Optional[Dict]:
  """ testreport on projectid(id)/pipelineid(pipid) if it exits """
  global s
  url = config.get_config()['gitlaburl'] + f"/{id}/pipelines/{pipid}/test_report"
  try:
    r = await s.get(url, headers={'PRIVATE-TOKEN': config.get_config()['gitlabtoken']})
    r.raise_for_status()
    except httpx.HTTPError as exc:
      #raise HTTPException(status_code=500, detail=f"unspecified error: {exc.request.url} - {exc}")
      log.error(f"gitlab failed us: {exc.request.url} - {exc}")
      return None
  <continue if successful>

It seems gitlab has a limit on getting older pipelines, if I limit it to only the last 5 piplelines I've not got it to fail. — MortenB, Apr 28 '23 at 13:59

score 0 · Answer 1 · answered Aug 23 '23 at 06:34

The solution was to limit httpx along with reducing the heavy call to get old testreports:

timeout = httpx.Timeout(10.0, connect=60.0)
limits = httpx.Limits(max_keepalive_connections=5, max_connections=20)
s = httpx.AsyncClient(http1=False, http2=True, timeout=timeout, limits=limits)

httpx async http2 errorhandling slow down and retry on error

1 Answers1