Skip to content

Python API no longer retries requests which may have temporarily failed (e.g. 502) #925

Closed

Description

@alahiff

When metrics are sent a single POST to /api/metrics with status 502 results in an error visibile to users:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/dispatch/queued.py", line 172, in run
    self._callback(_buffer, queue_label)
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/run.py", line 501, in _dispatch_callback
    return _metrics.commit()
           ^^^^^^^^^^^^^^^^^
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/api/objects/base.py", line 560, in commit
    _response = self._post_single(**self._staging)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/api/objects/metrics.py", line 143, in _post_single
    return super()._post_single(is_json=False, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/api/objects/base.py", line 652, in _post_single
    _json_response = get_json_from_response(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/api/request.py", line 277, in get_json_from_response
    raise RuntimeError(error_str)
RuntimeError: Creation of metric failed for url 'http://localhost:8000/api/metrics' with status 502

Errors such as this should definitely be retried.

Activity

  1. added theissue type on Feb 26, 2026
  2. alahiff commented on Feb 26, 2026

    @alahiff
    MemberAuthor

    http.HTTPStatus.BAD_GATEWAY (502) is missing from RETRY_STATUS_CODES, but adding it has no effect anyway. As far as I can tell Tenacity is doing nothing.

  3. alahiff commented on Feb 26, 2026

    @alahiff
    MemberAuthor

    Note that fixing Tenacity doesn't help (e.g. using retry_if_result), as we then get this error before any of the retries:

    RuntimeError: Cannot append item '{'values': {'resources/cpu.usage.percentage': 1.0, 'resources/cpu.usage.memory': 116.8828125}, 'time': 1.0539274215698242, 'timestamp': '2026-02-26T14:28:53.348861', 'step': 0}' to queue 'metrics_regular', termination called.
    

    Even just adding time.sleep(5) to the beginning of the post function results in an error like this.

  4. changed the title [-]Ptython API no longer retries requests which may have temporarily failed (e.g. 502)[/-] [+]Python API no longer retries requests which may have temporarily failed (e.g. 502)[/+] on Feb 26, 2026
  5. self-assigned this
    on Feb 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions