Skip to content

Orchestrator - Retry the Kubernetes API server errors #50

Description

@Ark-kun
kubernetes.client.exceptions.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Audit-Id': '7512bc49-8a1e-4c57-a488-9cc9088b88c2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'cb8e61d0-d4ac-4497-9215-e39742c07fc8', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e0167ddd-e4d7-449d-b305-9dce12e027ee', 'Date': 'Sat, 06 Dec 2025 02:13:12 GMT', 'Content-Length': '154'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"rpc error: code = Internal desc = Internal error encountered.","code":500}

This is non-trivial since we do not want to stall the queue.

Other common errors are Kubernetes API Server connection errors.

Anther error (only happened once):

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/http/client.py", line 603, in _read_chunked
    value.append(self._safe_read(chunk_left))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 644, in _safe_read
    raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(8493 bytes read, 7057 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/urllib3/response.py", line 903, in _error_catcher
    yield
  File "/app/.venv/lib/python3.12/site-packages/urllib3/response.py", line 1028, in _raw_read
    data = self._fp_read(amt, read1=read1) if not fp_closed else b""
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/response.py", line 1011, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
                                                      ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 473, in read
    return self._read_chunked(amt)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 609, in _read_chunked
    raise IncompleteRead(b''.join(value)) from exc
http.client.IncompleteRead: IncompleteRead(0 bytes read)

The above exception was the direct cause of the following exception:

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions