Skip to content

[shell-operator] fix: wait for kube events to finish processing - #939

Merged
ldmonster merged 2 commits into
mainfrom
fix/wait-for-kube-events
Oct 21, 2026
Merged

ldmonster merged 2 commits into
mainfrom
fix/wait-for-kube-events

Conversation

@fuldaxxx

@fuldaxxx fuldaxxx commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Description

  • BindingContextController.ChangeState (what deckhouse's KubeStateSet calls) no longer stops collecting events on a fixed 100ms timer (STOP_EVENTS). Every 10ms it checks whether informers have handled every change in the fake cluster. It returns once three conditions hold:

    • each informer's cache matches a fresh List of the cluster by checksum;
    • no watch event is being handled;
    • the event channel is empty.

    The 20s timeout stays as a backstop.

  • Added Synced(ctx) bool to KubeEventsManager and Monitor. It lists objects on every call and is meant for the fake cluster in tests; the operator never calls it at runtime. The only change on the production event path is an atomic in-flight counter in handleWatchEvent.

  • NewStateController no longer takes the unused KubeEventsManager argument. deckhouse does not call it.

  • Two regression tests:

    • Test_ChangeState_SlowEvents: a filter that sleeps 50ms per event, standing in for a loaded runner. It fails on the old harness.
    • Test_ChangeState_FilterError: ChangeState must not wait for objects the filter fails on. It hangs until the 20s timeout if the check does not mirror handleWatchEvent.

Why do we need it, and what problem does it solve?

Symptom. deckhouse hook unit tests flake on CI in hooks and tests that nobody changed. For example:

  • user-authn/hooks/get_dex_client_crds_test.go: the credentials Secret is missing from the snapshot, so the hook generates a random password instead of "test". After a deletion, BindingContexts comes back empty.
  • 000-common/hooks/internal_tls_test.go: the existing TLS Secret is missing from the snapshot, so a new certificate is generated. This one probably has the same cause but was not checked separately.

Root cause. ChangeState writes objects to the fake cluster. Events reach the harness asynchronously: fake watch → reflector → DeltaFIFO → listener goroutine → handleWatchEvent (filter, checksum, cache) → event channel. The harness sent STOP_EVENTS 100ms after the writes and dropped anything that arrived later. Late events stayed in the channel and leaked into the next ChangeState. With go test -race on a loaded runner, 100ms is not always enough.

Reproduced: with a filter that takes about 50ms per event, the old ChangeState returns 0 of 3 binding contexts on every run. CPU load alone on a 12-core laptop did not reproduce it (30/30 runs passed).

@fuldaxxx fuldaxxx self-assigned this Sep 29, 2026
@fuldaxxx fuldaxxx added the bug Something isn't working label Sep 29, 2026
@fuldaxxx fuldaxxx changed the title fix: wait for kube events to finish processing [shell-operator] fix: wait for kube events to finish processing Sep 29, 2026
Keep synchronization helpers out of public interfaces while retaining
fake-cluster test support

Signed-off-by: Ruslan Gorbunov <[email protected]>
@fuldaxxx
fuldaxxx marked this pull request as ready for review September 30, 2026 08:02
@ldmonster
ldmonster merged commit 59f659e into main Oct 21, 2026
9 checks passed
@ldmonster
ldmonster deleted the fix/wait-for-kube-events branch October 21, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants