fix: decouple result writeback from task dispatch to stop cross-region edge deadlock

[710leo]

Problem

On a cross-region deployment (edge in Tokyo, center in Shanghai), a categraf agent stopped receiving any tasks at all. Its log showed one line every 6 seconds, for hours, with no self-recovery:

E! AI ibex rpc call Server.Report fail: timeout

The RPC service itself was healthy. A minimal msgpack-rpc probe against the same port showed:

Server.Ping                        ->  0.063s
Server.Report, ReportTasks = []    ->  0.000s   (returns AssignTasks normally)
Server.Report, ReportTasks = 1356  ->  never returns within the client's 5s budget

That isolates the blocking to handleDoneTask.

Root cause

Report does two unrelated things in one synchronous call: it writes back execution results, and it hands out pending tasks. The writeback is per-result, and on an edge node each result costs one HTTP round trip to center — about 200ms across regions.

Once an agent has accumulated a batch of results (1356 in this incident), a single Report needs ~141s while the agent's RPC timeout is 5s. That closes a loop:

agent times out and reconnects
  -> Locals.Clean never runs
  -> local results are never dropped
  -> next round resends the identical batch
  -> server starts over
  -> times out again

The agent's own cleanup logic is locked out by the very thing it is supposed to clean up. Meanwhile AssignTasks is never delivered, so the host stops receiving tasks permanently. In this incident four legitimate tasks queued after the deadlock formed were never picked up.

The ceiling here is low for a component whose whole purpose is cross-region deployment: 5s / 200ms is roughly 25 results before the channel wedges for good.

Changes

src/server/rpc/method.go

• Move the writeback to a background goroutine so Report returns AssignTasks immediately. A sync.Map keyed by ident keeps at most one writeback in flight per agent, so a resending agent cannot pile up goroutines — during the incident, edge-to-center connections grew from 126 to 160 for exactly this reason. A skipped batch is logged at warning level, because the agent drops its copy after Clean and the results are not recoverable.
• handleDoneTask now logs and continues on a per-result failure instead of returning. Previously a single unwritable result prevented every result after it in the same batch from ever being persisted.

src/server/timer/timer_host_doing.go

Return early when either fetch fails, keeping the previous cache. The error was previously logged but not returned, and the partial result was then written over the cache. When center was unreachable this made agents silently stop receiving tasks, with models.TableRecordGets fail as the only trace.

src/storage/redis.go

IdInit uses SetNX. This id is the fallback allocator used when an edge cannot reach center, so it must not repeat across restarts. An unconditional Set restarts allocation from IDINITIAL on every boot and hands out ids that collide with historical tasks. The affected edge's id key was sitting at exactly 2^32.

Trade-off worth reviewing

The sync.Map guard trades "never pile up goroutines" against "a batch may be skipped while an earlier one is still draining". Skipped batches are logged, and the window only opens while a backlog is being flushed — normal writeback is sub-millisecond. If losing results is the greater concern, the guard can be dropped in favour of a plain go handleDoneTask(req): once the agent gets a successful response it clears Locals.M, so the next round carries nothing and pile-up is self-limiting. The conservative side was chosen here because this incident was driven by an agent resending the same 200KB batch every second.

Verification

• gofmt -l src/ clean
• go build and go vet pass on the three changed packages
• Full go build ./... of n9e-plus passes with a replace pointing at this branch

Note that src/server/router fails to compile standalone on this repo; that is pre-existing and reproduces on an unmodified tree. It only builds inside n9e-plus, which carries the nightingale replace. The repository has no test files.

Release note

Next tag should be v1.3.12. n9e-plus currently pins v1.3.9, so bumping straight to v1.3.12 also pulls in v1.3.10 (DaMeng TaskHost.Upsert) and v1.3.11 (task table ids no longer treated as auto-increment) — worth a regression pass, since the latter touches the same area as the edge fallback ids described above.