Skip to content

FLOGO-19401: Fix data race on the shared logger's tracing context - #299

Open
awakchau-tibco wants to merge 2 commits into
masterfrom
FLOGO-19401-logger-tracing-context-race
Open

FLOGO-19401: Fix data race on the shared logger's tracing context#299
awakchau-tibco wants to merge 2 commits into
masterfrom
FLOGO-19401-logger-tracing-context-race

Conversation

@awakchau-tibco

Copy link
Copy Markdown
Collaborator

What kind of change does this PR introduce? (check one with "x")

[x] Bugfix
[] Feature
[] Code style update (formatting, local variables)
[] Refactoring (no functional changes, no api changes)
[] Other... Please describe:

Fixes: FLOGO-19401

What is the current behavior?

Apps crash with an unrecovered SIGSEGV under concurrent load, and the container is restarted. Reported in the field as pods restarting under high traffic; three separate pods produced the same panic:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xe58685]

github.com/project-flogo/core/support/log.(*zapLoggerImpl).Debugf(...)
	core/support/log/zap.go:84 +0x65
github.com/project-flogo/flow.(*FlowAction).Run.func1()
	flow/action.go:412 +0x39e

One *zapLoggerImpl is shared by every concurrently executing flow instance and activity — flow/action.go does instLogger := logger and only replaces it with a per-instance child when FLOGO_LOG_CTX=true, which is off by default. SetTracingContext therefore mutates that shared object while every log call on it reads the same fields, with no synchronisation:

Role Site
writer flow/action.go:355 — sets the prefix on flow start
writer flow/action.go:392, :397 — clears it on flow completion
writer flow/instance/taskinst.go:349 — sets it per activity
reader support/log/zap.go:84l.mainLogger.Debugf(l.tracePrefix+template, args...)
reader support/log/zap.go:92l.mainLogger.Infof(l.tracePrefix+template, args...)

tracePrefix was a plain string field. A Go string is a two-word {data *byte, len int} value and assigning one is not atomic, so a racing reader could observe the torn combination {data: nil, len: N}:

  • zap.go:83 — the l.tracePrefix != "" guard passes, because len is non-zero
  • zap.go:84l.tracePrefix + template then copies N bytes from address 0

Running the reproduction with GOTRACEBACK=system unhides the runtime frames and shows exactly that:

runtime.memmove()                                 <- faults; pc equals the signal pc
    runtime/memmove_amd64.s:215
runtime.concatstrings(...)
    runtime/string.go:56
runtime.concatstring2(_, {0x0?, 0x10?}, {...})    <- first string: data 0x0, len 16
    runtime/string.go:66
(*zapLoggerImpl).Debugf(...)
    support/log/zap.go:84 +0x65

Two things make this easy to hit:

  1. The concatenation is an argument to Debugf, so it is evaluated on every call regardless of the configured log level. Apps running at INFO or ERROR still crash inside Debugf.
  2. DefaultLogTracingContextEnabled = true, so no opt-in is required to reach the code.

traceContext map[string]string has the same problem — it is assigned unsynchronised on the line above.

What is the new behavior?

The prefix and the trace context are published together as one immutable *traceState held in an atomic.Value, so a reader always observes a complete, self-consistent state. Every log method loads it once into a local.

  • Contained entirely in support/log/zap.go. No API change, no behaviour change.
  • Uses atomic.Value rather than atomic.Pointer[T] so it stays within the module's existing go 1.18 directive.
  • Read-path cost is unchanged in practice — 209 M reads in 20 s patched vs 239 M unpatched in the stress harness.

Testing

Adds support/log/zap_test.go (the package previously had no tests):

  • TestSetTracingContext — pins prefix set/clear/round-trip behaviour, including a context with no trace id
  • TestSetTracingContextDisabled — covers FLOGO_LOG_TRACE_CTX_ENABLED=false
  • TestSetTracingContextConcurrentWithLogging — the regression test: 2 writers against 8 readers exercising Debugf/Infof/Debug/Info. Reports a data race under -race before this change, clean after

Verified on Ubuntu 24.04, Go 1.25.11, 16 CPUs:

Scenario Before After
stress harness, no race detector SIGSEGV addr=0x0 within approx 1 s survives 20 s, 209 M log reads
stress harness, -race WARNING: DATA RACE, exit 66 clean

gofmt and go vet are clean. go test ./... passes on 30 packages; the one failure, TestURLStringToFilePath in support, is a pre-existing Windows-only path-separator assertion that reproduces identically on unmodified master.

Note for reviewers

There is a related correctness issue this PR deliberately does not address: because the logger is shared, one flow's trace id can be stamped onto another flow's log lines. That is misleading observability data rather than a crash, and fixing it means giving each flow instance its own logger — a larger change in flow. Raising separately.

One *zapLoggerImpl is shared by every concurrently executing flow instance
and activity - flow/action.go does `instLogger := logger` and only replaces
it with a per-instance child when FLOGO_LOG_CTX=true, which is off by
default. SetTracingContext therefore mutates that shared object while every
log call on it reads the same fields, with no synchronisation.

tracePrefix was a plain string field. A Go string is a two-word
{data, len} value and assigning one is not atomic, so a racing reader could
observe the torn combination {data: nil, len: N}: the `!= ""` guard passed
because len was non-zero, and the following prefix concatenation then copied
N bytes from address 0 and segfaulted, taking the whole app down.

Because the concatenation is an argument to Debugf/Infof it is evaluated on
every call regardless of the configured log level, so apps running at INFO or
ERROR were still crashing inside Debugf. Reported in the field as pods
restarting under high traffic, with SIGSEGV addr=0x0 at zap.go:84 and :92.

Publish the prefix and the trace context together as one immutable value held
in an atomic.Value, so a reader always observes a complete state. Uses
atomic.Value rather than atomic.Pointer[T] to stay within the module's
existing go 1.18 directive. No API or behaviour change.

Verified with the race detector: the added test reports a data race and the
standalone reproduction segfaults within about a second before this change,
and both are clean after it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@awakchau-tibco

Copy link
Copy Markdown
Collaborator Author

Regression test verified in both directions

The claim in the PR body that the new test catches this has now been checked both ways, rather than only green.

Green — patched, with the race detector (Ubuntu 24.04, Go 1.25.11, 16 CPUs):

$ go test -race -run TestSetTracingContextConcurrentWithLogging ./support/log/
ok      github.com/project-flogo/core/support/log       1.125s

Red — unpatched. Reverting only zap.go does not work as a check: TestSetTracingContext and TestSetTracingContextDisabled call logger.tracePrefix(), which is a method only in the patched version — unpatched it is a struct field, so the package fails to compile and you get a build error rather than a race report.

Running just the concurrency test against an unmodified [email protected] tree instead (that file is byte-identical to master), it fails 3 out of 3 runs without the race detector at all:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0]

goroutine 12 [running]:
github.com/project-flogo/core/support/log.(*zapLoggerImpl).Debugf(...)
	support/log/zap.go:84 +0x65

Same signature as the field reports: addr=0x0 at zap.go:84.

So the test is a genuine regression test — it goes red on the defect and green on the fix, and it is fast (0.02 s) and quiet (level is set to ERROR, and the prefix is still built on every call because it is an argument).

Note for anyone reproducing this: git worktree does not work across the Windows/WSL boundary — the worktree's .git file holds a C:/... path that git on Linux cannot resolve, so git checkout master -- support/log/zap.go fails with fatal: not a git repository. Copy the file rather than using git, or work in a throwaway copy of the module.

Comment-only change, no functional difference.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant