Skip to content

[BUG] LogMonitor crashes with 0xC0000005 on shutdown after normal child exit #245

Closed

Description

@bobsira

Describe the bug

A freshly built production LogMonitor.exe captures logs successfully, but exits with access violation 0xC0000005 (signed process exit code -1073741819) when its monitored child finishes normally.

This was observed on unmodified main at 42247c6a23f65ae8e3c4f2d483812972cda1ec6f, with Event Log, File, Process, and ETW monitoring enabled together. The final run captured all four sources without LogMonitor error diagnostics before shutdown.

Shutdown was requested by creating a file that the child checks. The child printed demo generator stopped normally and returned from its main function. No Docker termination signal, Ctrl+C, forced process termination, or historical Event Log replay was involved in this run.

Configuration

  • Tool: production Log Monitor executable, file version 2.1.1.0.
  • Source commit: 42247c6a23f65ae8e3c4f2d483812972cda1ec6f.
  • Build: clean native MSBuild solution build, x64 Release, checked-in C++14 defaults.
  • Host: Windows 11 Enterprise N, version 10.0.26300, build 26300, 64-bit; elevated execution.
  • Child: Python 3.12 generating local file writes, stdout/stderr, short-lived cmd.exe children, and localhost DNS lookups.
  • LogMonitor stdout/stderr were redirected to local files.
  • Executable SHA256: 9204D54197D06A5C54D9A5FBAD19A3131726E8CA7DD0E8C2393474C8E8684BF8.

Observed reproduction

  1. Build the production executable from the commit above.

  2. Create an isolated directory without spaces, with input, input\worker, and output subdirectories.

  3. Configure all four source types as shown below. Set the File source's absolute directory to the local input directory.

  4. Launch LogMonitor with the Python child shown below, using an absolute /Config path and redirecting LogMonitor's stdout/stderr to files:

    LogMonitor.exe /Config C:\Temp\LogMonitorShutdownRepro\LogMonitorConfig.json python.exe -u C:\Temp\LogMonitorShutdownRepro\generate_activity.py
    
  5. Allow records from all four sources to arrive, then create stop.request beside the Python script:

    New-Item 'C:\Temp\LogMonitorShutdownRepro\stop.request' -ItemType File -Force
  6. Observe the child's normal-stop message, followed by LogMonitor exiting with 0xC0000005. Windows Application Error event 1000 also records the access violation.

The paths above are normalized examples. This describes the observed combined-source reproduction, not a minimized reproducer; the minimum failing source combination has not yet been isolated.

Final configuration (local path normalized)
{
  "LogConfig": {
    "logFormat": "JSON",
    "sources": [
      {
        "type": "EventLog",
        "startAtOldestRecord": false,
        "eventFormatMultiLine": false,
        "channels": [
          { "name": "Application", "level": "Warning" },
          { "name": "System", "level": "Warning" }
        ]
      },
      {
        "type": "File",
        "directory": "C:\\Temp\\LogMonitorShutdownRepro\\input",
        "filter": "*.log",
        "includeSubdirectories": true,
        "includeFileNames": true,
        "waitInSeconds": 0
      },
      { "type": "Process" },
      {
        "type": "ETW",
        "eventFormatMultiLine": false,
        "providers": [
          {
            "providerName": "Microsoft-Windows-DNS-Client",
            "providerGuid": "1c95126e-7eea-49a9-a3fe-a378b03ddb4d",
            "level": "Verbose",
            "keywords": "0x0"
          }
        ]
      }
    ]
  }
}
Child activity generator used in the final run

Save as generate_activity.py beside the configuration. The input and input\worker directories must already exist, and stop.request must not exist before starting. The script writes one clearly labeled Application WARNING event using the Windows EventCreate source.

import ctypes
from ctypes import wintypes
from datetime import datetime, timezone
import json
import os
from pathlib import Path
import socket
import subprocess
import sys
import time


ROOT = Path(__file__).resolve().parent
STOP = ROOT / "stop.request"
LOGS = (ROOT / "input" / "application.log", ROOT / "input" / "worker" / "worker.log")


def main():
    sys.stdout.reconfigure(encoding="utf-8", line_buffering=True)
    sys.stderr.reconfigure(encoding="utf-8", line_buffering=True)

    kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
    kernel32.OpenProcess.argtypes = (wintypes.DWORD, wintypes.BOOL, wintypes.DWORD)
    kernel32.OpenProcess.restype = wintypes.HANDLE
    kernel32.WaitForSingleObject.argtypes = (wintypes.HANDLE, wintypes.DWORD)
    kernel32.WaitForSingleObject.restype = wintypes.DWORD
    kernel32.CloseHandle.argtypes = (wintypes.HANDLE,)
    kernel32.CloseHandle.restype = wintypes.BOOL
    parent_pid = os.getppid()
    parent = kernel32.OpenProcess(0x00100000, False, parent_pid)
    if not parent:
        raise ctypes.WinError(ctypes.get_last_error())

    def parent_alive():
        result = kernel32.WaitForSingleObject(parent, 0)
        if result == 0xFFFFFFFF:
            raise ctypes.WinError(ctypes.get_last_error())
        return result == 0x102

    def stopping():
        return STOP.exists() or not parent_alive()

    try:
        (ROOT / "generator.json").write_text(
            json.dumps(
                {
                    "pid": os.getpid(),
                    "parent_pid": parent_pid,
                    "started_at": datetime.now(timezone.utc).isoformat(),
                },
                indent=2,
            ),
            encoding="utf-8",
        )
        print(f"demo generator started run={ROOT.name} pid={os.getpid()}")
        time.sleep(3)
        if stopping():
            return

        subprocess.run(
            [
                str(Path(os.environ["SystemRoot"]) / "System32" / "eventcreate.exe"),
                "/L", "APPLICATION",
                "/SO", "EventCreate",
                "/T", "WARNING",
                "/ID", "777",
                "/D", f"LogMonitor local demo startup probe run={ROOT.name}",
            ],
            check=True,
        )

        iteration = 0
        while not stopping():
            iteration += 1
            stamp = datetime.now(timezone.utc).isoformat()
            for index, path in enumerate(LOGS, start=1):
                with path.open("a", encoding="utf-8", newline="\n") as log:
                    log.write(
                        f"{stamp} demo file={index} iteration={iteration} run={ROOT.name}\n"
                    )

            print(f"{stamp} demo stdout iteration={iteration}")
            print(f"{stamp} demo stderr iteration={iteration}", file=sys.stderr)
            addresses = socket.getaddrinfo(
                "localhost", 80, socket.AF_INET, socket.SOCK_STREAM
            )
            print(f"demo localhost lookup iteration={iteration} results={len(addresses)}")
            subprocess.run(
                [
                    str(Path(os.environ["SystemRoot"]) / "System32" / "cmd.exe"),
                    "/d", "/c", f"echo demo child-process iteration={iteration}",
                ],
                check=True,
            )

            for _ in range(10):
                if stopping():
                    break
                time.sleep(0.2)

        if parent_alive():
            print("demo generator stopped normally")
            time.sleep(0.5)
    finally:
        if not kernel32.CloseHandle(parent):
            raise ctypes.WinError(ctypes.get_last_error())


if __name__ == "__main__":
    main()

Actual results

The same access-violation exit code occurred in 4/4 observed combined-source runs on this host. Event Log, File, and Process sources remained enabled while the ETW provider selection changed:

Run ETW providers LogMonitor exit code
1 Kernel-Process + DNS-Client 0xC0000005
2 PowerShell + DNS-Client 0xC0000005
3 User-Diagnostic + DNS-Client 0xC0000005
4 DNS-Client only 0xC0000005

This does not establish that ETW is the cause or that any individual source is sufficient to trigger the crash.

The final run captured:

Source Records
Process 428
ETW 1,831
File 212
EventLog 1
  • No [LOGMONITOR] ERROR: diagnostics appeared during the final capture.
  • The child's normal-stop message was present; no Python traceback was present.
  • The final run used startAtOldestRecord: false, so it did not replay the historical Event Log backlog.
  • The monitor, generator, and capture wrapper had all exited after shutdown; no Log Monitor ETW Session remained active.

Sanitized Windows Application Error event 1000 from the final run:

Time (UTC):                2026-11-11T19:31:40.6117467Z
Faulting application:      LogMonitor.exe
Application version:       2.1.1.0
Faulting module:           LogMonitor.exe
Module version:            2.1.1.0
Exception code:            c0000005
Faulting offset:           0000000000004331
Captured process exit:    -1073741819

The first run also recorded offset 0x4331. A debugger stack/crash dump has not been collected for these reproductions; the faulting offset alone does not establish the root cause.

Expected behavior

After a monitored child finishes normally, LogMonitor should shut down its monitors safely and return the child's exit code rather than terminate with an access violation.

Additional context / related issues

No existing report reviewed clearly describes this current-main normal-child-exit scenario. These references are related reports, not a claim that their root causes match.

The local executable provenance, configurations, generator/capture scripts, exit metadata, and captured output have been retained. Raw host Event Log and DNS payloads are intentionally not included in this public report. No production source changes were made while collecting this evidence.

Activity

  1. bobsira commented on Nov 13, 2026

    @bobsira
    ContributorAuthor

    Resolution: fixed by #250

    Merged into main as 8940d8a.

    The native debugger confirmed a shutdown-lifetime bug: the ETW worker was still using the function-local c_DecodingSourceToString table in EtwMonitor::FormatMetadata after CRT teardown had destroyed it. The main thread was stopping ETW from the global monitor's atexit destructor, which could deliver pending events to that worker.

    The fix explicitly destroys the ETW, file, and Event Log monitors before wmain returns, while their formatter tables are still alive. It preserves the existing monitor destruction order and the child's exit code.

    Validation included production-executable regressions that reproduce 0xC0000005 on signed RC2, followed by 18/18 repeated passing cases across native x64, native x86, and CMake x64. Six four-source runs also passed with child exit codes 0 and 37, retaining output from all four sources without error diagnostics. The regression tests are included in #250.

    The follow-up investigation of #248 confirmed the same lifetime problem in Event Log replay; the EventLog cleanup in this fix addresses that case too.

    Release status: this fix is on main for the v2.2.2 release train. The published v2.2.2-rc.2 executable predates the fix and remains affected; its tag and asset have not been replaced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions