diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
new file mode 100644
index 0000000..63bd30e
--- /dev/null
+++ b/.github/workflows/ci.yml
@@ -0,0 +1,22 @@
+name: ci
+on:
+ push:
+ branches: [main]
+ pull_request:
+jobs:
+ test:
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ python: ["3.10", "3.11", "3.12"]
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v5
+ with:
+ python-version: ${{ matrix.python }}
+ - run: pip install -e ".[dev]"
+ - run: ruff check src tests && ruff format --check src tests
+ - run: mypy
+ - run: pytest -q
+ - run: make bench-build # compile-only; benchmarks are not executed in CI
+ - run: kptk demo numa_misplaced > /dev/null && kptk inspect
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..ca5a759
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,12 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.venv/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+/recording*.jsonl*
+/report*.md
+/report*.json
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000..b84ef20
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,38 @@
+# Changelog
+
+## 0.2.0: analysis toolkit
+
+Rebuilt from the 0.1 text-to-JSON scaffold into a packaged toolkit (`kptk`).
+
+### Added
+- **Recorder** that captures raw /proc and /sys text into portable JSONL recordings, with
+ deadline scheduling, per-sample cost tracking, and rate-limited page-table-walking reads.
+- **perf_event_open via ctypes**: hardware/software/cache events, multiplexing correction,
+ and the fork → attach → exec (`enable_on_exec`) handshake for `kptk profile`.
+- **Analyzers** for CPU scheduling (schedstat run-queue delay, PSI, voluntary/involuntary
+ switches, steal, imbalance), memory (faults, direct reclaim, swap, PSI), NUMA (allocation vs
+ access locality, numa_maps placement vs thread residency, SLIT distance, node exhaustion),
+ cache/TLB (IPC, MPKI, multiplexing), huge pages (THP mode, fallback, compaction, coverage,
+ idle hugetlb), and a config audit.
+- Findings engine with severities, evidence and recommendations; USE summary; sparklines;
+ data-quality report; tunable thresholds (JSON); CI gate (`--fail-on`).
+- `kptk compare`: multi-run A/B with a permutation test.
+- Topology reconstruction (SMT, sockets, NUMA nodes, cache hierarchy).
+- Synthetic two-socket recordings (`kptk demo`) and a test suite asserting exact findings per scenario.
+- C microbenchmarks: pointer-chase latency (4K vs THP, NUMA), page-fault cost, wakeup latency, false sharing.
+- Docs: architecture, metric reference, kernel mechanics, workflows, design decisions, interview guide. CI workflow.
+
+### Changed
+- NUMA "remote" no longer uses `numa_miss` (a policy counter); locality uses
+ `local_node`/`other_node` and hinting faults.
+- "Scheduler latency" now measures run-queue delay rather than context-switch counts.
+- `perf stat` parsing keeps the multiplexing percentage. The v0.1 workflow is now `kptk parse`.
+- Code is an installable package (`src/kptk`) instead of flat scripts with `sys.path` edits;
+ `parser.py` no longer shadows a standard-library module name.
+- Configs are JSON (stdlib-only). LICENSE was empty and is now Apache-2.0.
+
+### Removed
+- Empty placeholder images and docs; trivial C programs replaced by the microbenchmarks above.
+
+## 0.1.0
+- Initial scaffold.
diff --git a/LICENSE b/LICENSE
index e69de29..261eeb9 100644
--- a/LICENSE
+++ b/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/Makefile b/Makefile
index f899556..cf6f116 100644
--- a/Makefile
+++ b/Makefile
@@ -1,23 +1,51 @@
-.PHONY: test sample-report benchmarks clean
-
+PYTHON ?= python3
CC ?= cc
-CFLAGS ?= -O2 -Wall -Wextra -std=c11
+CFLAGS ?= -O2 -Wall -Wextra -std=c11 -pthread
BUILD := build
-BENCHMARKS := cache_benchmark cpu_stress memory_stress numa_benchmark page_fault_generator
+BENCH := mem_latency page_fault_cost wakeup_latency false_sharing
+export PYTHONPATH := src
+
+.PHONY: install dev test lint typecheck fmt check demo samples bench-build clean
+
+install:
+ $(PYTHON) -m pip install -e .
+
+dev:
+ $(PYTHON) -m pip install -e ".[dev]"
test:
- python3 -m pytest -q
+ $(PYTHON) -m pytest -q
+
+lint:
+ ruff check src tests
+ ruff format --check src tests
+
+typecheck:
+ mypy
+
+fmt:
+ ruff check --fix src tests
+ ruff format src tests
+
+check: lint typecheck test
+
+demo:
+ $(PYTHON) -m kptk demo numa_misplaced
-sample-report:
- python3 src/perf_runner.py --perf-file samples/perf_output.txt --vmstat-file samples/vmstat.txt --numastat-file samples/numastat.txt --output samples/report.json
+samples:
+ $(PYTHON) -m kptk demo numa_misplaced --save samples/numa_misplaced.jsonl.gz --md samples/numa_misplaced_report.md --json samples/numa_misplaced_report.json
+ $(PYTHON) -m kptk parse --perf-file samples/perf_output.txt --vmstat-file samples/vmstat.txt --numastat-file samples/numastat.txt -o samples/parse_report.json
-benchmarks: $(BENCHMARKS:%=$(BUILD)/%)
+# Microbenchmarks are compiled here but never run by `make check`: they are
+# deliberately CPU/memory heavy. Run them by hand on the machine under study.
+bench-build: $(BENCH:%=$(BUILD)/%)
$(BUILD):
mkdir -p $@
-$(BUILD)/%: benchmarks/%.c | $(BUILD)
+$(BUILD)/%: benchmarks/c/%.c | $(BUILD)
$(CC) $(CFLAGS) $< -o $@
clean:
- rm -rf $(BUILD)
+ rm -rf $(BUILD) .pytest_cache .mypy_cache .ruff_cache src/*.egg-info
+ find . -name __pycache__ -type d -prune -exec rm -rf {} +
diff --git a/README.md b/README.md
index 7b9ded6..9d584c5 100644
--- a/README.md
+++ b/README.md
@@ -1,76 +1,132 @@
-# Kernel Performance Toolkit
+# Kernel Performance Toolkit (`kptk`)
-A small, dependency-free toolkit for turning Linux `perf stat`, `vmstat`, and
-`numastat` output into a single JSON report. It works in two modes:
+**A Linux kernel performance analysis toolkit for CPU scheduling, memory behaviour, NUMA
+locality, cache efficiency, page faults and huge pages.** It explains *why* a workload is slow,
+with evidence from kernel counters and a concrete fix.
-- **Offline analysis** parses saved command output. This is portable and needs
- no special permissions.
-- **Live collection** runs a command under `perf stat` and captures a `vmstat`
- and `numastat` snapshot when those tools are available.
+- **Zero dependencies**: Python standard library only; safe to drop onto a production host.
+- **Low overhead**: records raw `/proc` and `/sys` text (~1.4 ms per sample measured on a laptop) and analyses later, anywhere.
+- **Hardware counters without `perf`**: calls `perf_event_open(2)` directly via ctypes, with multiplexing correction.
+- **Correct kernel semantics**: run-queue delay instead of "CPU %", allocation vs access locality for NUMA, THP fallback and compaction instead of just "THP on".
-## Quick start
+```
+$ kptk demo numa_misplaced
-Generate a report from the bundled fixtures:
+### 🔴 About 85% of the process's memory accesses are likely remote: threads ran mostly on node 0, memory sits mostly on node 1
+Evidence: `cpu_node_residency={0: 1}`, `memory_by_node={0: 0.15, 1: 0.85}`, `placement_mismatch=0.85`, `expected_slit_distance=19.4`
+Action: Start it with numactl --cpunodebind=1 --membind=1, or migratepages 4242 1 0.
-```bash
-make sample-report
-python3 -m json.tool samples/report.json
+### 🟠 Likely memory-bound: IPC 0.60, LLC 17.9 misses per 1k instructions; anonymous working set 49152 MiB vs 32 MiB L3
+### 🟠 dTLB miss rate 3.0% (10.00 per 1k instructions): page walks are significant
+### 🟠 THP enabled=always with defrag=always: page faults may stall on synchronous compaction
+### 🟠 44% of huge-page faults fell back to 4 KiB pages
+...
```
+Synthetic two-socket scenario. Full report: [`samples/numa_misplaced_report.md`](samples/numa_misplaced_report.md);
+a real laptop recording is in [`samples/laptop_profile_report.md`](samples/laptop_profile_report.md).
-Run the test suite:
+## What it measures
-```bash
-python3 -m pytest -q
-```
+| Domain | Key signals | Example findings |
+|---|---|---|
+| **CPU scheduling** | `/proc/schedstat` run-queue delay, per-task wait share, PSI cpu, voluntary vs involuntary switches, steal, per-CPU imbalance | CPU saturation; process waits 45% of runnable time in run queues; preemption storm |
+| **Memory and faults** | minor/major fault rates, direct reclaim (`allocstall`), swap, PSI memory, MemAvailable | Direct reclaim stalls; major faults; minor-fault storm from allocator churn |
+| **NUMA locality** | `local_node`/`other_node`, AutoNUMA hinting-fault locality, `numa_maps` placement vs thread residency, SLIT distances, per-node free memory | Process misplaced across nodes; node exhausted, so allocations spill remote |
+| **Cache efficiency** | IPC, LLC MPKI, branch and L1D miss rates, multiplexing coverage | Likely memory-bound (working set vs LLC); bad speculation |
+| **TLB and huge pages** | dTLB MPKI, THP mode and defrag, fallback %, compaction stalls, heap THP coverage, idle hugetlb pools | Sync compaction risk; THP fallback (fragmentation); low coverage with TLB pressure |
+| **Configuration** | `zone_reclaim_mode`, governors, `numa_balancing`, `perf_event_paranoid` | Risky sysctls, profiling blocked |
-Build the small workload fixtures (a C compiler is required):
+Every report also has a **USE table** (utilisation / saturation / errors per resource),
+**sparklines** over time, and a **data-quality** section listing anything the kernel didn't expose.
+
+## Quick start
```bash
-make benchmarks
+pip install -e . # or just: PYTHONPATH=src python3 -m kptk ...
+
+kptk inspect # topology, caches, THP/hugetlb, sysctl audit
+kptk demo --list && kptk demo cpu_saturated # synthetic two-socket scenarios
+kptk profile -- ./my_workload --args # counters + recording for one command
+kptk record -p -d 60 -o svc.jsonl.gz # watch a running process
+kptk analyze svc.jsonl.gz --md report.md --json report.json --fail-on warning
+kptk compare --before a*.json --after b*.json # permutation-tested A/B
```
-## Usage
+More workflows, including the CI gate and a full case study: [docs/examples.md](docs/examples.md).
-Parse existing outputs:
+## How it works
-```bash
-python3 src/perf_runner.py \
- --perf-file samples/perf_output.txt \
- --vmstat-file samples/vmstat.txt \
- --numastat-file samples/numastat.txt \
- --output report.json --markdown report.md
+```mermaid
+flowchart LR
+ P["/proc + /sys"] -->|verbatim text| R[Recorder] --> J[("recording.jsonl.gz")]
+ K["perf_event_open
(ctypes)"] -->|"fork → attach → exec"| J
+ J --> E["pure analysis
cpu · memory · numa · cache · hugepages · config"]
+ E --> F["findings + USE + time series"] --> O["Markdown / JSON / CI exit code"]
+ F --> C["compare (permutation test)"]
```
-Collect metrics while a program runs:
+* **Capture, then analyse** ([ADR-1](docs/design-decisions.md)): the recorder never parses, so
+ recordings are portable, re-analysable and cheap to take.
+* **Exact counting from the first instruction** (ADR-4): fork, child blocks before exec,
+ parent attaches counters with `enable_on_exec`, then releases it (the `perf stat` handshake).
+* **Expensive files are rate-limited** (ADR-8): `numa_maps`/`smaps_rollup` walk the target's
+ page tables, so they're read every N samples.
-```bash
-python3 src/perf_runner.py --command ./my-workload --size 1024
+Details: [architecture](docs/architecture.md) · [metric reference](docs/metrics.md) ·
+[kernel mechanics](docs/linux-performance.md) · [design decisions](docs/design-decisions.md)
+
+## Microbenchmarks
+
+`benchmarks/c/` reproduces each phenomenon on real hardware. They're built by `make bench-build`
+but never run in CI, because they are deliberately heavy:
+
+| Benchmark | Shows |
+|---|---|
+| `mem_latency [--huge]` | Pointer-chase latency vs working set: L1/L2/L3/DRAM cliffs, and the TLB-reach shift with huge pages; under `numactl` the remote-node penalty |
+| `page_fault_cost [MiB]` | First-touch cost: base pages vs `MAP_POPULATE` vs THP vs pre-faulted |
+| `wakeup_latency [n cpuA cpuB]` | Block/wake latency percentiles; same core vs same socket vs cross-socket |
+| `false_sharing [threads]` | Cache-line ping-pong: packed vs padded per-thread counters |
+
+## Project layout
+
+```
+src/kptk/
+ parsers/ proc.py · sysfs.py · tools.py (perf stat, vmstat, numastat text)
+ analysis/ cpu · memory · numa · cache · hugepages · engine · findings · window
+ record.py Source abstraction, deadline-scheduled Recorder, JSONL recordings
+ perf_event.py perf_event_open via ctypes (attr struct, encodings, multiplex scaling)
+ profile.py fork → attach → exec profiling
+ topology.py CPUs, SMT, sockets, NUMA nodes/distances, cache hierarchy
+ compare.py multi-run A/B with permutation test
+ synth.py synthetic two-socket recordings (tests, demo)
+ report.py · cli.py
+benchmarks/c/ mem_latency · page_fault_cost · wakeup_latency · false_sharing
+samples/ real laptop recording, synthetic scenario, v0.1 text inputs, generated reports
+docs/ architecture · metrics · linux-performance · examples · design-decisions · interview-guide
```
-For live collection, install the Linux `perf` tools and `numastat` (usually
-provided by the `numactl` package). Some systems restrict performance counters;
-adjust `kernel.perf_event_paranoid` or use an administrator-approved setup when
-appropriate. No `sudo` is used by the toolkit itself.
+## Requirements and permissions
-## Report contents
+Linux, Python ≥ 3.10. Everything except hardware counters works unprivileged. Counters need
+`kernel.perf_event_paranoid ≤ 2` or `CAP_PERFMON`; without them the report says so and
+continues. Run-queue latency needs `CONFIG_SCHEDSTATS`; PSI needs `CONFIG_PSI`.
+`scripts/setup_hugepages.sh` reserves hugetlb pages and requires root.
-The JSON report preserves the raw perf, vmstat, and NUMA inputs and derives:
+## Development
-- cache references, misses, miss rate, and instructions per miss;
-- minor/major fault totals and major-fault percentage;
-- context switches, CPU migrations, and switches per task-clock millisecond;
-- local versus remote NUMA allocation percentage.
+```bash
+make check # ruff + mypy + pytest (43 tests, < 1 s; no CPU-heavy tests)
+make bench-build # compile microbenchmarks (does not run them)
+make samples # regenerate the sample reports
+```
-Missing or unsupported counters are represented as zero-derived metrics, so a
-report can still be compared across hosts with different capabilities.
+## Limitations and roadmap
-## Layout
+- Counting, not sampling: no stacks or code attribution. Pair it with `perf record` or eBPF.
+- Generic PMU events give a coarse "top-down lite"; next is Intel topdown slots / AMD equivalents.
+- Process view is pid-level. Next: per-thread residency from `/proc//task/*`.
+- Next: eBPF run-queue latency histograms, and cgroup v2 (`cpu.stat` throttling, per-cgroup PSI).
-- `src/perf_runner.py` — command-line entry point.
-- `src/parser.py` — parsers for human-readable and CSV perf output.
-- `src/report.py` — report construction plus JSON/Markdown output.
-- `samples/` — known-good offline inputs and a generated example report.
-- `scripts/` — convenience collection and huge-page setup scripts.
+## License
-`scripts/setup_hugepages.sh` changes a kernel setting and intentionally requires
-root. Run it only when huge pages are part of your workload configuration.
+Apache-2.0. See [LICENSE](LICENSE).
diff --git a/benchmarks/c/false_sharing.c b/benchmarks/c/false_sharing.c
new file mode 100644
index 0000000..9fe5d7d
--- /dev/null
+++ b/benchmarks/c/false_sharing.c
@@ -0,0 +1,54 @@
+/*
+ * false_sharing: N threads increment their *own* counters, packed vs padded.
+ *
+ * Packed counters share 64-byte cache lines, so every increment invalidates the
+ * line in the other cores' caches (MESI ping-pong), even though no data is
+ * logically shared. Padding each counter to its own line removes the coherence
+ * traffic. Expect a large slowdown for "packed" with >= 2 threads on different cores.
+ *
+ * ./false_sharing [threads=4] [iterations=100000000]
+ */
+#define _GNU_SOURCE
+#include
+#include
+#include
+#include
+#include
+
+#define MAX_THREADS 64
+struct padded { _Alignas(64) volatile uint64_t v; char pad[64 - sizeof(uint64_t)]; };
+
+static volatile uint64_t packed[MAX_THREADS];
+static struct padded padded[MAX_THREADS];
+static long iters = 100000000;
+
+static void *bump_packed(void *arg) {
+ long id = (long)arg;
+ for (long i = 0; i < iters; i++) packed[id]++;
+ return NULL;
+}
+
+static void *bump_padded(void *arg) {
+ long id = (long)arg;
+ for (long i = 0; i < iters; i++) padded[id].v++;
+ return NULL;
+}
+
+static double run(void *(*fn)(void *), int n) {
+ pthread_t t[MAX_THREADS];
+ struct timespec t0, t1;
+ clock_gettime(CLOCK_MONOTONIC, &t0);
+ for (long i = 0; i < n; i++) pthread_create(&t[i], NULL, fn, (void *)i);
+ for (int i = 0; i < n; i++) pthread_join(t[i], NULL);
+ clock_gettime(CLOCK_MONOTONIC, &t1);
+ return (t1.tv_sec - t0.tv_sec) * 1e3 + (t1.tv_nsec - t0.tv_nsec) / 1e6;
+}
+
+int main(int argc, char **argv) {
+ int n = argc > 1 ? atoi(argv[1]) : 4;
+ if (n > MAX_THREADS) n = MAX_THREADS;
+ if (argc > 2) iters = atol(argv[2]);
+ double a = run(bump_packed, n), b = run(bump_padded, n);
+ printf("threads=%d packed=%.1fms padded=%.1fms slowdown=%.1fx\n", n, a, b, a / b);
+ return 0;
+}
diff --git a/benchmarks/c/mem_latency.c b/benchmarks/c/mem_latency.c
new file mode 100644
index 0000000..333d457
--- /dev/null
+++ b/benchmarks/c/mem_latency.c
@@ -0,0 +1,69 @@
+/*
+ * mem_latency: load-to-use latency vs working-set size, by pointer chasing.
+ *
+ * Each element stores the index of the next one, in a random cyclic permutation
+ * (Sattolo's algorithm), so every load depends on the previous one: no
+ * memory-level parallelism, and the hardware prefetcher can't guess the next
+ * address. ns/load vs size shows the L1 -> L2 -> L3 -> DRAM cliffs, and with
+ * --huge the TLB-reach difference between 4 KiB and 2 MiB pages.
+ *
+ * ./mem_latency # 4 KiB .. 1 GiB, base pages
+ * ./mem_latency --huge # same, madvise(MADV_HUGEPAGE)
+ * numactl --cpunodebind=0 --membind=1 ./mem_latency # remote-node latency
+ *
+ * Output: CSV size_bytes,ns_per_load,huge
+ */
+#define _GNU_SOURCE
+#include
+#include
+#include
+#include
+#include
+#include
+
+static uint64_t rng_state = 0x9E3779B97F4A7C15ull;
+static uint64_t xorshift(void) {
+ rng_state ^= rng_state << 13;
+ rng_state ^= rng_state >> 7;
+ rng_state ^= rng_state << 17;
+ return rng_state;
+}
+
+static double now_ns(void) {
+ struct timespec ts;
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * 1e9 + ts.tv_nsec;
+}
+
+static double chase(size_t bytes, int huge, size_t min_loads) {
+ size_t n = bytes / sizeof(uint64_t);
+ /* One cache line per element would hide spatial locality; one u64 per element
+ with a random permutation already defeats it while keeping the size exact. */
+ uint64_t *a = mmap(NULL, bytes, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (a == MAP_FAILED) return -1;
+ if (huge) madvise(a, bytes, MADV_HUGEPAGE);
+ for (size_t i = 0; i < n; i++) a[i] = i;
+ for (size_t i = n - 1; i > 0; i--) { /* Sattolo: a single cycle through all elements */
+ size_t j = xorshift() % i;
+ uint64_t t = a[i]; a[i] = a[j]; a[j] = t;
+ }
+ size_t loads = n * 4 > min_loads ? n * 4 : min_loads;
+ volatile uint64_t p = 0;
+ for (size_t i = 0; i < n; i++) p = a[p]; /* warm up caches and TLB */
+ double t0 = now_ns();
+ for (size_t i = 0; i < loads; i++) p = a[p];
+ double dt = now_ns() - t0;
+ munmap(a, bytes);
+ return dt / loads + 0 * (double)p;
+}
+
+int main(int argc, char **argv) {
+ int huge = argc > 1 && strcmp(argv[1], "--huge") == 0;
+ size_t max_bytes = (size_t)1 << 30;
+ printf("size_bytes,ns_per_load,huge\n");
+ for (size_t b = 4096; b <= max_bytes; b *= 2) {
+ printf("%zu,%.2f,%d\n", b, chase(b, huge, 20 * 1000 * 1000), huge);
+ fflush(stdout);
+ }
+ return 0;
+}
diff --git a/benchmarks/c/page_fault_cost.c b/benchmarks/c/page_fault_cost.c
new file mode 100644
index 0000000..1521add
--- /dev/null
+++ b/benchmarks/c/page_fault_cost.c
@@ -0,0 +1,62 @@
+/*
+ * page_fault_cost: what a first-touch page fault costs, and how to avoid paying it.
+ *
+ * Touches a fresh anonymous mapping one byte per 4 KiB in four modes:
+ * base - plain mmap, one minor fault per 4 KiB page
+ * populate - MAP_POPULATE, faults taken up front inside mmap()
+ * thp - madvise(MADV_HUGEPAGE), ideally one fault per 2 MiB
+ * prefault - touch once, then time a second pass (no faults: the floor)
+ * Fault counts come from getrusage(RUSAGE_SELF).ru_minflt.
+ *
+ * ./page_fault_cost [MiB=512]
+ * Output: CSV mode,mib,ms,minor_faults,ns_per_4k_page
+ */
+#define _GNU_SOURCE
+#include
+#include
+#include
+#include
+#include
+#include
+
+static double now_ms(void) {
+ struct timespec ts;
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * 1e3 + ts.tv_nsec / 1e6;
+}
+
+static long minflt(void) {
+ struct rusage ru;
+ getrusage(RUSAGE_SELF, &ru);
+ return ru.ru_minflt;
+}
+
+static void run(const char *mode, size_t mib) {
+ size_t bytes = mib << 20, pages = bytes / 4096;
+ int flags = MAP_PRIVATE | MAP_ANONYMOUS | (strcmp(mode, "populate") == 0 ? MAP_POPULATE : 0);
+ long f0 = minflt();
+ double t0 = now_ms();
+ char *p = mmap(NULL, bytes, PROT_READ | PROT_WRITE, flags, -1, 0);
+ if (p == MAP_FAILED) { perror("mmap"); exit(1); }
+ if (strcmp(mode, "thp") == 0) madvise(p, bytes, MADV_HUGEPAGE);
+ if (strcmp(mode, "prefault") == 0) {
+ for (size_t i = 0; i < pages; i++) p[i * 4096] = 1;
+ f0 = minflt();
+ t0 = now_ms();
+ }
+ for (size_t i = 0; i < pages; i++) p[i * 4096] = (char)i;
+ double ms = now_ms() - t0;
+ long faults = minflt() - f0;
+ printf("%s,%zu,%.2f,%ld,%.1f\n", mode, mib, ms, faults, ms * 1e6 / pages);
+ munmap(p, bytes);
+}
+
+int main(int argc, char **argv) {
+ size_t mib = argc > 1 ? strtoull(argv[1], NULL, 10) : 512;
+ printf("mode,mib,ms,minor_faults,ns_per_4k_page\n");
+ run("base", mib);
+ run("populate", mib);
+ run("thp", mib);
+ run("prefault", mib);
+ return 0;
+}
diff --git a/benchmarks/c/wakeup_latency.c b/benchmarks/c/wakeup_latency.c
new file mode 100644
index 0000000..5508d1e
--- /dev/null
+++ b/benchmarks/c/wakeup_latency.c
@@ -0,0 +1,70 @@
+/*
+ * wakeup_latency: thread wakeup latency distribution via pipe ping-pong.
+ *
+ * Two threads bounce a byte through a pair of pipes; each round trip is two
+ * block/wake cycles through the scheduler. Optional pinning shows the effect of
+ * placement: same core (SMT sibling), same socket, or across sockets.
+ * Comparable to `perf bench sched pipe`, but reports percentiles, not just the mean.
+ *
+ * ./wakeup_latency [iterations=200000] [cpuA cpuB]
+ * Output: p50/p90/p99/p999/max one-way latency in ns.
+ */
+#define _GNU_SOURCE
+#include
+#include
+#include
+#include
+#include
+#include
+
+static int ab[2], ba[2];
+static long iters = 200000;
+static int cpu_b = -1;
+
+static void pin(int cpu) {
+ if (cpu < 0) return;
+ cpu_set_t set;
+ CPU_ZERO(&set);
+ CPU_SET(cpu, &set);
+ pthread_setaffinity_np(pthread_self(), sizeof(set), &set);
+}
+
+static void *echo(void *arg) {
+ (void)arg;
+ pin(cpu_b);
+ char c;
+ for (long i = 0; i < iters; i++) {
+ if (read(ab[0], &c, 1) != 1 || write(ba[1], &c, 1) != 1) break;
+ }
+ return NULL;
+}
+
+static int cmp(const void *a, const void *b) {
+ double x = *(const double *)a, y = *(const double *)b;
+ return (x > y) - (x < y);
+}
+
+int main(int argc, char **argv) {
+ if (argc > 1) iters = atol(argv[1]);
+ int cpu_a = argc > 3 ? atoi(argv[2]) : -1;
+ cpu_b = argc > 3 ? atoi(argv[3]) : -1;
+ if (pipe(ab) || pipe(ba)) return 1;
+ pin(cpu_a);
+ pthread_t t;
+ pthread_create(&t, NULL, echo, NULL);
+ double *lat = malloc(sizeof(double) * iters);
+ char c = 'x';
+ for (long i = 0; i < iters; i++) {
+ struct timespec t0, t1;
+ clock_gettime(CLOCK_MONOTONIC, &t0);
+ if (write(ab[1], &c, 1) != 1 || read(ba[0], &c, 1) != 1) break;
+ clock_gettime(CLOCK_MONOTONIC, &t1);
+ lat[i] = ((t1.tv_sec - t0.tv_sec) * 1e9 + (t1.tv_nsec - t0.tv_nsec)) / 2; /* one-way */
+ }
+ pthread_join(t, NULL);
+ qsort(lat, iters, sizeof(double), cmp);
+ printf("cpus=%d,%d iterations=%ld\n", cpu_a, cpu_b, iters);
+ printf("p50=%.0fns p90=%.0fns p99=%.0fns p999=%.0fns max=%.0fns\n", lat[iters / 2], lat[iters * 9 / 10], lat[iters * 99 / 100], lat[iters * 999 / 1000], lat[iters - 1]);
+ free(lat);
+ return 0;
+}
diff --git a/benchmarks/cache_benchmark.c b/benchmarks/cache_benchmark.c
deleted file mode 100644
index ee5f948..0000000
--- a/benchmarks/cache_benchmark.c
+++ /dev/null
@@ -1,13 +0,0 @@
-#include
-#include
-#include
-
-int main(int argc, char **argv) {
- size_t count = argc > 1 ? strtoull(argv[1], NULL, 10) : 8 * 1024 * 1024;
- uint64_t *data = calloc(count, sizeof(*data));
- if (!data) return 1;
- for (size_t i = 0; i < count; i += 16) data[i] += i;
- printf("value=%llu\n", (unsigned long long)data[count - 16]);
- free(data);
- return 0;
-}
diff --git a/benchmarks/cpu_stress.c b/benchmarks/cpu_stress.c
deleted file mode 100644
index 6840022..0000000
--- a/benchmarks/cpu_stress.c
+++ /dev/null
@@ -1,11 +0,0 @@
-#include
-#include
-#include
-
-int main(int argc, char **argv) {
- long iterations = argc > 1 ? strtol(argv[1], NULL, 10) : 100000000L;
- volatile uint64_t value = 1;
- for (long i = 0; i < iterations; i++) value = value * 1664525u + 1013904223u;
- printf("checksum=%llu\n", (unsigned long long)value);
- return 0;
-}
diff --git a/benchmarks/memory_stress.c b/benchmarks/memory_stress.c
deleted file mode 100644
index db50bfb..0000000
--- a/benchmarks/memory_stress.c
+++ /dev/null
@@ -1,14 +0,0 @@
-#include
-#include
-#include
-
-int main(int argc, char **argv) {
- size_t mib = argc > 1 ? strtoull(argv[1], NULL, 10) : 64;
- size_t bytes = mib * 1024 * 1024;
- char *buffer = malloc(bytes);
- if (!buffer) return 1;
- memset(buffer, 0x5a, bytes);
- printf("allocated=%zu MiB\n", mib);
- free(buffer);
- return 0;
-}
diff --git a/benchmarks/numa_benchmark.c b/benchmarks/numa_benchmark.c
deleted file mode 100644
index 9942ed4..0000000
--- a/benchmarks/numa_benchmark.c
+++ /dev/null
@@ -1,14 +0,0 @@
-#include
-#include
-#include
-
-/* Allocation placement is controlled externally (for example, numactl). */
-int main(int argc, char **argv) {
- size_t count = argc > 1 ? strtoull(argv[1], NULL, 10) : 16 * 1024 * 1024;
- uint64_t *data = calloc(count, sizeof(*data));
- if (!data) return 1;
- for (size_t i = 0; i < count; i += 512) data[i] = i;
- printf("pages touched=%zu\n", count * sizeof(*data) / 4096);
- free(data);
- return 0;
-}
diff --git a/benchmarks/page_fault_generator.c b/benchmarks/page_fault_generator.c
deleted file mode 100644
index 4b714a2..0000000
--- a/benchmarks/page_fault_generator.c
+++ /dev/null
@@ -1,14 +0,0 @@
-#include
-#include
-#include
-
-int main(int argc, char **argv) {
- size_t pages = argc > 1 ? strtoull(argv[1], NULL, 10) : 4096;
- long page_size = sysconf(_SC_PAGESIZE);
- char *memory = malloc(pages * (size_t)page_size);
- if (!memory) return 1;
- for (size_t i = 0; i < pages; ++i) memory[i * (size_t)page_size] = (char)i;
- printf("touched=%zu pages\n", pages);
- free(memory);
- return 0;
-}
diff --git a/configs/benchmark_profiles.yaml b/configs/benchmark_profiles.yaml
deleted file mode 100644
index c082f2e..0000000
--- a/configs/benchmark_profiles.yaml
+++ /dev/null
@@ -1,6 +0,0 @@
-cache:
- command: ./build/cache_benchmark
- events: cache-references,cache-misses,cycles,instructions
-memory:
- command: ./build/memory_stress
- events: page-faults,minor-faults,major-faults,task-clock
diff --git a/configs/default.yaml b/configs/default.yaml
deleted file mode 100644
index b131f5d..0000000
--- a/configs/default.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-events:
- - cycles
- - instructions
- - cache-references
- - cache-misses
- - context-switches
- - cpu-migrations
- - page-faults
-duration_seconds: 10
diff --git a/configs/thresholds.example.json b/configs/thresholds.example.json
new file mode 100644
index 0000000..208b31b
--- /dev/null
+++ b/configs/thresholds.example.json
@@ -0,0 +1,11 @@
+{
+ "cpu_psi_some_warn": 0.10,
+ "runq_wait_share_warn": 0.10,
+ "involuntary_switch_rate_warn": 1000,
+ "major_fault_rate_warn": 10,
+ "remote_alloc_warn": 0.10,
+ "numa_placement_mismatch_warn": 0.30,
+ "llc_mpki_warn": 5.0,
+ "dtlb_miss_warn": 0.01,
+ "thp_fallback_warn": 0.20
+}
diff --git a/docs/architecture.md b/docs/architecture.md
index e69de29..302e238 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -0,0 +1,84 @@
+# Architecture
+
+## Goals
+
+1. **Safe to run on production hosts**: no kernel modules, no eBPF, no `perf` binary, no
+ runtime dependencies, sub-millisecond-per-sample overhead, read-only.
+2. **Correct kernel semantics**: every metric traces back to a documented kernel counter
+ and is interpreted the way the kernel defines it (see [metrics.md](metrics.md)).
+3. **Actionable output**: findings with evidence and a concrete next step, not a wall of numbers.
+4. **Reproducible analysis**: what was captured can be re-analysed anywhere, later.
+
+## Capture, then analyse
+
+```mermaid
+flowchart LR
+ subgraph host["Host under study"]
+ P["/proc + /sys
text files"] -->|read verbatim| R["Recorder
deadline-scheduled"]
+ K["perf_event_open
(ctypes)"] -->|counters| PR["profile
fork → attach → exec"]
+ R --> PR
+ end
+ R --> J[("recording.jsonl.gz
meta · static · samples · counters")]
+ PR --> J
+ J --> E["analysis engine
(pure functions)"]
+ T["perf stat text
(optional)"] --> E
+ E --> D1[cpu] & D2[memory] & D3[numa] & D4[cache] & D5[hugepages] & D6[config]
+ D1 & D2 & D3 & D4 & D5 & D6 --> F["findings · USE table · time series"]
+ F --> O["Markdown / JSON report
CI gate (--fail-on)"]
+ F --> C["compare: permutation test
across runs"]
+```
+
+The recorder **never parses**. It copies file contents into the recording and timestamps
+them. This split matters for three reasons:
+
+* **Overhead**: a sample is ~20-40 `open/read/close` calls, ~1 ms on a laptop and far below 1%
+ of a core at 1 s intervals. Parsing happens off the hot path.
+* **Portability**: a recording from a locked-down production host can be analysed on a laptop,
+ attached to a ticket, and re-analysed after a parser fix or threshold change.
+* **Testability**: analysis is a pure function `Recording -> Report`. Tests feed it synthetic
+ recordings (`kptk/synth.py`) of a two-socket server and assert exact findings.
+
+## Recording format
+
+JSONL (optionally gzipped), one object per line:
+
+| line | content |
+|---|---|
+| `{"type": "meta", ...}` | host, kernel, arch, `user_hz`, page size, interval, target pid/command, `static` files |
+| `{"type": "sample", "ts", "cost_s", "files"}` | dynamic files at one instant, plus the recorder's own cost |
+| `{"type": "counters", "events"}` | hardware/software counter totals (optional) |
+
+**Static** (captured once): CPU topology, cache hierarchy, NUMA cpulists and SLIT distances,
+THP and khugepaged settings, hugetlb pools, relevant sysctls, cpufreq governors.
+**Dynamic** (every sample): `/proc/{stat,schedstat,vmstat,meminfo,loadavg}`, PSI, per-node
+`numastat` and `meminfo`, and for a target pid `stat`, `status`, `schedstat`.
+**Slow** (every N samples): `/proc//{numa_maps,smaps_rollup}`. These walk the target's
+page tables under its mmap lock, so reading them every sample would perturb the process being
+measured.
+
+## Modules
+
+| Module | Role |
+|---|---|
+| `parsers/proc.py`, `parsers/sysfs.py` | Pure parsers for /proc and /sys formats |
+| `parsers/tools.py` | `perf stat` (CSV + human, keeps multiplexing %), `vmstat`, `numastat` text |
+| `record.py` | `Source` abstraction (live / dict), `Recorder`, `Recording` I/O |
+| `perf_event.py` | `perf_event_open(2)` via ctypes: attr struct, event encoding, multiplex scaling |
+| `profile.py` | fork → attach counters (`enable_on_exec`) → exec, plus recording until exit |
+| `topology.py` | CPUs, SMT, packages, NUMA nodes and distances, cache hierarchy |
+| `analysis/window.py` | First/last-sample deltas and rates |
+| `analysis/{cpu,memory,numa,cache,hugepages}.py` | Domain analyzers → metrics + findings |
+| `analysis/engine.py` | Orchestration, config audit, USE summary, time series, data quality |
+| `analysis/findings.py` | `Finding`, `Severity`, tunable `Thresholds` |
+| `compare.py` | Multi-run A/B comparison with a permutation test |
+| `report.py` | Markdown rendering |
+| `synth.py` | Synthetic two-socket recordings for tests and `kptk demo` |
+| `cli.py` | `inspect`, `record`, `profile`, `analyze`, `compare`, `parse`, `demo` |
+| `benchmarks/c/` | Microbenchmarks that reproduce each phenomenon on real hardware |
+
+## Failure handling
+
+Kernels differ (CONFIG_SCHEDSTATS, PSI, THP, perf permissions, VMs without a PMU), and
+processes exit mid-recording. Every reader returns `None` instead of raising, every analyzer
+skips what it can't compute, and the report's **data quality** section lists what was missing
+and why. A partial report says so explicitly instead of failing silently.
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
new file mode 100644
index 0000000..89b2c74
--- /dev/null
+++ b/docs/design-decisions.md
@@ -0,0 +1,62 @@
+# Design decisions
+
+## ADR-1: Capture raw text, parse later
+**Decision.** The recorder copies /proc and /sys files verbatim; all parsing happens at analysis time.
+**Why.** It keeps per-sample overhead minimal, makes recordings portable (capture on prod,
+analyse on a laptop), lets old recordings benefit from parser fixes, and turns analysis into
+a pure, testable function. **Cost**: recordings are larger than parsed numbers (~1 KB/s gzipped
+for system-wide sampling), which is acceptable for minutes-long captures.
+
+## ADR-2: Zero runtime dependencies
+**Why.** The tool runs on production hosts where installing packages is a change-management
+event. Python 3.10+ stdlib covers file I/O, ctypes (for `perf_event_open`), JSON, gzip and
+statistics. YAML configs from v0.1 became JSON for the same reason.
+
+## ADR-3: `perf_event_open` via ctypes instead of shelling out to `perf`
+**Why.** `perf` isn't installed on many hosts, its version must match the kernel, and parsing
+its output is brittle. The syscall ABI is stable: the 112-byte `PERF_ATTR_SIZE_VER5` struct is
+accepted by every kernel since 4.1. **Trade-off**: no sampling, call graphs or symbolisation.
+This is a counting tool. Sampling profilers (perf record, eBPF) complement it.
+
+## ADR-4: Attach counters before exec (`enable_on_exec`), not after spawn
+**Why.** Attaching after the child starts misses its startup and races with short commands.
+The fork → block on pipe → attach → release → exec handshake (the one `perf stat` uses)
+counts exactly the target from its first instruction. `subprocess.Popen` can't express it,
+because it waits for exec to complete before returning, which deadlocks the handshake.
+Hence raw `os.fork` / `os.execvp`.
+
+## ADR-5: Ungrouped counters with `inherit` for commands
+**Why.** Counting all threads and children needs `inherit=1`, and the kernel doesn't support
+`PERF_FORMAT_GROUP` reads of inherited events. Each counter is therefore read separately and
+scaled by its own `time_enabled/time_running`. The report flags multiplexing so that
+cross-counter ratios (IPC, miss rates) are read as estimates when it occurred.
+
+## ADR-6: Saturation over utilisation
+**Why.** Utilisation is the most-reported and least-diagnostic metric. Findings trigger on
+saturation signals (run-queue delay, PSI, direct reclaim, compaction stalls) that directly
+represent waiting. Utilisation is kept as context in the USE table.
+
+## ADR-7: Correct NUMA semantics, and mismatch instead of "remote %"
+**Why.** `numa_miss` is a policy counter, not a locality counter. Locality comes from
+`local_node`/`other_node` (allocation) and AutoNUMA hinting faults (access). For a process, a
+naive "remote %" under uniform access flags any process spread across nodes at ~50%, even a
+perfectly interleaved one. The finding triggers on the **mismatch** (total variation distance)
+between where threads ran and where memory lives. A unit test pins that distinction.
+
+## ADR-8: Rate-limit page-table-walking reads
+**Why.** `numa_maps` and `smaps_rollup` walk the target's page tables while holding its mmap
+lock. For a 100 GB process that takes milliseconds, and it contends with the target's own page
+faults. They're read every N samples (`--slow-every`), and the latest value is used.
+
+## ADR-9: Synthetic recordings as the test oracle
+**Why.** Most findings need hardware a CI runner doesn't have (two sockets, a PMU, memory
+pressure). `synth.py` generates internally consistent /proc text for a two-socket server
+under named scenarios. Tests assert that the *exact* set of findings fires, and that metrics
+recover the injected ground truth. Real recordings in `samples/` keep the parsers honest
+against live kernel output.
+
+## ADR-10: Noise-aware comparison
+**Why.** Performance changes of a few percent are often within run-to-run noise. `compare`
+uses a permutation test (exact for small n, distribution-free, stdlib-only) and requires both
+significance and a minimum effect size. With n=1 per side it explicitly refuses to call a
+result significant.
diff --git a/docs/examples.md b/docs/examples.md
index e69de29..00b2dba 100644
--- a/docs/examples.md
+++ b/docs/examples.md
@@ -0,0 +1,82 @@
+# Workflows
+
+## 1. "Is this host healthy?" (30 seconds, unprivileged)
+
+```bash
+kptk inspect # topology, caches, THP/hugetlb, risky sysctls
+kptk record -d 30 -i 1 -o host.jsonl.gz
+kptk analyze host.jsonl.gz
+```
+
+## 2. Profile one command end to end
+
+```bash
+kptk profile -i 0.25 -o run.jsonl.gz -- ./my_service --bench
+```
+
+Hardware counters attach at `exec`, so the profiler itself isn't counted. If
+`perf_event_paranoid` blocks them, the report says so and still covers scheduling,
+memory, NUMA and huge pages from /proc.
+
+No counter access, but someone can run `perf stat` for you? Pass the output in:
+
+```bash
+perf stat -x';' -e cycles,instructions,cache-references,cache-misses,dTLB-loads,dTLB-load-misses -- ./my_service 2> perf.txt
+kptk analyze run.jsonl.gz --perf-stat perf.txt
+```
+
+## 3. Watch a running service
+
+```bash
+kptk record -p $(pidof postgres | cut -d' ' -f1) -d 120 -i 1 -o pg.jsonl.gz
+kptk analyze pg.jsonl.gz --md pg.md --json pg.json
+```
+
+## 4. Prove a tuning change (A/B with significance)
+
+```bash
+for i in 1 2 3 4 5; do kptk profile -i 0.25 --json before_$i.json --md /dev/null -- ./bench; done
+# apply change: numactl binding, THP mode, thread-pool size, ...
+for i in 1 2 3 4 5; do kptk profile -i 0.25 --json after_$i.json --md /dev/null -- ./bench; done
+kptk compare --before before_*.json --after after_*.json --fail-on-regression
+```
+
+Each metric gets a two-sided permutation test. "regressed" requires both p < 0.05 and a
+change of at least 5% (`--min-change`).
+
+## 5. CI performance gate
+
+```bash
+kptk profile --fail-on warning --md perf-report.md -- ./integration_bench
+# exit status 3 if any warning/critical finding: fail the pipeline, attach perf-report.md
+```
+
+## 6. Case study (synthetic): the misplaced database
+
+`kptk demo numa_misplaced` (report: [`samples/numa_misplaced_report.md`](../samples/numa_misplaced_report.md)):
+
+1. **Critical NUMA misplacement**: threads ran ~100% on node 0, while 85% of the heap sits on
+ node 1 (placement mismatch 0.85, expected SLIT distance 19.4 vs 10 local).
+2. That explains **memory-bound** counters: IPC 0.60 and LLC MPKI 17.9.
+3. Node 1 is **nearly full**, so new allocations go remote too (35% remote allocations; AutoNUMA
+ sees only 25% local accesses).
+4. **THP always/always** with a 44% fallback rate and 12 compaction stalls/s: the heap is only
+ 4% huge-page backed, while dTLB misses are 3%.
+5. And 4 GiB of **hugetlb pool** is reserved but unused: memory that could have relieved node 1.
+
+The fix order follows from the evidence: bind the database to node 1 (or migrate its pages
+to node 0), switch THP defrag to `defer+madvise`, then either give the 4 GiB hugetlb pool to
+the DB (as huge-page shared buffers) or release it.
+
+## 7. Reproducing phenomena with the microbenchmarks
+
+```bash
+make bench-build
+./build/mem_latency > lat_4k.csv; ./build/mem_latency --huge > lat_2m.csv # cache + TLB cliffs
+numactl --cpunodebind=0 --membind=1 ./build/mem_latency > lat_remote.csv # NUMA penalty
+./build/page_fault_cost 1024 # fault cost by mode
+./build/wakeup_latency 200000 0 1; ./build/wakeup_latency 200000 0
+./build/false_sharing 8
+```
+
+These are deliberately heavy. Run them on an otherwise idle machine, and never in CI.
diff --git a/docs/linux-performance.md b/docs/linux-performance.md
index e69de29..4116bc5 100644
--- a/docs/linux-performance.md
+++ b/docs/linux-performance.md
@@ -0,0 +1,81 @@
+# Linux performance: the kernel mechanics behind each finding
+
+Background for each domain the toolkit analyses: what the kernel does, how it shows up in the
+counters, and which microbenchmark in `benchmarks/c/` reproduces it.
+
+## Method: USE first, then drill down
+
+For every resource, check **U**tilisation, **S**aturation and **E**rrors (Brendan Gregg's USE
+method). Utilisation alone misleads: a CPU at 70% average can have a run queue ten tasks deep
+in bursts. `kptk` reports a USE row per resource, then domain findings.
+
+## CPU scheduling (CFS / EEVDF)
+
+* Every runnable task waits on a per-CPU run queue. The time between becoming runnable and
+ actually running is **scheduler latency**, counted per CPU in `/proc/schedstat` (`run_delay`)
+ and per task in `/proc//schedstat`. It's invisible in `top`.
+* **Voluntary** switches (blocking) vs **involuntary** switches (preemption) tell lock and I/O
+ waits apart from CPU oversubscription.
+* **Migrations** across cores cost cache warmth; across sockets they also cost memory locality.
+ The load balancer trades these against idle CPUs.
+* Mitigations: right-size thread pools, `taskset`/cpusets, `isolcpus`/`nohz_full` for dedicated
+ cores, IRQ affinity, SCHED_FIFO for the few threads that truly need it.
+* Reproduce: `wakeup_latency` (block/wake latency percentiles, pinned to the same core, the same
+ socket or across sockets).
+
+## Memory and page faults
+
+* Anonymous memory is allocated lazily: `mmap`/`malloc` reserve address space, and the first
+ touch of each page takes a **minor fault** that zero-fills it. File-backed pages not in the
+ page cache take **major faults**.
+* Allocators return memory to the OS (`munmap`, `MADV_DONTNEED`), so steady-state churn can
+ keep faulting. Mitigations are buffer reuse, `MAP_POPULATE`, and huge pages (one fault per 2 MiB).
+* Under pressure, `kswapd` reclaims in the background. When it falls behind, allocating tasks
+ enter **direct reclaim** (`allocstall_*`), and PSI `memory full` shows lost throughput.
+* Reproduce: `page_fault_cost` (base vs `MAP_POPULATE` vs THP vs pre-faulted).
+
+## NUMA
+
+* Each socket (or sub-NUMA cluster) owns memory. Remote access costs ~1.3-2x local latency
+ and consumes interconnect (UPI/Infinity Fabric) bandwidth.
+* Default policy is **first touch**: a page lands on the node of the CPU that first writes it.
+ The classic bug is a single thread initialising a big buffer that many threads on other
+ nodes then use.
+* AutoNUMA (`kernel.numa_balancing`) samples accesses with hinting faults and migrates pages
+ or tasks. It helps long-running unpinned workloads and hurts when threads bounce around.
+* Policies: `numactl --cpunodebind --membind`, `--interleave` for shared read-mostly data,
+ cpuset `mems`, `mbind`/`set_mempolicy` in code, and `migratepages` after the fact.
+* Reproduce: `numactl --cpunodebind=0 --membind=1 ./mem_latency` vs `--membind=0`.
+
+## Caches
+
+* Typical server hierarchy: L1d 32-48 KiB and L2 1-2 MiB per core, L3 tens of MiB per socket,
+ then DRAM. Latency roughly 1 ns, 4 ns, 15-40 ns, then 80-120 ns.
+* Dependent loads (pointer chasing) expose full latency. Independent streams let the
+ prefetcher and memory-level parallelism hide it.
+* **False sharing**: independent variables on one 64-byte line ping-pong between cores through
+ coherence (MESI) traffic.
+* Reproduce: `mem_latency` (latency cliffs per level) and `false_sharing` (packed vs padded).
+
+## TLB and huge pages
+
+* The TLB caches virtual→physical translations. With a few thousand entries of 4 KiB, reach is
+ only a few MiB. Beyond that, every miss walks a 4-level (or 5-level) page table.
+* A 2 MiB huge page covers 512x more per entry. Two ways to get them:
+ * **THP**: automatic. `enabled=always|madvise|never` decides who gets them, and
+ `defrag` decides whether a fault may *stall* to compact memory (`always`) or defers to
+ khugepaged (`defer`, `defer+madvise`). `always/always` is the classic latency-spike setting.
+ * **hugetlbfs**: an explicit reserved pool, predictable, but the memory is set aside even when unused.
+* Fragmentation makes huge-page allocation fail (`thp_fault_fallback`), and compaction to fix
+ it costs CPU and latency (`compact_stall`).
+* Reproduce: `mem_latency` vs `mem_latency --huge` (TLB-reach cliff moves out).
+
+## Hardware counters and their limits
+
+* `perf_event_open` programs the PMU. A core has only a few programmable counters. Beyond
+ that the kernel **multiplexes** and scales by `time_enabled / time_running`. Ratios are exact
+ only within an event group.
+* Generic events (`cache-misses`) map to model-specific events, so compare values only on the
+ same CPU model.
+* Access is gated by `kernel.perf_event_paranoid` (Ubuntu defaults to 4 = no unprivileged
+ access) or `CAP_PERFMON`.
diff --git a/docs/metrics.md b/docs/metrics.md
index e69de29..61e5500 100644
--- a/docs/metrics.md
+++ b/docs/metrics.md
@@ -0,0 +1,66 @@
+# Metric reference
+
+Each metric: where it comes from, how it's computed over the window `[first, last]` sample,
+and how to read it. `Δx` is the change in a cumulative counter, `Δt` the elapsed time.
+
+## CPU and scheduler
+
+| Metric | Source | Formula | Reading |
+|---|---|---|---|
+| `utilization` | `/proc/stat` cpu line | Δbusy / Δtotal (busy excludes idle, iowait, and guest, which is already in user) | How busy, not how *contended* |
+| `steal_frac` | `/proc/stat` | Δsteal / Δtotal | vCPU runnable but the hypervisor ran someone else |
+| `iowait_frac` | `/proc/stat` | Δiowait / Δtotal | Idle with I/O outstanding. It's idle time, not CPU work |
+| `runq_waiting_tasks_avg` | `/proc/schedstat` field 8 (`run_delay`) | ΣΔrun_delay_ns / 1e9 / Δt | Average number of tasks runnable but waiting: direct **saturation** |
+| `runq_wait_per_timeslice_ms` | `/proc/schedstat` fields 8, 9 | ΣΔrun_delay / ΣΔpcount | Mean scheduler latency per time slice |
+| `psi_cpu_some` | `/proc/pressure/cpu` `some total=` | Δtotal_us / 1e6 / Δt | Share of wall time at least one task waited for CPU |
+| `process_runq_wait_share` | `/proc//schedstat` | Δwait / (Δwait + Δon_cpu) | Share of the process's runnable time spent waiting: its scheduler latency tax |
+| `process_involuntary_switches_per_s` | `/proc//status` `nonvoluntary_ctxt_switches` | Δ / Δt | Preempted while runnable, so **CPU contention** |
+| `process_voluntary_switches_per_s` | `/proc//status` `voluntary_ctxt_switches` | Δ / Δt | Blocked (I/O, futex, sleep), so **blocking/lock contention** |
+| `process_node_residency` | `/proc//stat` field 39 (`processor`) per sample | histogram of CPU → node | Where the threads actually ran |
+
+## Memory and page faults
+
+| Metric | Source | Reading |
+|---|---|---|
+| `faults_per_s`, `major_faults_per_s` | `/proc/vmstat` `pgfault`, `pgmajfault` | Minor faults cost ~0.1-1 µs; major faults block on storage (~10 µs NVMe to ~10 ms HDD) |
+| `process_minor/major_faults_per_s` | `/proc//stat` fields 10, 12 | Per-process fault rates |
+| `direct_reclaim_stalls_per_s` | `/proc/vmstat` `allocstall_*` | An allocating task had to reclaim memory itself: latency spikes |
+| `pgscan_direct_per_s` vs `pgscan_kswapd_per_s` | `/proc/vmstat` | Reclaim by allocators vs by the background daemon |
+| `swap_pages_per_s` | `/proc/vmstat` `pswpin + pswpout` | Any sustained value means the working set doesn't fit |
+| `psi_memory_some` / `psi_memory_full` | `/proc/pressure/memory` | *full* = no task could progress because of memory: lost throughput |
+| `mem_available_min_frac` | `/proc/meminfo` `MemAvailable / MemTotal` | Minimum over the window |
+
+## NUMA
+
+| Metric | Source | Reading |
+|---|---|---|
+| `remote_alloc_frac` | node `numastat` `other_node / (local_node + other_node)` | Allocations placed on a node other than the allocating CPU's. **Locality** |
+| `policy_miss_frac` | node `numastat` `numa_miss / (numa_hit + numa_miss)` | Allocation-*policy* misses (preferred node full). **Not** locality |
+| `hint_fault_local_frac` | `/proc/vmstat` `numa_hint_faults_local / numa_hint_faults` | AutoNUMA's sampled *access* locality (needs `numa_balancing=1`) |
+| `pages_migrated_per_s` | `/proc/vmstat` `numa_pages_migrated` | AutoNUMA migration cost |
+| `process_memory_by_node` | `/proc//numa_maps` `N=` | Where the process's pages live (huge pages normalised to 4 KiB units) |
+| `process_placement_mismatch` | residency vs memory_by_node | Total variation distance ½Σ\|cpu_n − mem_n\|: 0 = matched, 1 = fully apart |
+| `process_remote_access_frac_est` | same | Σ cpu_c · mem_m over c ≠ m: remote share *if* every thread touches all memory uniformly |
+| `process_expected_distance` | + SLIT `distance` | Σ cpu_c · mem_m · dist(c, m): 10 = all local |
+| `node_free_frac` | node `meminfo` | A full node pushes allocations remote (with `zone_reclaim_mode=0`) |
+
+## Cache, branch, TLB (hardware counters)
+
+| Metric | Formula | Reading |
+|---|---|---|
+| `ipc` | instructions / cycles | < 1 on a modern core usually means stalls; > 2 is pipeline-efficient |
+| `llc_mpki` | cache-misses / (instructions / 1000) | ~80 ns per DRAM miss: 10 MPKI ≈ 2.4 cycles/instr of exposure at 3 GHz |
+| `llc_miss_ratio` | cache-misses / cache-references | Generic events; mapping is CPU-specific (LLC on most Intel) |
+| `branch_miss_ratio`, `branch_mpki` | branch-misses / branches | ~15-20 cycles per mispredict |
+| `dtlb_miss_ratio`, `dtlb_mpki` | dTLB-load-misses / dTLB-loads | Page-walk pressure, which huge pages reduce |
+| `min_running_frac` | time_running / time_enabled | < 1 means counters were multiplexed and values are extrapolated |
+
+## Huge pages
+
+| Metric | Source | Reading |
+|---|---|---|
+| `thp_enabled`, `thp_defrag` | `/sys/kernel/mm/transparent_hugepage/*` | `always`+`always` = synchronous compaction in the fault path |
+| `thp_fallback_frac` | vmstat `thp_fault_fallback / (alloc + fallback)` | Huge-page faults that got 4 KiB pages: fragmentation |
+| `compact_stall_per_s` | vmstat `compact_stall` | Tasks that compacted memory synchronously |
+| `process_thp_coverage` | smaps_rollup `AnonHugePages / Anonymous` | Share of the heap backed by THP |
+| `hugetlb_pools` | `/sys/kernel/mm/hugepages/hugepages-*/` | `free - resv` pages are reserved but unused memory |
diff --git a/images/architecture.png b/images/architecture.png
deleted file mode 100644
index e69de29..0000000
diff --git a/images/cache.png b/images/cache.png
deleted file mode 100644
index e69de29..0000000
diff --git a/images/numa.png b/images/numa.png
deleted file mode 100644
index e69de29..0000000
diff --git a/images/sample_dashboard.png b/images/sample_dashboard.png
deleted file mode 100644
index e69de29..0000000
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..d1925eb
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,54 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "kernel-performance-toolkit"
+version = "0.2.0"
+description = "Linux kernel performance analysis: CPU scheduling, memory, NUMA locality, cache efficiency, page faults and huge pages"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "Apache-2.0" }
+authors = [{ name = "nareshns2004" }]
+keywords = ["linux", "kernel", "performance", "numa", "perf", "hugepages", "scheduler", "observability"]
+classifiers = [
+ "Programming Language :: Python :: 3",
+ "Operating System :: POSIX :: Linux",
+ "License :: OSI Approved :: Apache Software License",
+ "Topic :: System :: Monitoring",
+ "Topic :: System :: Operating System Kernels :: Linux",
+]
+dependencies = [] # standard library only, by design
+
+[project.optional-dependencies]
+dev = ["pytest>=7.0", "ruff>=0.5", "mypy>=1.10"]
+
+[project.scripts]
+kptk = "kptk.cli:main"
+
+[project.urls]
+Repository = "https://github.com/nareshns2004/kernel-performance-toolkit"
+
+[tool.setuptools]
+package-dir = { "" = "src" }
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 160
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP", "SIM", "RUF"]
+ignore = ["E501", "RUF001", "RUF002", "RUF003"]
+
+[tool.mypy]
+python_version = "3.10"
+files = ["src"]
+disallow_untyped_defs = true
+warn_unused_ignores = true
diff --git a/requirements.txt b/requirements.txt
index 55c084d..ecb4476 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,2 +1,3 @@
-# The core toolkit uses only Python's standard library.
-# pytest>=7.0 # required only for running tests
+# kptk has no runtime dependencies: Python >= 3.10 standard library only.
+# Development tools:
+# pip install -e ".[dev]"
diff --git a/samples/laptop_profile.jsonl.gz b/samples/laptop_profile.jsonl.gz
new file mode 100644
index 0000000..2597c50
Binary files /dev/null and b/samples/laptop_profile.jsonl.gz differ
diff --git a/samples/laptop_profile_report.md b/samples/laptop_profile_report.md
new file mode 100644
index 0000000..cf171ce
--- /dev/null
+++ b/samples/laptop_profile_report.md
@@ -0,0 +1,152 @@
+# Kernel performance report
+
+**Host** notebook · kernel 6.8.0-138-generic · 8 CPUs (4 cores, 1 socket(s), SMT on) · 1 NUMA node(s) · L1D 48 KiB, L1I 32 KiB, L2 1280 KiB, L3 8192 KiB
+
+**Window** 1.201 s, 8 samples, recorder cost 1.379 ms/sample · **target** pid 2579584 (python3) · `python3 -c import time; x=[bytearray(1<<20) for _ in range(150)]; time.sleep(1)`
+
+## Findings
+
+No warnings: nothing crossed a threshold in this window.
+
+### 🔵 No hardware counters in this recording; cache and TLB efficiency can't be measured
+
+**Action:** Use `kptk profile -- ` with perf_event_paranoid <= 2 (or CAP_PERFMON), or pass `--perf-stat` with saved `perf stat -x;` output.
+
+### 🔵 CPU frequency governor 'powersave' is active
+
+Evidence: `governors=['powersave']`
+
+**Action:** On intel_pstate/amd-pstate 'powersave' is dynamic and often fine; for latency-critical services compare against 'performance' and check C-state exit latency.
+
+### 🔵 kernel.perf_event_paranoid=4: unprivileged hardware counters are disabled
+
+Evidence: `perf_event_paranoid=4`
+
+**Action:** For profiling sessions grant CAP_PERFMON to the profiler, or temporarily set perf_event_paranoid=2 (own processes, user+kernel).
+
+### 🔵 Single NUMA node: locality analysis doesn't apply
+
+Evidence: `nodes=1`
+
+**Action:** On multi-socket or sub-NUMA-clustered servers, record there to get locality findings.
+
+## USE summary
+
+| resource | utilisation | saturation | errors |
+|---|---:|---|---|
+| CPU | 0.0833 | psi_some: 0.00788, runq_waiting_tasks: 0.0478 | steal_frac: 0 |
+| Memory capacity | 0.853 | psi_some: 0, direct_reclaim_per_s: 0, swap_pages_per_s: 0 | oom_kills: 0 |
+| NUMA interconnect | - | - | - |
+| Huge pages | 0 | compact_stall_per_s: 0 | - |
+
+## Over time
+
+```
+cpu util ▆▃▄▄▄█▁ max 0.155
+runq waiting ▇▁▁▁▂█▁ max 0.13
+faults/s █▁▁▁▁▁▁ max 200,929
+major flt/s ▁▁▁▁▁▁▁ max 0
+mem pressure ▁▁▁▁▁▁▁ max 0
+```
+
+## Metrics
+
+cpu
+
+| metric | value |
+|---|---:|
+| utilization | 0.0833 |
+| user_frac | 0.0556 |
+| system_frac | 0.0267 |
+| iowait_frac | 0.00321 |
+| steal_frac | 0 |
+| softirq_frac | 0.00107 |
+| irq_frac | 0 |
+| context_switches_per_s | 11,538 |
+| procs_running_mean | 1.25 |
+| procs_running_per_cpu | 0.156 |
+| per_cpu_spread | 0.093 |
+| runq_waiting_tasks_avg | 0.0478 |
+| runq_wait_per_timeslice_ms | 0.00806 |
+| psi_cpu_some | 0.00788 |
+| process_comm | python3 |
+| process_threads | 1 |
+| process_cpus_used | 0.0666 |
+| process_system_frac | 0.75 |
+| process_runq_wait_share | 0.00171 |
+| process_runq_wait_per_slice_ms | 0.0184 |
+| process_voluntary_switches_per_s | 1.67 |
+| process_involuntary_switches_per_s | 5 |
+| process_cpus_allowed | 0-7 |
+
+memory
+
+| metric | value |
+|---|---:|
+| faults_per_s | 43,415 |
+| major_faults_per_s | 0 |
+| swap_pages_per_s | 0 |
+| direct_reclaim_stalls_per_s | 0 |
+| pgscan_direct_per_s | 0 |
+| pgscan_kswapd_per_s | 0 |
+| oom_kills | 0 |
+| workingset_refault_per_s | 35 |
+| mem_total_kib | 16110848 |
+| mem_available_min_frac | 0.147 |
+| psi_memory_some | 0 |
+| psi_memory_full | 0 |
+| process_minor_faults_per_s | 32,501 |
+| process_major_faults_per_s | 0 |
+| process_rss_kib | 0 |
+| process_anon_kib | 158500 |
+| process_swap_kib | 0 |
+
+numa
+
+| metric | value |
+|---|---:|
+| nodes | 1 |
+| applicable | False |
+
+cache
+
+| metric | value |
+|---|---:|
+| available | False |
+
+hugepages
+
+| metric | value |
+|---|---:|
+| thp_enabled | madvise |
+| thp_defrag | madvise |
+| khugepaged_max_ptes_none | 511 |
+| thp_fault_alloc_per_s | 0 |
+| thp_collapse_per_s | 0 |
+| thp_split_per_s | 0 |
+| compact_stall_per_s | 0 |
+| anon_huge_kib_system | 0 |
+| process_anon_kib | 158500 |
+| process_thp_coverage | 0 |
+| process_hugetlb_kib | 0 |
+
+config
+
+| metric | value |
+|---|---:|
+| swappiness | 60 |
+| zone_reclaim_mode | 0 |
+| numa_balancing | 0 |
+| perf_event_paranoid | 4 |
+
+cpu
+
+| metric | value |
+|---|---:|
+| utilization | 0.4 |
+| user_frac | 0.32 |
+| system_frac | 0.08 |
+| iowait_frac | 0 |
+| steal_frac | 0 |
+| softirq_frac | 0 |
+| irq_frac | 0 |
+| context_switches_per_s | 50,000 |
+| procs_running_mean | 13 |
+| procs_running_per_cpu | 0.406 |
+| per_cpu_spread | 0 |
+| runq_waiting_tasks_avg | 0.05 |
+| runq_wait_per_timeslice_ms | 0.00156 |
+| psi_cpu_some | 0.01 |
+| process_comm | pgdb |
+| process_threads | 32 |
+| process_cpus_used | 4 |
+| process_system_frac | 0.15 |
+| process_runq_wait_share | 0.02 |
+| process_runq_wait_per_slice_ms | 0.0163 |
+| process_voluntary_switches_per_s | 2,000 |
+| process_involuntary_switches_per_s | 100 |
+| process_cpus_allowed | 0-31 |
+
+memory
+
+| metric | value |
+|---|---:|
+| faults_per_s | 20,000 |
+| major_faults_per_s | 0 |
+| swap_pages_per_s | 0 |
+| direct_reclaim_stalls_per_s | 0 |
+| pgscan_direct_per_s | 0 |
+| pgscan_kswapd_per_s | 0 |
+| oom_kills | 0 |
+| workingset_refault_per_s | 0 |
+| mem_total_kib | 134217728 |
+| mem_available_min_frac | 0.258 |
+| psi_memory_some | 0 |
+| psi_memory_full | 0 |
+| process_minor_faults_per_s | 2,000 |
+| process_major_faults_per_s | 0 |
+| process_rss_kib | 50331648 |
+| process_anon_kib | 50331648 |
+| process_swap_kib | 0 |
+
+numa
+
+| metric | value |
+|---|---:|
+| nodes | 2 |
+| applicable | True |
+| remote_alloc_frac | 0.35 |
+| policy_miss_frac | 0 |
+| numa_balancing | 1 |
+| hint_fault_local_frac | 0.25 |
+| pages_migrated_per_s | 0 |
+| process_expected_distance | 19.4 |
+| process_remote_access_frac_est | 0.85 |
+| process_placement_mismatch | 0.85 |
+
+cache
+
+| metric | value |
+|---|---:|
+| available | True |
+| ipc | 0.6 |
+| llc_miss_ratio | 0.478 |
+| llc_mpki | 17.9 |
+| branch_miss_ratio | 0.005 |
+| branch_mpki | 0.833 |
+| dtlb_miss_ratio | 0.03 |
+| dtlb_mpki | 10 |
+| min_running_frac | 1 |
+
+hugepages
+
+| metric | value |
+|---|---:|
+| thp_enabled | always |
+| thp_defrag | always |
+| khugepaged_max_ptes_none | 511 |
+| thp_fault_alloc_per_s | 50 |
+| thp_fallback_frac | 0.444 |
+| thp_collapse_per_s | 0 |
+| thp_split_per_s | 0 |
+| compact_stall_per_s | 12 |
+| compact_fail_frac | 0.5 |
+| anon_huge_kib_system | 0 |
+| process_anon_kib | 50331648 |
+| process_thp_coverage | 0.04 |
+| process_hugetlb_kib | 0 |
+
+config
+
+| metric | value |
+|---|---:|
+| swappiness | 60 |
+| zone_reclaim_mode | 0 |
+| numa_balancing | 1 |
+| perf_event_paranoid | 2 |
+
+{domain}
", "", "| metric | value |", "|---|---:|"]
+ out += [f"| {k} | {_fmt(v)} |" for k, v in rows]
+ out += ["", "