feat(logs): make log-stream identity a per-tenant field policy

[tdakkota]

Implements steps 1-3 of #104's work breakdown, plus the attrsByteCol blocker named in the issue
thread. Stacks on #153 for its memory numbers (independent for correctness).

Design

Only the resource attributes named by tenant.Streams.Fields are hashed into a log stream's id —
default tenant.DefaultStreamFields (service.name/service.namespace +
k8s.namespace.name/node.name/deployment.name/pod.name), with AllFields to opt out. So
service.instance.id and friends stop minting a stream per process restart.

Every resource attribute is still stored on every record, in a new resource column — the
VictoriaLogs model, where a stream field is both hashed and stored. Excluding a key therefore
changes only how it is indexed: log.Resource(stream, blob) reassembles the full OTLP resource, and
a condition answers any key whether or not it is in the stream key.

That duplication is what makes the field set editable, which is the decisive argument. Checked
both references: VictoriaLogs takes stream fields as a per-request _stream_fields arg and hashes
the tag names into the id, so changing the set mints new ids while old streams keep theirs — and
queries survive because every field is also a regular field. Loki, which has no duplication, calls
changing the default label set "a breaking change for existing users" (docs/sources/shared/otel.md)
and has therefore never shipped the fix it recommends. Without duplication, a key moving out of
identity is postings-only in old parts and column-only in new ones, and no single pushdown covers
both.

Storage layer

• Schema supports several BloomAttrs columns (attrsByteCols, each with a Column.KeyScope).
  Row lookup takes the first hit in declaration order — attrs before resource, so a record
  attribute shadows a resource attribute of the same name — and part pruning survives if any
  attrs bloom may hold the key.
• keys.bin footer v2 adds a scope byte per key (v1 still read, taken as record-scoped).
• New KeyScopeIndexed bit: LogKeys now says which pushdown is sound per key — indexed ⇒ a
  Matcher, record ⇒ a Condition. A key duplicated into the resource column purely because it is
  also the stream key does not get KeyScopeRecord, so the embedder's existing all-or-nothing
  routing keeps working; both bits together still mean "genuinely both, neither pushdown is sound".
• log.Project(ld, fields, emit) takes a classifier called once per stream just before its emit;
  Batch.Identity is the narrowed identity (what the index and WAL store) and Batch.Route the full
  one, so RoutingIdentity() still derives the tenant from every resource attribute.

Cost

The resource column is dictionary-coded and record parts sort by (stream, ts), so a part holds
roughly one entry per stream. In the head it was one copy per record until #153, which brings
BenchmarkWriteLogs ingest B/op to 71.8 MB — below the 73.7 MB pre-change baseline.

Testing

Round-trip and classification tests in signal/log, multi-attrs-column tests in recordengine
(either-blob lookup, shadowing, cross-column bloom pruning, key scopes), and facade tests covering
the collapse, the AllFields opt-out, an excluded key answered as a condition, the LogKeys
routing contract, and a stream-field-set change staying queryable. Full suite green,
golangci-lint clean, ARCHITECTURE.md updated.

Not in this PR

Steps 5-7: the HLL cardinality warning metric, the level low-card value index, and the real-data
validation (started; the migration is what surfaced #152).

🤖 Generated with Claude Code

[github-actions]

📊 Golden benchmarks

Warning

2 benchmarks regressed and 5 improved across the golden set.
Largest regression: logs/merge allocs +95.3%.

_{benchstat base → head · linux/amd64 · AMD Ryzen 5 3600 6-Core Processor · n=6/benchmark · ~ = not significant (p≥0.05)}

⏱️ Time — sec/op (lower is better)

Benchmark	base	head	Δ
write/head	13.24ms	13.45ms	+1.6%	⚪
write/flush	27.88ms	26.69ms	−4.2%	🟢
write/concurrent	21.79ms	21.97ms	+0.8%	⚪
read/fetch_all	2.82ms	2.78ms	−1.3%	⚪
read/fetch_all_release	2.32ms	2.30ms	−0.5%	⚪
read/fetch_recent	1.84ms	1.79ms	−2.9%	🟢
query/promql_count_cpu_cores	700.52µs	687.66µs	−1.8%	🟢
query/promql_full_scan_count	645.61µs	630.80µs	−2.3%	🟢
query/promql_cpu_usage_range	2.22ms	2.19ms	−1.6%	⚪
density	47.48ms	45.37ms	−4.4%	🟢
logs/write_flush	1.76ms	2.33ms	+32.5%	🔴
logs/merge	16.43ms	19.81ms	+20.5%	🔴
geomean			+2.52%

🚀 Throughput — B/s (higher is better)

Benchmark	base	head	Δ
write/head	115.2MiB/s	113.4MiB/s	−1.6%	⚪
write/flush	54.7MiB/s	57.2MiB/s	+4.4%	🟢
write/concurrent	70.0MiB/s	69.5MiB/s	−0.8%	⚪
read/fetch_all	541.4MiB/s	548.5MiB/s	+1.3%	⚪
read/fetch_all_release	659.0MiB/s	662.2MiB/s	+0.5%	⚪
logs/write_flush	80.4MiB/s	60.7MiB/s	−24.5%	🔴
logs/merge	69.0MiB/s	57.2MiB/s	−17.0%	🔴
geomean			-5.97%

🚀 Throughput — Mpoints/s (higher is better)

Benchmark	base	head	Δ
write/head	9.31	9.31	≈0%	⚪
write/flush	3.59	3.75	+4.4%	🟢
write/concurrent	9.74	9.54	−2.1%	⚪
geomean			+0.72%

📦 Bytes per op — B/op (lower is better)

Benchmark	base	head	Δ
write/head	10.4MiB	10.7MiB	+2.5%	⚪
write/flush	47.5MiB	45.0MiB	−5.4%	🟢
write/concurrent	64.7MiB	64.7MiB	≈0%	⚪
read/fetch_all	2.3MiB	2.3MiB	≈0%	⚪
read/fetch_all_release	653.7KiB	653.2KiB	−0.1%	⚪
read/fetch_recent	795.1KiB	794.7KiB	≈0%	🟢
query/promql_count_cpu_cores	416.4KiB	416.6KiB	+0.1%	⚪
query/promql_full_scan_count	408.3KiB	408.6KiB	+0.1%	⚪
query/promql_cpu_usage_range	495.1KiB	494.9KiB	≈0%	⚪
density	76.6MiB	76.6MiB	≈0%	⚪
logs/write_flush	1.5MiB	1.9MiB	+30.7%	🔴
logs/merge	42.1MiB	46.1MiB	+9.5%	🔴
geomean			+2.77%

♻️ Allocations — allocs/op (lower is better)

Benchmark	base	head	Δ
write/head	6.1k	6.0k	−1.4%	⚪
write/flush	30.8k	30.5k	−0.8%	🟢
write/concurrent	9.0k	9.0k	≈0%	⚪
read/fetch_all	4.5k	4.5k	≈0%	⚪
read/fetch_all_release	2.5k	2.5k	≈0%	⚪
read/fetch_recent	4.5k	4.5k	≈0%	⚪
query/promql_count_cpu_cores	1.3k	1.3k	≈0%	⚪
query/promql_full_scan_count	1.3k	1.3k	≈0%	⚪
query/promql_cpu_usage_range	1.7k	1.7k	≈0%	⚪
density	50.1k	50.1k	≈0%	⚪
logs/write_flush	13.0k	17.8k	+37.0%	🔴
logs/merge	33.8k	65.9k	+95.3%	🔴
geomean			+8.34%

🔢 Rows per op — rows/op (informational)

Benchmark	base	head	Δ
read/fetch_all	100.0k	100.0k	≈0%	⚪
read/fetch_all_release	100.0k	100.0k	≈0%	⚪
read/fetch_recent	10.0k	10.0k	≈0%	⚪
geomean			+0.00%

🗜️ Density — B/point (lower is better)

Benchmark	base	head	Δ
density	1.498	1.498	≈0%	⚪
geomean			+0.00%

Raw benchstat table

goos: linux
goarch: amd64
pkg: github.com/oteldb/storage
cpu: AMD Ryzen 5 3600 6-Core Processor              
                                       │   base.txt   │              head.txt               │
                                       │    sec/op    │    sec/op     vs base               │
Golden/write/head-12                     13.24m ± 21%   13.45m ±  3%        ~ (p=0.093 n=6)
Golden/write/flush-12                    27.88m ±  1%   26.69m ±  4%   -4.24% (p=0.004 n=6)
Golden/write/concurrent-12               21.79m ±  8%   21.97m ± 51%        ~ (p=0.818 n=6)
Golden/read/fetch_all-12                 2.818m ±  4%   2.782m ±  5%        ~ (p=0.818 n=6)
Golden/read/fetch_all_release-12         2.315m ±  1%   2.304m ±  3%        ~ (p=0.485 n=6)
Golden/read/fetch_recent-12              1.844m ±  3%   1.791m ±  2%   -2.88% (p=0.004 n=6)
Golden/query/promql_count_cpu_cores-12   700.5µ ±  1%   687.7µ ±  4%   -1.84% (p=0.004 n=6)
Golden/query/promql_full_scan_count-12   645.6µ ±  2%   630.8µ ±  3%   -2.29% (p=0.002 n=6)
Golden/query/promql_cpu_usage_range-12   2.224m ±  2%   2.188m ±  1%        ~ (p=0.132 n=6)
Golden/density-12                        47.48m ±  2%   45.37m ±  3%   -4.43% (p=0.004 n=6)
Golden/logs/write_flush-12               1.762m ±  1%   2.333m ±  3%  +32.46% (p=0.002 n=6)
Golden/logs/merge-12                     16.43m ±  6%   19.81m ± 11%  +20.53% (p=0.002 n=6)
geomean                                  4.756m         4.875m         +2.52%

                                 │   base.txt    │               head.txt                │
                                 │      B/s      │      B/s        vs base               │
Golden/write/head-12               115.2Mi ± 26%   113.4Mi ±   3%        ~ (p=0.093 n=6)
Golden/write/flush-12              54.74Mi ±  1%   57.16Mi ±   4%   +4.43% (p=0.004 n=6)
Golden/write/concurrent-12         70.03Mi ±  9%   69.46Mi ± 104%        ~ (p=0.818 n=6)
Golden/read/fetch_all-12           541.4Mi ±  5%   548.5Mi ±   5%        ~ (p=0.818 n=6)
Golden/read/fetch_all_release-12   659.0Mi ±  1%   662.2Mi ±   3%        ~ (p=0.485 n=6)
Golden/logs/write_flush-12         80.45Mi ±  2%   60.73Mi ±   3%  -24.50% (p=0.002 n=6)
Golden/logs/merge-12               69.00Mi ±  7%   57.24Mi ±  12%  -17.05% (p=0.002 n=6)
geomean                            136.3Mi         128.2Mi          -5.97%

                           │  base.txt  │             head.txt              │
                           │ Mpoints/s  │  Mpoints/s   vs base              │
Golden/write/head-12         9.308 ± 2%   9.306 ±  2%       ~ (p=0.818 n=6)
Golden/write/flush-12        3.587 ± 1%   3.746 ±  4%  +4.43% (p=0.004 n=6)
Golden/write/concurrent-12   9.744 ± 3%   9.536 ± 43%       ~ (p=0.589 n=6)
geomean                      6.878        6.927        +0.72%

                                       │   base.txt    │               head.txt               │
                                       │     B/op      │     B/op       vs base               │
Golden/write/head-12                     10.42Mi ± 46%   10.68Mi ±  2%        ~ (p=0.288 n=6)
Golden/write/flush-12                    47.51Mi ±  3%   44.96Mi ±  6%   -5.37% (p=0.015 n=6)
Golden/write/concurrent-12               64.74Mi ±  7%   64.74Mi ± 64%        ~ (p=0.485 n=6)
Golden/read/fetch_all-12                 2.338Mi ±  0%   2.338Mi ±  0%        ~ (p=1.000 n=6)
Golden/read/fetch_all_release-12         653.7Ki ±  0%   653.2Ki ±  0%        ~ (p=0.128 n=6)
Golden/read/fetch_recent-12              795.1Ki ±  0%   794.7Ki ±  0%   -0.05% (p=0.004 n=6)
Golden/query/promql_count_cpu_cores-12   416.4Ki ±  0%   416.6Ki ±  0%        ~ (p=0.240 n=6)
Golden/query/promql_full_scan_count-12   408.3Ki ±  0%   408.6Ki ±  0%        ~ (p=0.240 n=6)
Golden/query/promql_cpu_usage_range-12   495.1Ki ±  0%   494.9Ki ±  0%        ~ (p=0.394 n=6)
Golden/density-12                        76.57Mi ±  0%   76.57Mi ±  0%        ~ (p=0.515 n=6)
Golden/logs/write_flush-12               1.470Mi ±  0%   1.921Mi ±  0%  +30.66% (p=0.002 n=6)
Golden/logs/merge-12                     42.08Mi ±  5%   46.08Mi ±  7%   +9.50% (p=0.009 n=6)
geomean                                  3.935Mi         4.044Mi         +2.77%

                                       │   base.txt   │               head.txt                │
                                       │  allocs/op   │  allocs/op    vs base                 │
Golden/write/head-12                     6.099k ± 29%   6.016k ±  1%        ~ (p=0.481 n=6)
Golden/write/flush-12                    30.76k ±  1%   30.51k ±  1%   -0.79% (p=0.019 n=6)
Golden/write/concurrent-12               9.048k ±  6%   9.046k ± 46%        ~ (p=0.294 n=6)
Golden/read/fetch_all-12                 4.451k ±  0%   4.451k ±  0%        ~ (p=1.000 n=6) ¹
Golden/read/fetch_all_release-12         2.464k ±  0%   2.464k ±  0%        ~ (p=1.000 n=6) ¹
Golden/read/fetch_recent-12              4.451k ±  0%   4.451k ±  0%        ~ (p=1.000 n=6) ¹
Golden/query/promql_count_cpu_cores-12   1.285k ±  0%   1.285k ±  0%        ~ (p=1.000 n=6)
Golden/query/promql_full_scan_count-12   1.308k ±  0%   1.308k ±  0%        ~ (p=1.000 n=6)
Golden/query/promql_cpu_usage_range-12   1.666k ±  0%   1.665k ±  0%        ~ (p=0.545 n=6)
Golden/density-12                        50.12k ±  0%   50.12k ±  0%        ~ (p=1.000 n=6)
Golden/logs/write_flush-12               13.01k ±  0%   17.82k ±  0%  +36.95% (p=0.002 n=6)
Golden/logs/merge-12                     33.77k ±  0%   65.93k ±  0%  +95.25% (p=0.002 n=6)
geomean                                  6.442k         6.980k         +8.34%
¹ all samples are equal

                                 │  base.txt   │              head.txt               │
                                 │   rows/op   │   rows/op    vs base                │
Golden/read/fetch_all-12           100.0k ± 0%   100.0k ± 0%       ~ (p=1.000 n=6) ¹
Golden/read/fetch_all_release-12   100.0k ± 0%   100.0k ± 0%       ~ (p=1.000 n=6) ¹
Golden/read/fetch_recent-12        10.00k ± 0%   10.00k ± 0%       ~ (p=1.000 n=6) ¹
geomean                            46.42k        46.42k       +0.00%
¹ all samples are equal

                  │  base.txt  │           head.txt            │
                  │  B/point   │  B/point    vs base           │
Golden/density-12   1.498 ± 0%   1.498 ± 0%  ~ (p=1.000 n=6) ¹
¹ all samples are equal

[tdakkota]

Heads-up for review: this carries a real merge/flush regression

The golden-bench on this PR flags it, and it is mechanism-consistent, not noise: the per-record
resource column is stored full-copy per row (interning was rejected — #153), so it flows through
flush and merge as raw bytes.

• logs/merge allocs +95.3%, time +17.6%
• logs/write_flush time +34.1%, throughput −25%
• write/head flat (≈0%) — the cost is the flush/merge re-encode, not the head append

This lands on the merge path that #155 already measures as ~29% of backfill CPU. The root cause is
that every merge decodes a dictionary-encoded column, flattens it, and re-hashes it to rebuild an
equivalent dictionary — filed as #159, whose fix (a dictionary-aware encoder + carrying the dict
through the merge accumulator) removes most of this cost without re-opening the interning decision.

Merging #154 is still the right call — the cardinality/enumeration wins are large and independent
(otelcol-contrib 15,199 → 65 streams; LogSeries 35 → 13 ms) — but the merge cost should be a known,
tracked consequence (#159), not a surprise.

[codecov]

Codecov Report

❌ Patch coverage is 92.99363% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
recordengine/bloom.go	64.70%	3 Missing and 3 partials ⚠️
recordengine/keys.go	93.93%	1 Missing and 1 partial ⚠️
recordengine/recordkeys.go	92.00%	1 Missing and 1 partial ⚠️
recordengine/cols.go	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!