diff --git a/README.md b/README.md index 911c54d..5f05714 100644 --- a/README.md +++ b/README.md @@ -32,8 +32,8 @@ Additional commands: **[Cross-role planning](#cross-role-planning)** and the runnable **[`examples/manifest/`](examples/manifest/)**. - **drift** — detect cross-node schema drift across per-node HCL dumps. -- **locate** — find every declaration site of an object across manifest - layers and per-node dumps; `-duplicates` audits the once-only rule. +- **locate** — find object declaration sites or search resolved models for +table columns; `-duplicates` audits the object once-only rule. - **dump-cluster** — enumerate a cluster's nodes and dump one `.hcl` per node. **dump-sql** — dump a database's CREATE statements as replayable DDL. @@ -756,7 +756,7 @@ in the reference for the manifest format, and **[`examples/manifest/`](examples/manifest/)** for a runnable two-role × three-environment example. -## Locate declarations +## Locate declarations and resolved columns `hclexp locate` answers "where does a name live?" across a manifest's whole layer tree: every declaration site of each matching object @@ -764,7 +764,7 @@ whole layer tree: every declaration site of each matching object raw blocks), with its inheritance markers and the `(role, env)` stacks whose layer lists include each declaring layer. It is query-only: nothing is resolved or diffed. See -**[Locating declarations](docs/README.hcl.md#locating-declarations--hclexp-locate)** +**[Locating declarations and resolved columns](docs/README.hcl.md#locating-declarations-and-resolved-columns--hclexp-locate)** for the full reference. ```sh @@ -778,7 +778,7 @@ hclexp locate -manifest manifest.hcl -layer-root ./schema 'events*' person # Also report which per-node dump files declare it hclexp locate -manifest manifest.hcl -layer-root ./schema -dump prod/eu events -# Find selected columns within selected tables across every dump snapshot +# Resolve every dump snapshot, then find selected table columns in each model hclexp locate -dump prod/eu \ -tables 'flag_evaluations,sharded_flag_evaluations' \ -columns 'person_properties,group*_properties' -format json @@ -812,26 +812,31 @@ or a `[raw ]` block. An object extended by others lists its children Column lookup is a separate selector mode: provide both `-tables` and `-columns` as comma-separated exact names or globs, with no positional object -patterns. It reports authored `column`, `patch_column`, and `modify_column` -blocks, grouped by database, table, and column; dump sites include the snapshot -node. It remains syntax-only, so raw SQL, `drop_columns`, and inherited columns -not written on the selected table are not invented. A valid query with no -matches succeeds and JSON includes `"columns": []`. +patterns. Before searching, locate loads every supplied source into resolved +models: each manifest `(role, env)` layer stack, the complete ordered `-layer` +stack, and each per-node dump file. Independent models load concurrently in a +bounded pool (up to 32 at once), so 30 node dumps are loaded in parallel while +result ordering remains deterministic. Locate then searches the final table +columns in every model, so inheritance, patches, modifications, overrides, and +drops are reflected exactly as resolution produced them. Results are grouped by +database, table, and column and list every matching model, including dump node +identity. A valid query with no matches succeeds and JSON includes +`"columns": []`. **Flags:** - `-manifest` — the same role manifest `plan`/`validate`/`load` consume. - Unlike those commands there is no `-env`: locate scans the union of - every layer named by any `(role, env)` stack and derives placement for - all of them. + Unlike those commands there is no `-env`: object lookup scans the union of + every referenced layer, while column lookup composes and resolves every + `(role, env)` model. - `-layer-root` — root directory the manifest's layer paths resolve under (default `.`) -- `-layer` — comma-separated ad-hoc layer dirs or `.hcl` files to search - too (or instead of a manifest); resolved as given, no placement info, - deduped against the manifest's layers +- `-layer` — comma-separated ad-hoc layer dirs or `.hcl` files. Object lookup + searches them as source layers; column lookup loads the whole list, in the + declared order, as one resolved model. - `-dump` — directory of per-node `.hcl` dumps (as written by - `introspect`/`dump-cluster`); reports which node files also declare - each matching object + `introspect`/`dump-cluster`); object lookup reports declaration sites, while + column lookup loads and resolves every node model concurrently - `-tables` / `-columns` — enter column-selector mode; both are required and accept comma-separated exact names or globs. Table patterns match bare or `database.table`; column patterns match bare, `table.column`, or @@ -839,7 +844,8 @@ matches succeeds and JSON includes `"columns": []`. - `-format` — `text` (default) or `json` (a `{"patterns": [...], "objects": [...]}` / `{"duplicates": [...]}` document with per-site file/line/layer/markers/placements, or a column-selector document containing - `table_patterns`, `column_patterns`, and `columns`) + `table_patterns`, `column_patterns`, and `columns`; every column has its + matching `models`) - `-duplicates` — takes no name argument and requires `-manifest` or `-layer` (mutually exclusive with `-dump` and column selectors); lists every object defined at more than one site and exits non-zero when any is found. Patch sites, diff --git a/cmd/hclexp/hclexp.go b/cmd/hclexp/hclexp.go index 5593d63..58b785c 100644 --- a/cmd/hclexp/hclexp.go +++ b/cmd/hclexp/hclexp.go @@ -105,8 +105,8 @@ Commands: (-format json for structured output) decompose turn per-environment node dumps into shared and env HCL layers, with an exact compose-against-dump round-trip gate - locate find object or table-column declaration sites across manifest - layers and dumps (-duplicates audits the object once-only rule) + locate find object declarations or resolved table columns across + manifests, layer stacks, and dumps (-duplicates audits objects) sql2hcl apply SQL DDL edits (CREATE/ALTER/DROP/RENAME) to an HCL schema load parse and resolve an HCL config, layer stack, or manifest role (default when flags are given) diff --git a/cmd/hclexp/locate.go b/cmd/hclexp/locate.go index 4f17eeb..4b7da13 100644 --- a/cmd/hclexp/locate.go +++ b/cmd/hclexp/locate.go @@ -10,6 +10,7 @@ import ( "path/filepath" "sort" "strings" + "sync" hclload "github.com/posthog/chschema/internal/loader/hcl" ) @@ -49,27 +50,20 @@ type locateDump struct { Type string `json:"type"` } -type locateColumnSite struct { - File string `json:"file"` - Line int `json:"line"` - Layer string `json:"layer,omitempty"` - Type string `json:"type"` - Placements []locatePlacement `json:"placements,omitempty"` -} - -type locateColumnDump struct { - File string `json:"file"` - Line int `json:"line"` - Node string `json:"node"` - Type string `json:"type"` +type locateModelRef struct { + Source string `json:"source"` + File string `json:"file,omitempty"` + Node string `json:"node,omitempty"` + Role string `json:"role,omitempty"` + Env string `json:"env,omitempty"` + Layers []string `json:"layers,omitempty"` } type locateColumn struct { - Database string `json:"database"` - Table string `json:"table"` - Name string `json:"column"` - Declarations []locateColumnSite `json:"declarations,omitempty"` - Dumps []locateColumnDump `json:"dumps,omitempty"` + Database string `json:"database"` + Table string `json:"table"` + Name string `json:"column"` + Models []locateModelRef `json:"models"` } // locateColumnDoc is deliberately separate from the existing object-query @@ -345,6 +339,19 @@ func indexLocateLayers(stacks []locateStack, layerRoot string, extraLayers []str return stacksByLayer, layerOrder } +type locateModel struct { + Ref locateModelRef + Schema *hclload.Schema +} + +type locateModelTask struct { + Ref locateModelRef + Resolved []string + Dump bool +} + +const locateLoadParallelism = 32 + func buildLocateColumnDoc(stacks []locateStack, layerRoot string, extraLayers []string, dumpDir string, tablePatterns, columnPatterns []string) (locateColumnDoc, error) { doc := locateColumnDoc{ TablePatterns: tablePatterns, @@ -353,68 +360,35 @@ func buildLocateColumnDoc(stacks []locateStack, layerRoot string, extraLayers [] } type key struct{ db, table, column string } index := map[key]int{} - upsert := func(d hclload.ColumnDeclaration) *locateColumn { - k := key{d.Database, d.Table, d.Name} + upsert := func(database, table, column string) *locateColumn { + k := key{database, table, column} if i, ok := index[k]; ok { return &doc.Columns[i] } index[k] = len(doc.Columns) - doc.Columns = append(doc.Columns, locateColumn{Database: d.Database, Table: d.Table, Name: d.Name}) + doc.Columns = append(doc.Columns, locateColumn{ + Database: database, Table: table, Name: column, Models: []locateModelRef{}, + }) return &doc.Columns[len(doc.Columns)-1] } - matches := func(d hclload.ColumnDeclaration) bool { - return matchesTablePatterns(tablePatterns, d.Database, d.Table) && matchesColumnPatterns(columnPatterns, d.Database, d.Table, d.Name) - } - stacksByLayer, layerOrder := indexLocateLayers(stacks, layerRoot, extraLayers) - layerByFile := map[string]string{} - for _, layer := range layerOrder { - files, err := hclload.LayerFiles(layer) - if err != nil { - return locateColumnDoc{}, err - } - for _, file := range files { - if _, ok := layerByFile[file]; ok { - continue - } - layerByFile[file] = layer - decls, _, err := hclload.ScanFileColumnDeclarations(file) - if err != nil { - return locateColumnDoc{}, err - } - for _, d := range decls { - if !matches(d) { - continue - } - column := upsert(d) - column.Declarations = append(column.Declarations, locateColumnSite{ - File: d.File, Line: d.Line, Layer: layer, Type: d.Type, - Placements: stacksByLayer[layer], - }) - } - } + models, err := loadLocateModels(stacks, layerRoot, extraLayers, dumpDir) + if err != nil { + return locateColumnDoc{}, err } - - if dumpDir != "" { - files, err := filepath.Glob(filepath.Join(dumpDir, "*.hcl")) - if err != nil { - return locateColumnDoc{}, fmt.Errorf("dump dir %q: %w", dumpDir, err) - } - sort.Strings(files) - for _, file := range files { - decls, node, err := hclload.ScanFileColumnDeclarations(file) - if err != nil { - return locateColumnDoc{}, err - } - if node == "" { - node = strings.TrimSuffix(filepath.Base(file), ".hcl") - } - for _, d := range decls { - if !matches(d) { + for _, model := range models { + for _, database := range model.Schema.Databases { + for _, table := range database.Tables { + if !matchesTablePatterns(tablePatterns, database.Name, table.Name) { continue } - column := upsert(d) - column.Dumps = append(column.Dumps, locateColumnDump{File: d.File, Line: d.Line, Node: node, Type: d.Type}) + for _, column := range table.Columns { + if !matchesColumnPatterns(columnPatterns, database.Name, table.Name, column.Name) { + continue + } + match := upsert(database.Name, table.Name, column.Name) + match.Models = append(match.Models, model.Ref) + } } } } @@ -432,6 +406,125 @@ func buildLocateColumnDoc(stacks []locateStack, layerRoot string, extraLayers [] return doc, nil } +func loadLocateModels(stacks []locateStack, layerRoot string, extraLayers []string, dumpDir string) ([]locateModel, error) { + var tasks []locateModelTask + for _, stack := range stacks { + resolved := make([]string, len(stack.Layers)) + for i, layer := range stack.Layers { + resolved[i] = filepath.Join(layerRoot, layer) + } + tasks = append(tasks, locateModelTask{ + Ref: locateModelRef{ + Source: "manifest", Role: stack.Role, Env: stack.Env, + Layers: append([]string(nil), stack.Layers...), + }, + Resolved: resolved, + }) + } + if len(extraLayers) > 0 { + tasks = append(tasks, locateModelTask{ + Ref: locateModelRef{Source: "layer", Layers: append([]string(nil), extraLayers...)}, + Resolved: append([]string(nil), extraLayers...), + }) + } + if dumpDir != "" { + entries, err := os.ReadDir(dumpDir) + if err != nil { + return nil, fmt.Errorf("read dump dir %q: %w", dumpDir, err) + } + for _, entry := range entries { + if entry.IsDir() || filepath.Ext(entry.Name()) != ".hcl" { + continue + } + file := filepath.Join(dumpDir, entry.Name()) + tasks = append(tasks, locateModelTask{ + Ref: locateModelRef{Source: "dump", File: file}, + Resolved: []string{file}, + Dump: true, + }) + } + } + return loadLocateModelTasks(tasks, locateLoadParallelism, loadLocateModel) +} + +func loadLocateModel(task locateModelTask) (locateModel, error) { + var schema *hclload.Schema + var err error + if task.Dump { + schema, err = loadDumpSchema(task.Ref.File) + } else { + schema, err = hclload.LoadLayers(task.Resolved) + if err == nil { + err = hclload.Resolve(schema) + } + } + if err != nil { + return locateModel{}, fmt.Errorf("load %s: %w", locateModelLabel(task.Ref), err) + } + ref := task.Ref + if task.Dump { + ref.Node = strings.TrimSuffix(filepath.Base(ref.File), ".hcl") + if len(schema.Nodes) > 0 && schema.Nodes[0].Name != "" { + ref.Node = schema.Nodes[0].Name + } + } + return locateModel{Ref: ref, Schema: schema}, nil +} + +func locateModelLabel(ref locateModelRef) string { + switch ref.Source { + case "manifest": + return fmt.Sprintf("manifest model (%s, %s)", ref.Role, ref.Env) + case "layer": + return fmt.Sprintf("layer model %v", ref.Layers) + default: + return fmt.Sprintf("dump model %s", ref.File) + } +} + +type locateModelLoader func(locateModelTask) (locateModel, error) + +// loadLocateModelTasks bounds parallel parsing while preserving task order in +// the result. Thirty dump files therefore load concurrently, but arbitrarily +// large directories cannot create an unbounded number of goroutines. +func loadLocateModelTasks(tasks []locateModelTask, parallelism int, loader locateModelLoader) ([]locateModel, error) { + if len(tasks) == 0 { + return []locateModel{}, nil + } + if parallelism < 1 { + parallelism = 1 + } + if parallelism > len(tasks) { + parallelism = len(tasks) + } + + models := make([]locateModel, len(tasks)) + errs := make([]error, len(tasks)) + jobs := make(chan int, len(tasks)) + for i := range tasks { + jobs <- i + } + close(jobs) + + var wg sync.WaitGroup + wg.Add(parallelism) + for range parallelism { + go func() { + defer wg.Done() + for i := range jobs { + models[i], errs[i] = loader(tasks[i]) + } + }() + } + wg.Wait() + for _, err := range errs { + if err != nil { + return nil, err + } + } + return models, nil +} + func matchesTablePatterns(patterns []string, database, table string) bool { for _, pattern := range patterns { if hclload.MatchesPattern(pattern, database, table) { @@ -576,18 +669,15 @@ func renderLocateText(w io.Writer, doc locateDoc) { func renderLocateColumnText(w io.Writer, doc locateColumnDoc) { for _, column := range doc.Columns { fmt.Fprintf(w, "column %s.%s.%s\n", column.Database, column.Table, column.Name) - for _, d := range column.Declarations { - marker := "" - if d.Type != "column" { - marker = " [" + d.Type + "]" + for _, model := range column.Models { + switch model.Source { + case "manifest": + fmt.Fprintf(w, " manifest: (%s, %s) layers: %s\n", model.Role, model.Env, strings.Join(model.Layers, ",")) + case "layer": + fmt.Fprintf(w, " layers: %s\n", strings.Join(model.Layers, ",")) + case "dump": + fmt.Fprintf(w, " dump: %s (node %s)\n", model.File, model.Node) } - fmt.Fprintf(w, " %s:%d%s\n", d.File, d.Line, marker) - if len(d.Placements) > 0 { - fmt.Fprintf(w, " %s\n", formatPlacements(d.Placements)) - } - } - for _, d := range column.Dumps { - fmt.Fprintf(w, " dump: %s:%d (node %s)\n", d.File, d.Line, d.Node) } } } @@ -606,15 +696,16 @@ func renderDuplicatesText(w io.Writer, doc locateDoc) { // runLocate answers "where is object X declared?" across a manifest's layer // tree, ad-hoc -layer entries, and/or a dump directory, or (with // -duplicates) audits the layer tree for objects defined at more than one -// site. Column selector mode returns success with an empty result. Object -// pattern mode exits 1 when any pattern matches nothing; duplicate mode exits -// 1 when duplicates exist. Usage errors exit 2. +// site. Column selector mode loads resolved models before searching and +// returns success with an empty result. Object pattern mode exits 1 when any +// pattern matches nothing; duplicate mode exits 1 when duplicates exist. +// Usage errors exit 2. func runLocate(args []string) { fs := flag.NewFlagSet("hclexp locate", flag.ExitOnError) - manifestFlag := fs.String("manifest", "", "HCL manifest: role blocks with env blocks; every (role, env) stack is searched") + manifestFlag := fs.String("manifest", "", "HCL manifest: object mode scans its layers; column mode resolves every (role, env) stack") layerRootFlag := fs.String("layer-root", ".", "root directory the manifest's layer paths resolve under") - layersFlag := fs.String("layer", "", "comma-separated ad-hoc layer dirs or .hcl files to search too (or instead of a manifest); no placement info") - dumpFlag := fs.String("dump", "", "directory of per-node .hcl dumps to search as well") + layersFlag := fs.String("layer", "", "comma-separated ad-hoc layer dirs or .hcl files; column mode resolves them in order as one model") + dumpFlag := fs.String("dump", "", "directory of per-node .hcl dumps; column mode resolves node models concurrently") formatFlag := fs.String("format", "text", "output format: text (default) or json") duplicatesFlag := fs.Bool("duplicates", false, "list every object defined at more than one site (patch/override/extend sites refine; abstracts define); takes no name argument") tablesFlag := fs.String("tables", "", "comma-separated table names or globs for column lookup; requires -columns and no name argument") diff --git a/cmd/hclexp/locate_test.go b/cmd/hclexp/locate_test.go index f562a8d..f76f3af 100644 --- a/cmd/hclexp/locate_test.go +++ b/cmd/hclexp/locate_test.go @@ -3,11 +3,14 @@ package main import ( "bytes" "encoding/json" + "fmt" "os" "os/exec" "path/filepath" "testing" + "time" + hclload "github.com/posthog/chschema/internal/loader/hcl" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -48,7 +51,7 @@ database "posthog" { column "uuid" { type = "UUID" } } table "person" { - engine "MergeTree" {} + engine "merge_tree" {} order_by = ["id"] column "id" { type = "UInt64" } } @@ -58,7 +61,7 @@ database "posthog" { database "posthog" { table "events" { extend = "events_base" - engine "MergeTree" {} + engine "merge_tree" {} order_by = ["uuid"] } } @@ -66,7 +69,7 @@ database "posthog" { mustWrite("aux/dup.hcl", ` database "posthog" { table "person" { - engine "MergeTree" {} + engine "merge_tree" {} order_by = ["id"] column "id" { type = "UInt64" } } @@ -76,7 +79,7 @@ database "posthog" { node "node1" {} database "posthog" { table "events" { - engine "MergeTree" {} + engine "merge_tree" {} order_by = ["uuid"] column "uuid" { type = "UUID" } } @@ -85,7 +88,7 @@ database "posthog" { mustWrite("dumps/node2.hcl", ` database "posthog" { table "only_live" { - engine "MergeTree" {} + engine "merge_tree" {} order_by = ["id"] column "id" { type = "UInt64" } } @@ -223,36 +226,60 @@ func TestBuildLocateDocMultiplePatterns(t *testing.T) { assert.Equal(t, []string{"person", "only_*", "nosuch*"}, doc.Patterns) } -func TestBuildLocateColumnDocFindsLayerAndDumpSites(t *testing.T) { +func TestBuildLocateColumnDocSearchesResolvedManifestAndDumpModels(t *testing.T) { root := locateTree(t) stacks, err := parseManifestAllEnvs(filepath.Join(root, "manifest.hcl")) require.NoError(t, err) - doc, err := buildLocateColumnDoc(stacks, root, nil, filepath.Join(root, "dumps"), - []string{"events*"}, []string{"uuid"}) + // The aux stack deliberately redeclares person for the object-duplicate + // tests. The two ingestion stacks are valid resolved models. + doc, err := buildLocateColumnDoc(stacks[:2], root, nil, filepath.Join(root, "dumps"), + []string{"events"}, []string{"uuid"}) require.NoError(t, err) - assert.Equal(t, []string{"events*"}, doc.TablePatterns) + assert.Equal(t, []string{"events"}, doc.TablePatterns) assert.Equal(t, []string{"uuid"}, doc.ColumnPatterns) - require.Len(t, doc.Columns, 2) + require.Len(t, doc.Columns, 1) events := doc.Columns[0] assert.Equal(t, "posthog", events.Database) assert.Equal(t, "events", events.Table) assert.Equal(t, "uuid", events.Name) - assert.Empty(t, events.Declarations, "inherited columns are not invented by syntax-only locate") - assert.Equal(t, []locateColumnDump{{ - File: filepath.Join(root, "dumps", "node1.hcl"), Line: 7, Node: "node1", Type: "column", - }}, events.Dumps) - - base := doc.Columns[1] - assert.Equal(t, "events_base", base.Table) - require.Len(t, base.Declarations, 1) - assert.Equal(t, filepath.Join(root, "shared", "base.hcl"), base.Declarations[0].File) - assert.Equal(t, []locatePlacement{ - {Role: "ingestion", Env: "prod-us"}, - {Role: "ingestion", Env: "prod-eu"}, - {Role: "aux", Env: "prod-us"}, - }, base.Declarations[0].Placements) + assert.Equal(t, []locateModelRef{ + {Source: "manifest", Role: "ingestion", Env: "prod-us", Layers: []string{"shared", "ingestion"}}, + {Source: "manifest", Role: "ingestion", Env: "prod-eu", Layers: []string{"shared", "ingestion"}}, + {Source: "dump", File: filepath.Join(root, "dumps", "node1.hcl"), Node: "node1"}, + }, events.Models, "uuid is inherited into both composed manifest models and loaded from the dump model") +} + +func TestBuildLocateColumnDocSearchesResolvedLayerStack(t *testing.T) { + root := t.TempDir() + base := writeLocateLayer(t, root, "base.hcl", ` +database "posthog" { + table "events_base" { + abstract = true + column "uuid" { type = "UUID" } + } + table "events" { + extend = "events_base" + engine "log" {} + } +}`) + patch := writeLocateLayer(t, root, "patch.hcl", ` +database "posthog" { + patch_table "events" { + column "person_properties" { type = "String" } + } +}`) + + doc, err := buildLocateColumnDoc(nil, "", []string{base, patch}, "", + []string{"events"}, []string{"uuid", "person_properties"}) + require.NoError(t, err) + require.Len(t, doc.Columns, 2) + assert.Equal(t, "person_properties", doc.Columns[0].Name) + assert.Equal(t, "uuid", doc.Columns[1].Name) + for _, column := range doc.Columns { + assert.Equal(t, []locateModelRef{{Source: "layer", Layers: []string{base, patch}}}, column.Models) + } } func TestBuildLocateColumnDocNoMatchIsEmpty(t *testing.T) { @@ -393,20 +420,64 @@ func TestRenderLocateColumnText(t *testing.T) { Database: "posthog", Table: "flag_evaluations", Name: "person_properties", - Declarations: []locateColumnSite{{ - File: "schema/flags.hcl", Line: 12, Type: "modify_column", - Placements: []locatePlacement{{Role: "ingestion", Env: "prod-us"}}, - }}, - Dumps: []locateColumnDump{{File: "prod-us/node1.hcl", Line: 20, Node: "node1", Type: "column"}}, + Models: []locateModelRef{ + {Source: "manifest", Role: "ingestion", Env: "prod-us", Layers: []string{"shared", "prod-us"}}, + {Source: "layer", Layers: []string{"shared", "local"}}, + {Source: "dump", File: "prod-us/node1.hcl", Node: "node1"}, + }, }}} var buf bytes.Buffer renderLocateColumnText(&buf, doc) out := buf.String() assert.Contains(t, out, "column posthog.flag_evaluations.person_properties") - assert.Contains(t, out, "schema/flags.hcl:12 [modify_column]") - assert.Contains(t, out, "(ingestion, prod-us)") - assert.Contains(t, out, "dump: prod-us/node1.hcl:20 (node node1)") + assert.Contains(t, out, "manifest: (ingestion, prod-us) layers: shared,prod-us") + assert.Contains(t, out, "layers: shared,local") + assert.Contains(t, out, "dump: prod-us/node1.hcl (node node1)") +} + +func TestLoadLocateModelTasksLoadsThirtyNodesInParallel(t *testing.T) { + tasks := make([]locateModelTask, 30) + for i := range tasks { + tasks[i].Ref = locateModelRef{Source: "dump", File: fmt.Sprintf("node-%02d.hcl", i)} + } + started := make(chan struct{}, len(tasks)) + release := make(chan struct{}) + released := false + defer func() { + if !released { + close(release) + } + }() + type result struct { + models []locateModel + err error + } + done := make(chan result, 1) + go func() { + models, err := loadLocateModelTasks(tasks, locateLoadParallelism, func(task locateModelTask) (locateModel, error) { + started <- struct{}{} + <-release + return locateModel{Ref: task.Ref, Schema: &hclload.Schema{}}, nil + }) + done <- result{models: models, err: err} + }() + + for range tasks { + select { + case <-started: + case <-time.After(time.Second): + require.FailNow(t, "all 30 loaders did not start concurrently") + } + } + close(release) + released = true + got := <-done + require.NoError(t, got.err) + require.Len(t, got.models, 30) + for i := range got.models { + assert.Equal(t, tasks[i].Ref, got.models[i].Ref, "parallel results preserve deterministic task order") + } } func TestLocateColumnsCLIProcess(t *testing.T) { @@ -437,12 +508,15 @@ database "posthog" { table "flag_evaluations" { column "person_properties" { type = "String" } column "group0_properties" { type = "String" } + engine "log" {} } table "sharded_flag_evaluations" { column "group4_properties" { type = "String" } + engine "log" {} } table "not_selected" { column "person_properties" { type = "String" } + engine "log" {} } } `) @@ -451,13 +525,16 @@ node "node-b" {} database "posthog" { table "flag_evaluations" { column "person_properties" { type = "String" } + engine "log" {} } table "writable_flag_evaluations" { column "group2_properties" { type = "String" } + engine "log" {} } table "kafka_flag_evaluations" { column "group3_properties" { type = "String" } column "event" { type = "String" } + engine "log" {} } } `) @@ -472,7 +549,7 @@ database "posthog" { assert.Equal(t, "flag_evaluations", doc.Columns[0].Table) assert.Equal(t, "group0_properties", doc.Columns[0].Name) assert.Equal(t, "person_properties", doc.Columns[1].Name) - assert.Equal(t, []string{"node-a", "node-b"}, []string{doc.Columns[1].Dumps[0].Node, doc.Columns[1].Dumps[1].Node}) + assert.Equal(t, []string{"node-a", "node-b"}, []string{doc.Columns[1].Models[0].Node, doc.Columns[1].Models[1].Node}) assert.Equal(t, "kafka_flag_evaluations", doc.Columns[2].Table) assert.Equal(t, "sharded_flag_evaluations", doc.Columns[3].Table) assert.Equal(t, "writable_flag_evaluations", doc.Columns[4].Table) @@ -494,6 +571,36 @@ database "posthog" { assert.Equal(t, 1, exitErr.ExitCode(), string(failed)) } +func TestLocateColumnsCLIThirtyNodesEndToEnd(t *testing.T) { + dumps := t.TempDir() + for i := range 30 { + node := fmt.Sprintf("node-%02d", i) + body := fmt.Sprintf(` +node %q {} +database "posthog" { + table "events" { + column "uuid" { type = "UUID" } + engine "log" {} + } +} +`, node) + require.NoError(t, os.WriteFile(filepath.Join(dumps, node+".hcl"), []byte(body), 0o600)) + } + + output, err := runLocateColumnsCLI(t, "-dump", dumps, "-tables", "events", "-columns", "uuid", "-format", "json") + require.NoError(t, err, string(output)) + var doc locateColumnDoc + require.NoError(t, json.Unmarshal(output, &doc), string(output)) + require.Len(t, doc.Columns, 1) + require.Len(t, doc.Columns[0].Models, 30) + for i, model := range doc.Columns[0].Models { + expectedNode := fmt.Sprintf("node-%02d", i) + assert.Equal(t, "dump", model.Source) + assert.Equal(t, expectedNode, model.Node) + assert.Equal(t, filepath.Join(dumps, expectedNode+".hcl"), model.File) + } +} + func runLocateColumnsCLI(t *testing.T, args ...string) ([]byte, error) { t.Helper() commandArgs := append([]string{"-test.run=^TestLocateColumnsCLIProcess$", "--"}, args...) @@ -501,3 +608,10 @@ func runLocateColumnsCLI(t *testing.T, args ...string) ([]byte, error) { cmd.Env = append(os.Environ(), "HCLEXP_LOCATE_COLUMNS_HELPER=1") return cmd.CombinedOutput() } + +func writeLocateLayer(t *testing.T, root, name, body string) string { + t.Helper() + path := filepath.Join(root, name) + require.NoError(t, os.WriteFile(path, []byte(body), 0o600)) + return path +} diff --git a/docs/README.hcl.md b/docs/README.hcl.md index 4eaab15..83827a7 100644 --- a/docs/README.hcl.md +++ b/docs/README.hcl.md @@ -1049,7 +1049,7 @@ A proposed table name that already exists live as an unmanaged object is a collision, not an implicit adoption: the exact CREATE should fail until the ownership conflict is resolved explicitly. -## Locating declarations — `hclexp locate` +## Locating declarations and resolved columns — `hclexp locate` `locate` answers two questions the layer tree makes hard to grep for: *where is object X declared?* and *is X declared more than once?* It is @@ -1066,7 +1066,7 @@ hclexp locate -manifest manifest.hcl -layer-root ./schema 'posthog.person_*' eve # Also search per-node dumps (introspect / dump-cluster output) hclexp locate -manifest manifest.hcl -layer-root ./schema -dump ./dumps events -# Search several table/column names in every per-node snapshot at once +# Resolve every per-node snapshot, then search its final table columns hclexp locate -dump ./prod-us \ -tables 'flag_evaluations,sharded_flag_evaluations,writable_flag_evaluations,kafka_flag_evaluations' \ -columns 'person_properties,group0_properties,group1_properties,group2_properties,group3_properties,group4_properties' \ @@ -1099,11 +1099,22 @@ block, else the filename stem — the same identity `drift` uses). Column lookup uses both `-tables` and `-columns`; each flag is a comma-separated list of exact names or globs. Table selectors accept bare names and `database.table`; column selectors accept bare names, `table.column`, and -`database.table.column`. Results are grouped by database/table/column and list -every authored layer site plus every matching dump file and node. Ordinary -`column`, child-local `patch_column`, and cross-layer `modify_column` blocks are -reported. The scan deliberately does not resolve inheritance or interpret raw -SQL/drop lists, preserving `locate`'s source-site semantics. +`database.table.column`. + +Selector mode is model-first. It loads and resolves each manifest `(role, env)` +stack, loads the entire comma-separated `-layer` list in declared order as one +model, and loads every `.hcl` file under `-dump` as an independent node model. +These independent models are loaded concurrently through a bounded pool of up +to 32 workers: a 30-node dump can start all 30 loads in parallel without making +larger clusters create an unbounded number of goroutines. Results retain stable +source order regardless of completion order. + +Only after loading does locate search each resolved model's final tables and +columns. Inherited columns and the effects of `patch_table`, `patch_column`, +`modify_column`, `override`, and drop operations are therefore included in the +result instead of being reconstructed from syntax. Raw objects are not tables +in the resolved model. Results are grouped by database/table/column; each entry +lists the matching manifest model, ordered layer model, or dump file and node. Selector mode cannot be combined with positional object patterns or `-duplicates`, and both selector flags are required. A valid query with no diff --git a/internal/loader/hcl/locate.go b/internal/loader/hcl/locate.go index 38af667..4eca7b2 100644 --- a/internal/loader/hcl/locate.go +++ b/internal/loader/hcl/locate.go @@ -29,19 +29,6 @@ type Declaration struct { RawKind string // raw blocks only: the kind label } -// ColumnDeclaration is one authored column site within a table or patch_table -// block. Type is column, patch_column, or modify_column. Drop lists are not -// declarations and raw SQL remains opaque, matching locate's syntax-only -// object discovery. -type ColumnDeclaration struct { - Database string - Table string - Name string - File string - Line int - Type string -} - // ScanDeclarations records every object declaration site in the given files, // in file order. Files must be native HCL syntax, same as the loader; a // parse error aborts the scan. @@ -104,70 +91,6 @@ func ScanFileDeclarations(path string) ([]Declaration, string, error) { return out, node, nil } -// ScanFileColumnDeclarations returns every authored table-column site in one -// native HCL file and the first node{} label for dump attribution. It does not -// resolve extend chains or patches: callers see the source sites as written. -func ScanFileColumnDeclarations(path string) ([]ColumnDeclaration, string, error) { - parser := hclparse.NewParser() - f, diags := parser.ParseHCLFile(path) - if diags.HasErrors() { - return nil, "", formatDiagnostics(parser, diags) - } - body, ok := f.Body.(*hclsyntax.Body) - if !ok { - return nil, "", fmt.Errorf("%s: not native HCL syntax", path) - } - - var out []ColumnDeclaration - node := "" - for _, blk := range body.Blocks { - switch blk.Type { - case "database": - if len(blk.Labels) != 1 { - continue - } - for _, table := range blk.Body.Blocks { - out = append(out, columnDeclarations(table, blk.Labels[0], path)...) - } - case "node": - if node == "" && len(blk.Labels) == 1 { - node = blk.Labels[0] - } - } - } - return out, node, nil -} - -func columnDeclarations(table *hclsyntax.Block, database, path string) []ColumnDeclaration { - if len(table.Labels) != 1 || (table.Type != "table" && table.Type != "patch_table") { - return nil - } - var out []ColumnDeclaration - for _, child := range table.Body.Blocks { - if len(child.Labels) != 1 { - continue - } - valid := child.Type == "column" - if table.Type == "table" { - valid = valid || child.Type == "patch_column" - } else { - valid = valid || child.Type == "modify_column" - } - if !valid { - continue - } - out = append(out, ColumnDeclaration{ - Database: database, - Table: table.Labels[0], - Name: child.Labels[0], - File: path, - Line: child.DefRange().Start.Line, - Type: child.Type, - }) - } - return out -} - // objectDeclaration converts one block nested in a database{} into a // Declaration. Blocks that are not object declarations (node, column, // unknown types) report ok = false. diff --git a/internal/loader/hcl/locate_test.go b/internal/loader/hcl/locate_test.go index efb2c7a..6ac043e 100644 --- a/internal/loader/hcl/locate_test.go +++ b/internal/loader/hcl/locate_test.go @@ -158,37 +158,6 @@ database "posthog" { assert.Empty(t, node) } -func TestScanFileColumnDeclarations(t *testing.T) { - dir := t.TempDir() - path := writeHCL(t, dir, "columns.hcl", ` -node "prod-ch-1a" {} -database "posthog" { - table "events" { - column "uuid" { type = "UUID" } - patch_column "properties" { codec = "ZSTD(1)" } - } - patch_table "events" { - modify_column "uuid" { type = "String" } - column "person_properties" { type = "String" } - drop_columns = ["obsolete"] - } - materialized_view "events_mv" { - column "ignored" { type = "UInt64" } - } -} -`) - - decls, node, err := ScanFileColumnDeclarations(path) - require.NoError(t, err) - assert.Equal(t, "prod-ch-1a", node) - assert.Equal(t, []ColumnDeclaration{ - {Database: "posthog", Table: "events", Name: "uuid", File: path, Line: 5, Type: "column"}, - {Database: "posthog", Table: "events", Name: "properties", File: path, Line: 6, Type: "patch_column"}, - {Database: "posthog", Table: "events", Name: "uuid", File: path, Line: 9, Type: "modify_column"}, - {Database: "posthog", Table: "events", Name: "person_properties", File: path, Line: 10, Type: "column"}, - }, decls) -} - func TestMatchesPattern(t *testing.T) { assert.True(t, MatchesPattern("events", "posthog", "events")) assert.True(t, MatchesPattern("events*", "posthog", "events_mv"))