diff --git a/CHANGELOG.md b/CHANGELOG.md index b8f5c66..f4de5d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,34 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ## [Unreleased] +## [0.49.0] - 2026-10-20 + +### Added + +- **Environments that differ in more than values** (docs/existing-environments.md, decisions D1–D6). + Taking on an existing system with production and QA trees that have grown apart: + - **Which environments get what:** `environments` and `planned` on drivers (`driver.xml`), jobs, + role-based entitlement policies and templates. Compare, plan and deploy leave out what an + environment doesn't get; `planned` tells "not yet" from "never". `driver.set environments|planned`, + `environments.set` for several objects at once. + - **Names per environment:** `drivers/.name` in an environment's overrides file. Compare, plan + and deploy use that name; imports map it back; secrets answer to either name. + - **Servers per environment:** `servers. = `, several onto one, + `.prefer` to choose whose settings a shared server takes; validation `server-unmapped`. + - **Jobs per environment:** schedule, disabled and trace settings as override keys. + - **A new driver's server** chosen at deploy: `--driver-server "D="`. + - **Two environments compared:** `vault.compare-envs`: one-sided objects, values (expected or + not, with the tree's value), likely renames (feeds flagged), drift with likely causes, + case-only DN differences apart. + - **Known differences:** `overrides/.known.json`, `known.accept` / `known.forget`, matched by + fingerprint so a changed object shows again. + - **Import:** keeps the tree's drivers an environment doesn't get; `--mark-new-drivers`. + +### Changed + +- `override-missing-env` no longer warns about server maps and driver names, which belong to the + environments that need them. + ## [0.48.0] - 2026-10-19 ### Fixed diff --git a/docs/existing-environments.md b/docs/existing-environments.md new file mode 100644 index 0000000..a639f2c --- /dev/null +++ b/docs/existing-environments.md @@ -0,0 +1,191 @@ +# Taking on existing environments + +*Status: decided 2026-10-20; being built on the `driver-environments` branches (core and web).* + +The usual case is not a clean start. A customer has a production tree and one or more others (QA, +test, dev), each changed by hand over the years: drivers added in one and not the other, the same +driver under two names, hot fixes made in production and never carried back, work in QA never +promoted, host names typed into policies. The tool has to take that on without asking anyone to +clean it up first, and it has to help decide what genuinely needs resolving. + +## What a real pair of environments looks like + +A customer's QA and production driver sets, read from two LDIF exports (2026-10-20): + +| | Production | QA | +|---|---|---| +| objects in the driver set | 1,143 | 1,542 | +| drivers | 41 | 43 | +| servers | 4 | 2 | +| driver set DN | `…,ou=SERVICES,o=…` | `…,ou=services,o=…` | + +Compared driver by driver (QA against production, production as the tree): + +- **Drivers in one only:** 7 in production only, 9 in QA only. Among them one is plainly the same + driver renamed (20 of 20 policies in common, names one character apart); one production driver has + four QA counterparts (one QA certificate-revocation driver per CA where production has one); and two + drivers look alike because each pushes to the environment below (production→QA, QA→dev), which only + a person can call the same driver or not. +- **Values:** 81 scalar values differ: 69 shim parameters, 5 engine control values, 4 driver GCVs, 3 + driver-set GCVs. These are what overrides are for. +- **Everything else:** 879 differences. 52 differ only in letter case (`OU=services` against + `OU=SERVICES`; eDirectory reads them as the same) and are set aside. The other 827: + +| | count | what it suggests | +|---|---:|---| +| only in QA | 402 | QA-only objects; 350 are role-based entitlement policies | +| only in production | 57 | production has them, QA never got them | +| lines only production has | 37 | a change made in production, not carried back | +| lines only QA has | 26 | work in QA not yet promoted | +| package versions differ | 13 | the newer side is known | +| cosmetic (descriptions, comments, display names) | 6 | align freely | +| environment text inside content | 7 | a changed line names an environment or server: if it is a literal in a policy, it belongs in a GCV | +| both sides changed | 268 | needs a person; mostly User Application objects and PRDs | + +Two more things the data shows. Jobs differ by schedule (production runs them on a schedule, QA +doesn't), which an override can't carry yet. And jobs, the driver set's server list and per-server +settings name servers by DN. Tree names and server names are globally unique, so two environments +never share a server: every environment but the base needs its servers mapped, here four production +servers onto two in QA. + +## The process + +One environment at a time, the production tree first. + +1. **Base.** Import production into `main`. `main` is what production runs; everything any other + environment has beyond it shows up as a change to look at, not as something already on its way to + production. *(built)* +2. **Connect the next environment** under Connections, and add it to the project. +3. **Set up that environment**: a guided read (*Vault → Environments → Set up QA…*, proposed) that + proposes, for review, everything it needs. Nothing is written until you apply; you commit the + result. + 1. **Match drivers.** Same name: matched. Otherwise a likely match by shim class, name and policies + in common, which you confirm (a `drivers/.name` override) or reject. Proposals are one-to-one; + one production driver is never matched to two. *(matching proposed; the name override is built)* + 2. **Membership.** A driver only that environment has is marked as its own + (`environments="QA"` in `driver.xml`); a tree driver that environment lacks is marked as not + deployed there. *(built: the marks, compare, deploy and update follow them)* + 3. **Values.** Each differing value becomes that environment's override, ticked by default (the + tree holds production's value). *(built: recorded one at a time from the Environments tab; bulk + proposed)* + 4. **Servers.** Map the tree's servers to that environment's (several to one allowed), so jobs and + per-server settings line up. Required for every environment but the base, since no two + environments share a server. *(proposed: see "Servers")* + 5. **What's left: analyse, then decide** (next section). +4. **Compare again.** What remains is what you decided to leave, and from then on only new + differences show. + +## Analysing what's left + +Each remaining difference gets a likely cause from the comparison itself, as in the table above, and +the page groups them by driver with counts, cause first, so that 827 differences read as a short list +of drivers with a few kinds of problem each. For each you choose: + +- **Production is right:** leave the tree as it is; a deploy to QA (when you choose to) brings QA in + line. Suggested for *only in production*, *lines only production has* and an older package in QA. +- **QA is ahead:** take QA's version into a branch (*Update this branch from QA…*, ticking those + files), to promote through the normal flow. Suggested for *lines only QA has* and a newer package in + QA. +- **Needs fixing:** both sides wrong or a design problem, such as a host name typed into a policy. + Opens the file; the suggestion names the fix (move the literal into a GCV, then override it). +- **Different on purpose:** this environment's content genuinely differs and no value can carry it. + Recorded as known (**D1**). +- **Decide later:** recorded as known, with who and when, so it stops hiding new differences but + stays listed. + +What's known about a driver helps decide. When the repository holds notes on a driver (a summary, +known defects; for example `docs/drivers/.md` from an earlier analysis), the setup shows them +beside that driver's differences. In the customer data above, production carries an extra rule that +vetoes every user event on one driver's subscriber channel, and QA doesn't. Whether that is a stopgap +for a defect the earlier analysis found is the kind of question only such context can help answer. + +Resolving is then ordinary work on branches: each decision either leaves the tree, changes it on a +branch, or records the difference as known. + +## Drivers only some environments get: for now, or for good + +An item that only some environments get may be waiting (built in QA, not yet promoted to production) or +permanent (a test loopback, a feed that only exists in QA). The tree says which, beside the list: + +```xml + + +``` + +The same two attributes go on jobs, role-based entitlement policies and notification templates. A +compare against production lists the planned ones as waiting for promotion and leaves the permanent ones +quiet; promoting moves the environment from `planned` to `environments`, one reviewed change; a plan for +production names what is planned for it and not promoted yet. + +## Feeds between environments + +Keeping a lower environment's data current from production is common: an eDirectory-to-eDirectory link +whose sending end is a driver in production (for example "VaultProdToVaultQA") and whose receiving end is +a driver in QA ("VaultQAToVaultProd"); QA may feed dev the same way ("VaultQAToVaultDev"). Each end is its +own driver. Two ways to keep them: + +- **Each end its environment's, for good** (`environments="PROD"`, `environments="QA"`): the simple + default, and right when the ends differ in more than where they point. +- **One tree driver per role, named per environment**: "send to the environment below" is the same + configuration in production (VaultProdToVaultQA) and QA (VaultQAToVaultDev), aimed at a different tree. + One tree driver with `drivers/.name` per environment and the remote tree's address as each + environment's override keeps a fix to one end in step with the other. + +Names that carry environment names, and a remote address that is another environment of the project, are +how the setup recognises a feed; it proposes, and a person decides. Keep a feed's remote address an +override in every environment that gets the driver, so a deploy never points one environment's feed at +another's target. (A validation check for this is a possible follow-up; the setup flags feeds today.) + +## Known differences (D1) + +A file per environment beside the overrides, `overrides/.known.json`, listing differences +accepted as they are: the object's path, a fingerprint of both sides, the decision, who and when. A +compare shows them collapsed under "known" and shows anything whose fingerprint changed as new. A +deploy to that environment still writes what the tree holds; the plan says which of its changes +overwrite a known difference, so nobody flattens one by accident. + +## Servers + +Server names are globally unique, so the server DNs in the tree (the driver set's server list, each +driver's `servers//` settings, jobs' servers) are the base environment's. Every other +environment maps them in its overrides file, one line per tree server. The key names the tree's server by +its name (the `servers//` folder's), since the file splits a line at its first `=`; the value is the +environment's server's DN: + +``` +# overrides/QA.properties +servers.vault01 = cn=vault01qa,ou=servers,o=acme +servers.vault02 = cn=vault01qa,ou=servers,o=acme +servers.vault03 = cn=vault02qa,ou=servers,o=acme +servers.vault04 = cn=vault02qa,ou=servers,o=acme +``` + +The tree's servers act as roles. Compare, plan and deploy against QA read and write each role on the +server it maps to; an import from QA maps the other way. Two roles mapped to one server must agree +on that server's settings, or the setup flags the conflict. A server of QA that no role maps to is +reported, as is a role left unmapped (validation: `server-unmapped`). The guided setup proposes the +map by name and by which drivers run where, for you to confirm. + +## Decisions (taken 2026-10-20) + +- **D1 — Known differences: recorded** per environment in `overrides/.known.json`, as above. +- **D2 — Driver-set objects get an environments list like drivers:** role-based entitlement + policies, notification templates and driver-set jobs. Compare and deploy leave them out of other + environments; an update from an environment marks the ones new to the tree as its own. +- **D3 — Servers: required, mapped in each environment's overrides file** (`servers. = + `). Fewer servers outside production is the normal case, so several tree servers mapping + to one is the rule. A driver whose settings differ between two tree servers that map to the same + server is flagged, and the person picks which that server takes + (`servers..prefer = `). +- **D4 — Job schedules, enabled state and trace are override keys:** `drivers/.job..schedule` + (empty = no schedule), `.disabled`, `.trace-level`, `.trace-file`, `.trace-size-limit`, and + `driverset.job..…` for the set's jobs. The comparison lists them as values. In the customer data, + two production jobs run on a schedule QA's don't, and job trace levels and files differ. +- **D6 — A new driver's server, chosen at deploy:** promoting a driver to an environment with several + servers asks which one runs it (default: the one the deploy connects through). That server gets the + driver's start option, manual, and every other server of the set gets disabled + (`vault.deploy --driver-server "D="`; *Runs on* in the plan). A deploy choice, not a tree value: + it matters only when the driver is created there. The step waits for the driver to replicate to that + server first. +- **D5 — A guided *Set up …* flow** proposing 3.1–3.5 together, reviewed before anything + is written, then committed. diff --git a/docs/tree-layout.md b/docs/tree-layout.md index f37d817..2acf033 100644 --- a/docs/tree-layout.md +++ b/docs/tree-layout.md @@ -85,6 +85,29 @@ connection settings (`drivers/.gcv.`, `.ecv.`, files keep the base value; a deploy to that environment applies its file. See [getting-started.md](getting-started.md) §3.1. +What else differs per environment goes in the same file +([existing-environments.md](existing-environments.md)): + +- `drivers/.name = `: that environment's vault calls the driver + something else; compare, plan and deploy use that name, an import maps it + back. +- `servers. = `: one line + per server of the tree, several onto one where the environment has fewer; + `servers..prefer = ` picks whose + per-server settings a shared server takes. Required for every environment + but the base: no two environments share a server. +- `drivers/.job..schedule` (empty = none), `.disabled`, + `.trace-level`, `.trace-file`, `.trace-size-limit`, and + `driverset.job..…`: a job's schedule, state and trace. + +### `overrides/.known.json` + +Differences between two environments accepted as they are: each entry names +the other environment, the object's path, a fingerprint of the change, the +decision (`intended` or `later`), a note, who and when. A comparison leaves an +entry's difference out while the fingerprint matches, and shows it again once +the object changes. Written by `known.accept` / `known.forget`. + ### `library/` Policies, ECMAScript, mapping tables, and GCV objects shared by more than one @@ -95,7 +118,9 @@ the file stays in `library/`. ### `drivers//driver.xml` That driver's manifest: shim class, authentication server and id (not the -password), which config files exist, every policy and resource, the +password), which environments get it (`environments="QA, PROD"`, absent = every +environment) and which it is planned for but not promoted to yet +(`planned="PROD"`), which config files exist, every policy and resource, the entitlements, and the **linkage** — which object runs in which policy set, in which order. The linkage is what Designer's policy flow shows. `query tree/ chain "" sub` prints the subscriber order. @@ -131,6 +156,14 @@ path. Quote it when the driver name contains spaces. Passwords are not in these files. They live in `secrets-.properties`. +### `environments` and `planned` on other entries + +The ``, `` and `