design(state): store installation state as one JSON file per resource type

[bwl21]

Problem

ct-cli currently stores the complete installation state for one ChurchTools host in a single ct-state*.json file. That works for small configurations, but it couples unrelated ChurchTools domain objects into one growing document and one Git conflict surface.

For a process that adopts hundreds of resources, the important question is not whether one JSON file can technically hold them. A file containing 600 groups is still reasonable. The useful boundary is the canonical resource type, equivalent to a ChurchTools domain object.

The state should therefore be able to use a directory with one JSON file per canonical resource type. It should not introduce arbitrary size-based shards such as groups-001.json.

Proposed model

An environment points to an installation-state directory instead of one monolithic file:

instances/bgkorntal.church.tools/
└── state/
    ├── manifest.json
    ├── resources/
    │   ├── groups.json
    │   ├── group-types.json
    │   ├── group-roles.json
    │   └── security-levels.json
    ├── externals/
    │   └── group-types.json
    └── synthetic/
        └── dynamic-groups.json

The exact partitioning of managed resources, external bindings and synthetic sub-resources remains to be decided. The invariant is that each file has a semantic boundary derived from the canonical resource/domain type, not from item count or byte size.

For example, all 600 managed groups belong in resources/groups.json and remain keyed by their logical ct-cli keys.

manifest.json owns installation-wide metadata such as:

{
  "version": 3,
  "host": "https://bgkorntal.church.tools"
}

Why this boundary

• Git diffs and conflicts remain scoped to the affected ChurchTools domain.
• A resource type can be inspected, validated and migrated as one coherent unit.
• The layout follows the same canonical registry that drives CLI, REST and engine operations.
• Large synthetic values such as dynamic rulesets can evolve without inflating every unrelated state diff.
• No artificial sharding, lookup index or repartitioning threshold is needed.

This is primarily a semantic and collaboration improvement, not a claim that several hundred resources exceed JSON parsing limits.

Required design decisions

• How canonical resource-type identifiers map to stable filenames.
• Whether managed resources and external bindings use separate directories or a discriminated entry format.
• Whether synthetic ChurchTools sub-resources are embedded in their owning resource entry or receive their own domain-type files.
• How directory-wide validation, locking, fingerprinting and atomic persistence work.
• How a partial write is prevented when an apply changes more than one type file.
• Whether ct.envs.json may point state at either a legacy file or a new directory.
• How commands accepting --state distinguish and report file versus directory state.
• How backups, state list, state rm, adoption, destroy and ownership discovery consume the directory.

Atomicity requirement

The current state file can be replaced atomically after a successful operation. Splitting it must not weaken ct-cli's crash-resume guarantee.

Locking and confirmation fingerprints must cover the complete installation-state directory, not only the resource file being modified. A multi-file update must either become visible as one consistent revision or recover deterministically after interruption.

Compatibility and migration

• Existing version 1 and version 2 state files must remain readable.
• Provide an explicit, reviewable migration from a monolithic file to the directory layout.
• Migration must preserve host binding, logical keys, IDs, timestamps, preventDestroy, member-field identities and external bindings exactly.
• A migration must not contact or modify ChurchTools.
• A directory state should serialize deterministically so repeated writes without semantic changes produce no Git diff.
• ct init and environment documentation should use the chosen layout once it becomes the default.

Acceptance criteria

• A state directory containing multiple resource-type files behaves identically to the equivalent monolithic state for plan, apply, adopt, state inspection, destroy and external-reference ownership checks.
• A test with at least 600 groups remains in one groups.json file; no size-based sharding is introduced.
• Concurrent or interrupted writes cannot expose a silently partial state revision.
• Directory fingerprints are deterministic and invalidate a prepared REST apply when any constituent state file changes.
• Git diffs for a change to one resource type do not rewrite unrelated resource-type files.
• Legacy file states have a documented compatibility and migration path.

Relationships

• design(workspace): share environments while keeping process state local #166 defines workspace-wide environments while keeping environment-qualified process state in the process directory. This issue defines a scalable on-disk representation for that process state.
• fix(state): snapshot dynamic rulesets for three-way drift detection #167 requires a persisted baseline for dynamic rulesets. A state directory provides a natural place for that domain without prescribing whether it is embedded or stored separately.
• feat(server): versioned REST API for ChurchTools Extension clients #164 prepares and confirms operations through REST; its state fingerprint and OpenAPI-visible paths must remain correct for directory states.

This issue is architectural. It does not propose changing resource ownership, adopting additional ChurchTools objects, or altering the no-implicit-delete policy.

Sequencing: consolidate repository tools only after #166 and #168

Do not move existing consumer-repository validation or reporting helpers into ct-cli before both the workspace-environment model (#166) and this state-directory model have been implemented. Doing so earlier would make a new canonical validator depend on an intentionally temporary layout.

The follow-up after both issues land should:

• implement workspace/process/state validation once as a ct-cli application operation;
• expose that operation through CLI and REST/OpenAPI from the shared operation catalog;
• validate the directory state, environment ownership, host bindings and absence of undeclared fallback states;
• replace repository-local validation scripts and duplicated documentation;
• replace any remaining local plan-report renderer with ct-cli's canonical Markdown plan projection;
• provide a documented cleanup/migration step for existing consumer repositories.

Existing local tools stay in place until that follow-up is available and the repository has migrated successfully.