feat(cli): predefined policy packs — run many policies in one invocation

CHANGELOG.md

@@ -10,7 +10,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed — verdicts
+
+Read this section before upgrading: it changes what some policies report, on inputs where the
+previous answer was wrong.
+
+- `core`: a tolerated provider miss no longer erases a failure recorded by an earlier resource.
+  A check whose provider returned several results — one that violated the policy and one it could
+  not read, tolerated by `error_tolerance` — reported the *skip* if the tolerated one came last,
+  because the skip marker overwrote the verdict unconditionally. The verdict therefore depended on
+  the order the provider happened to emit resources. It now reports the failure regardless of
+  order. Under `--fail-on-error` the affected runs move from exit `1` to exit `3`: previously a
+  real violation could be reported as "nothing was checked". Pass-and-skip is unchanged and still
+  reports skipped, in both orders.
+
 ### Added
+
+- Running many policies in one invocation. `-policy-path` accepts a **directory**, walked
+  recursively for `*.json`, and `--pack NAME` runs a policy pack bundled with Tirith
+  (`--list-packs` lists them). Both are repeatable and combine, so bundled and local rules produce
+  one summary and one exit code. A single policy *file* is untouched — same result document, same
+  exit codes.
+- The first bundled pack: **`terraform-baseline`**, 104 baseline security and configuration checks
+  for Terraform plans across AWS, Azure, GCP, Kubernetes and several smaller providers. Every one
+  is verified end to end in CI against a compliant and a violating document, so an engine change
+  cannot silently turn a check into a no-op.
+- A set run reports `passed / failed / skipped / errored` and prints failures in full. **Skipped is
+  not a failure**: a policy only applies to plans that touch the resource it names, so most of a
+  large pack skips on any given plan, and counting that as an error would make every pack run red.
+  With `--fail-on-error`, a set exits `3` if any policy failed, `0` if none failed and at least one
+  reached a verdict, and `1` if nothing reached one.
 - `tirith ui`: an interactive interface with three tabs.
   - **Explorer** — read an evaluation's results down to the resource behind each one. The result
     document has always carried the resource address, the planned action and the before/after

MANIFEST.in

@@ -1,5 +1,6 @@
 include *.lock
 include *.md
+include NOTICE
 include Pipfile
 recursive-include ci *.gitkeep
 recursive-include cli *.sh

NOTICE

@@ -0,0 +1,23 @@
+Tirith
+Copyright StackGuardian
+
+This product includes software developed at StackGuardian (https://stackguardian.io).
+
+--------------------------------------------------------------------------------
+Bundled policy packs
+--------------------------------------------------------------------------------
+
+The policy documents under `src/tirith/packs/` are derived works: their rules were
+translated into Tirith's policy format from open-source policy libraries, then
+re-expressed, re-identified and independently verified against synthesized
+documents. They carry StackGuardian identifiers and no upstream identifier.
+
+The rules they express derive from:
+
+  checkov -- https://github.com/bridgecrewio/checkov
+  Copyright 2019 Bridgecrew, Inc.
+  Licensed under the Apache License, Version 2.0
+  Pinned at commit 9514f12d81ebec7a8f6e70ff1ae8c3783220a59f
+
+Tirith is itself licensed under the Apache License, Version 2.0; see LICENSE.
+Each pack's `pack.json` records the corpus commit its content was generated from.

README.md

@@ -186,14 +186,17 @@ Congratulations! Tirith has been setup in your system
 ## Usage
 
 ```
-usage: tirith [-h] [-policy-path PATH] [-input-path PATH] [-var-path PATH]
-              [-var PATH] [--json] [--verbose] [--fail-on-error] [--version]
+usage: tirith [-h] [-policy-path PATH] [--pack NAME] [--list-packs]
+              [-input-path PATH] [-var-path PATH] [-var PATH] [--json]
+              [--verbose] [--fail-on-error] [--version]
 
 Tirith (StackGuardian Policy Framework)
 
 options:
   -h, --help         show this help message and exit
-  -policy-path PATH  Path containing Tirith policy as code
+  -policy-path PATH  Path to a Tirith policy file, or a directory of them
+  --pack NAME        Bundled policy pack(s) to run. Repeatable, and combines with -policy-path.
+  --list-packs       List the policy packs bundled with this installation and exit
   -input-path PATH   Input file path
   -var-path PATH     Variable file path(s)
   -var PATH          Inline variable(s)

documentation/docs/tirith-policies/tirith-policy-packs.md

@@ -0,0 +1,106 @@
+---
+id: tirith-policy-packs
+title: Policy Packs
+sidebar_label: Policy Packs
+description: Run many Tirith policies in one invocation, and the packs bundled with Tirith.
+keywords:
+  - tirith
+  - policy pack
+  - predefined policies
+site_name: Tirith
+slug: tirith-policy-packs/
+---
+
+A **pack** is a named set of policies that ships inside Tirith. Nothing about a packed policy is
+special — each one is an ordinary Tirith policy document, run through the same engine as a file
+you pass with `-policy-path`. The pack is only a name for a set, so that there is something to
+type.
+
+## Running one
+
+```bash
+tirith --list-packs
+tirith --pack terraform-baseline -input-path plan.json --fail-on-error
+```
+
+`--pack` is repeatable, and combines with `-policy-path`, so your own rules run alongside the
+bundled ones in a single invocation and a single verdict:
+
+```bash
+tirith --pack terraform-baseline -policy-path .tirith/policies -input-path plan.json
+```
+
+## Running a directory
+
+`-policy-path` also accepts a directory, which is walked recursively for `*.json`:
+
+```bash
+tirith -policy-path .tirith/policies -input-path plan.json
+```
+
+## Reading the result
+
+A run of more than one policy reports a summary rather than every policy in full:
+
+```
+104 policies · 5 passed · 2 failed · 97 skipped
+Skipped policies found no resource of the type they check.
+
+✘ 2 policy/policies failed
+```
+
+**Most policies skipping is the normal outcome, not a problem.** A check applies only to plans
+that touch the resource it names, so a plan that creates one EC2 instance leaves almost every
+check in a large pack with nothing to look at. Those policies are counted as `skipped`; they are
+neither a pass nor a failure, and they do not affect the exit code. Failures are printed in full,
+because they are the reason you ran it. `--verbose` prints every policy.
+
+`--json` returns the aggregate document:
+
+```json
+{
+  "summary": {"total": 104, "passed": 5, "failed": 2, "skipped": 97, "errored": 0},
+  "final_result": false,
+  "policies": [
+    {"policy": "terraform-baseline/SG_TF_0042_aws_s3_bucket_versioning.json", "meta": {}, "final_result": false, "evaluators": []}
+  ]
+}
+```
+
+Each entry in `policies` is exactly the result document a single-policy run produces, plus a
+`policy` name, so anything that already reads a Tirith result can read one of these.
+
+A single policy **file** is unchanged: it returns the single-policy document and the exit codes it
+always has. The shape follows how the run was asked for — a directory or a `--pack` is a set — not
+how many policies happened to match, so a directory holding one policy still reports as a set.
+
+## Exit codes
+
+With `--fail-on-error`:
+
+| Situation | Code |
+|---|---|
+| At least one policy failed | `3` |
+| No failures, at least one policy reached a verdict | `0` |
+| Nothing ran, or every policy skipped | `1` |
+
+Skipped policies never produce `3`. A pack whose policies all skipped exits `1`, on the same rule
+the single-policy path applies to `final_result: null`: nothing was checked, so nothing can be
+reported as green.
+
+## Bundled packs
+
+| Pack | Policies | Scope |
+|---|---|---|
+| `terraform-baseline` | 104 | Baseline security and configuration checks for Terraform plans, across AWS, Azure, GCP, Kubernetes and several smaller providers |
+
+Every policy in a bundled pack has been verified end to end: it passes a compliant document and
+fails a violating one, and those fixtures are run in Tirith's own CI, so an engine change cannot
+silently turn a check into a no-op.
+
+### Identifiers and tags
+
+Packed policies carry a StackGuardian id — `SG_TF_0042` — that is allocated once and never
+reassigned, so a report can name a check and mean the same check next release. `meta.tags` carry a
+`cloud:<name>` tag plus the policy's category, and are the seam that compliance-framework tags
+will slot into.

documentation/docs/tirith-usage/cli-reference.md

@@ -30,7 +30,9 @@ Note the spelling: the path and variable options take a **single dash** (`-polic
 
 | Flag | Argument | What it does |
 |---|---|---|
-| `-policy-path` | `PATH` | Path to the Tirith policy file. Required. |
+| `-policy-path` | `PATH` | Path to a Tirith policy file, or a directory of them. Required unless `--pack` is given. |
+| `--pack` | `NAME` | A bundled policy pack to run. Repeatable, and combines with `-policy-path`. |
+| `--list-packs` | | List the packs bundled with this installation and exit. |
 | `-input-path` | `PATH` | Path to the document the policy is evaluated against. Required. |
 | `-var-path` | `PATH` | Path to a JSON file of policy variables. Repeatable. |
 | `-var` | `NAME=JSON` | One inline policy variable. Repeatable. |
@@ -48,7 +50,23 @@ See the [policy reference](../tirith-policies/tirith-policy-reference.md) for th
 [providers overview](../tirith-providers/overview.md) for what kinds of input each
 `required_provider` reads.
 
-If the flag is missing, `tirith` prints an error to stderr and exits `1`.
+Given a **directory**, it is walked recursively and every `*.json` in it is evaluated against the
+same input, with the verdicts rolled up into one summary and one exit code — see
+[policy packs](../tirith-policies/tirith-policy-packs.md).
+
+If neither this flag nor `--pack` is given, `tirith` prints an error to stderr and exits `1`.
+
+### `--pack`
+
+A set of policies bundled with Tirith, run by name:
+
+```bash
+tirith --list-packs
+tirith --pack terraform-baseline -input-path plan.json --fail-on-error
+```
+
+Repeatable, and combines with `-policy-path`, so your own rules and the bundled ones produce one
+verdict. An unknown pack name exits `1`. See [policy packs](../tirith-policies/tirith-policy-packs.md).
 
 ### `-input-path`
 

documentation/docs/tirith-usage/exit-codes.md

@@ -36,6 +36,17 @@ Both surfaces **fail closed**: anything that leaves the verdict unknown exits no
 of `--fail-on-error`. That flag governs policy verdicts, not tool health — a run that produced no
 verdict must never look like a pass.
 
+## Running more than one policy
+
+A directory of policies or a `--pack` produces one exit code for the whole set, on the same rules:
+`3` if any policy failed, `0` if none failed and at least one reached a verdict, `1` if nothing
+ran or every policy skipped.
+
+A **skipped** policy — one whose resource type is absent from the input — never produces `3`. For
+a pack of any size that is the normal outcome for most policies, since a check only applies to
+plans that touch the resource it names; counting skips as failures would make every pack run red
+regardless of the infrastructure.
+
 ## Without `--fail-on-error`
 
 The local command exits `0` whether the policy passed or failed, with the verdict in the output.

documentation/sidebars.js

@@ -40,6 +40,7 @@ module.exports = {
         "tirith-policies/tirith-policy-conditions",
         "tirith-policies/tirith-policy-variables",
         "tirith-policies/tirith-policy-cookbook",
+        "tirith-policies/tirith-policy-packs",
         // "tirith-policies/tirith-policy-examples"
       ]
     },

setup.py

@@ -45,6 +45,9 @@
     package_data={
         "tirith.tui": ["*.css"],
         "tirith.tui.examples": ["*/*.json", "*/*.md"],
+        # The bundled policy packs. Without these `--pack` finds nothing and `--list-packs`
+        # prints an empty list, which is the same silent-empty failure the TUI had.
+        "tirith.packs": ["*/pack.json", "*/policies/*.json"],
     },
     zip_safe=False,
     classifiers=[
@@ -56,7 +59,7 @@
         "Operating System :: POSIX",
         # 'Operating System :: Microsoft :: Windows',
         "Programming Language :: Python",
         # 'Programming Language :: Python :: 2.7',
         # 'Programming Language :: Python :: 3',
         # 'Programming Language :: Python :: 3.5',
         # 'Programming Language :: Python :: 3.6',