Improve resource grains, add persistent cache

changelog/+deploy_key_actions_present.added.md

@@ -1 +1 @@
-Added composite states that generate an SSH keypair in memory via the `ssh_pki` module and distribute the public half as a deploy key and the private half as an Actions secret, with optional rotation ({py:func}`github_repo.deploy_key_actions_present <saltext.github.states.github_repo.deploy_key_actions_present>` and the resource-type {py:func}`deploy_key.actions_present <saltext.github.resources.github.states.deploy_key.actions_present>`)
+Added composite states that generate an SSH keypair in memory via the {py:mod}`ssh_pki <salt.modules.ssh_pki>` module and distribute the public half as a deploy key and the private half as an Actions secret, with optional rotation ({py:func}`github_repo.deploy_key_actions_present <saltext.github.states.github_repo.deploy_key_actions_present>` and the resource-type {py:func}`deploy_key.actions_present <saltext.github.resources.github.states.deploy_key.actions_present>`)

changelog/+grains_files.added.md

@@ -0,0 +1 @@
+Added custom repository resource grains sourced from in-repo YAML/JSON files (e.g. copier answers) via the {ghconf}`grains_files <resource:grains_files>` resource declaration key, with per-namespace key filtering, renaming and optional merging into the root grains

changelog/+persistent_cache.added.md

@@ -0,0 +1 @@
+Added a persistent cross-process per-connection cache (on by default, opt out via the {ghconf}`cache` profile parameter), serving token/account identities and file-based custom resource grains (validated against each repository's `pushed_at` timestamp) without refetching, e.g. across `salt-call --local` invocations

changelog/+repo_info_fields.added.md

@@ -0,0 +1 @@
+Extended repository information ({py:func}`github_repo.info <saltext.github.modules.github_repo.info>`/{py:func}`github_repo.list <saltext.github.modules.github_repo.list_>`, but also affects the legacy {py:func}`github.get_repo_info <saltext.github.modules.github.get_repo_info>` and the resource grains) with `visibility`, `archived`, `disabled`, `is_template`, `license`, `topics`, `custom_properties`, `clone_url`, `ssh_url`, `created_at`, `has_discussions`, `has_pages` and `has_projects` (info additionally carries `updated_at`)

docs/ref/resources/github/saltext.github.resources.github.modules.grains.rst

@@ -0,0 +1,5 @@
+``grains``
+==========
+
+.. automodule:: saltext.github.resources.github.modules.grains
+    :members:

docs/ref/resources/github/saltext.github.resources.github.modules.rst

@@ -10,6 +10,7 @@ Execution Modules
     config
     deploy_key
     environment
+    grains
     issue
     label
     milestone

docs/ref/utils/index.rst

@@ -12,10 +12,12 @@ _________
     utils
     utils.gh
     utils.gh.actions
+    utils.gh.cache
     utils.gh.client
     utils.gh.deploy_keys
     utils.gh.environments
     utils.gh.exceptions
+    utils.gh.grains_files
     utils.gh.issues
     utils.gh.labels
     utils.gh.members

docs/ref/utils/saltext.github.utils.gh.cache.rst

@@ -0,0 +1,5 @@
+saltext.github.utils.gh.cache
+=============================
+
+.. automodule:: saltext.github.utils.gh.cache
+    :members:

docs/ref/utils/saltext.github.utils.gh.grains_files.rst

@@ -0,0 +1,5 @@
+saltext.github.utils.gh.grains_files
+====================================
+
+.. automodule:: saltext.github.utils.gh.grains_files
+    :members:

docs/topics/configuration.md

@@ -21,7 +21,7 @@ The following keys can be specified in the profile:
 :::{ghconf} token
 :::
 `token`
-: Required. Access token to authenticate with.
+: **Required**. Access token to authenticate with.
 
 :::{ghconf} owner
 :::
@@ -40,6 +40,44 @@ The following keys can be specified in the profile:
   `org_name` keeps being supported as a legacy alias.
   :::
 
+:::{ghconf} api_url
+:::
+`api_url`
+: Base URL of the GitHub API. Set this when targeting a GitHub
+  Enterprise Server instance, e.g. `https://github.example.com/api/v3`.
+  Defaults to `https://api.github.com`.
+
+:::{ghconf} cache
+:::
+`cache`
+: Keep a persistent cross-process cache for specific API data,
+  stored in the minion's {conf_minion}`cachedir`.
+  All entries are namespaced by a token/API URL-derived identity,
+  hence rotated credentials never serve stale data.
+  The data that is cached is either static or refreshed as soon
+  as it changes. Defaults to true.
+
+  :::{versionadded} 1.1.0
+  :::
+
+  :::{hint}
+  Always cached are:
+
+  * {ghconf}`token`-associated user handle (never changes)
+  * account name => account type lookups (should never change)
+
+  When the {py:mod}`github resource type <saltext.github.resources.github>` is configured
+  with {ghconf}`grains_files <resource:grains_files>`, file contents
+  (including negative lookups) are also cached in relation to
+  the repository's `pushed_at` timestamp (and automatically refreshed as soon
+  as that timestamp changes).
+  {py:func}`grains_refresh <saltext.github.resources.github.grains_refresh>`
+  bypasses and rewrites the cached file contents.
+  This especially benefits `salt-call --local` setups, where every
+  invocation starts a fresh process and would otherwise refetch all
+  file-based grains.
+  :::
+
 :::{ghconf} repo_name
 :::
 `repo_name`
@@ -54,13 +92,6 @@ The following keys can be specified in the profile:
 : Allow automated repository privacy modifications. Defaults to false since
   it can be dangerous to change the privacy of a repository in an automated way.
 
-:::{ghconf} api_url
-:::
-`api_url`
-: Base URL of the GitHub API. Set this when targeting a GitHub
-  Enterprise Server instance, e.g. `https://github.example.com/api/v3`.
-  Defaults to `https://api.github.com`.
-
 ## Multiple profiles
 
 Any number of profiles can be defined and selected per call:
@@ -179,6 +210,55 @@ the master's {conf_master}`pillar_opts` setting) – including SDB rendering.
   Can be overridden per owner in the {ghconf}`owners <resource:owners>`
   mapping.
 
+:::{ghconf} resource:grains_files
+:::
+`grains_files`
+: Custom resource grains sourced from YAML/JSON files inside each
+  repository. A mapping of grain namespace to file specification –
+  each namespace becomes a grain key next to the builtin ones
+  (`id`, `owner`, `repo`, ..., which namespaces must not shadow).
+  The special `_` namespace instead merges its keys into the root
+  grains level; keys that already exist – the builtin identity grains
+  or earlier namespaces – are never overwritten.
+
+  `path`
+  : File path(s) inside the repository, a string or a list. All listed
+    files are fetched; missing ones are skipped and the parsed mappings
+    are shallow-merged in list order, earlier files winning.
+
+  `include`
+  : Optional list of strings or single-key dicts selecting the exposed keys.
+
+    String entries are fnmatch patterns matched case-sensitively against
+    colon-delimited key paths; a match includes the whole subtree, preserving nesting.
+    `*` crosses `:` boundaries.
+
+    Single-entry mappings rename:
+    the value at the exact source path (scalar or subtree, no patterns)
+    is exposed under the new name as a flat or nested namespace key.
+
+    A plain mapping of new name to source path is shorthand for a
+    list of mostly renames. Unset means everything is included.
+
+    See the [Full example config][full-example-config] for detailed syntax examples.
+
+  `exclude`
+  : Optional list of fnmatch patterns pruning matching key paths at
+    any depth, applied after `include`. Does not apply to renamed
+    keys. Note that `*` also crosses `:` boundaries.
+
+  Can be overridden per owner in the {ghconf}`owners <resource:owners>`
+  mapping. Unlike the other keys, the override replaces the
+  declaration-level value entirely (no merging) – an empty mapping
+  thus disables custom grains for that owner.
+
+  :::{important}
+  These grains are controlled by anyone who can commit to the
+  respective repository. Restrict the exposed keys via `include`
+  allowlists and keep that trust model in mind when building targeting
+  expressions. The builtin identity grains cannot be shadowed.
+  :::
+
 ### Minimal example config
 This example showcases the minimally required config to use the modules
 and expose repositories as Salt resources.
@@ -232,20 +312,92 @@ other_enterprise_org:
   token: sdb://osenv/GITHUB_TOKEN
   owner: other-org
 
-# Expose specific repositories as resources
+# Expose select repositories as resources
 resources:
   github:
+
+    # By default, for any owner, only expose repositories that start with `foo-`,
+    # including archived ones.
+    include_archived: true
+    repos:
+      - foo-*
+
+    # Also try to load a file named `.github/data.json` from each repository and
+    # include all keys but those starting with an underscore in grains["data"]
+    grains_files:
+      data:
+        path: .github/data.json
+        exclude:
+          - "_*"
+
+    # Expose repositories specifically from these accounts.
     owners:
+
+      # Use the defaults above. Exposed are: my-github-handle/foo-*
       my-github-handle: {}
+
+      # Exclude archived ones and a specific one. Exposed are my-org/foo-*,
+      # but not the one repository named my-org/foo-not-this-repo.
       my-org:
         exclude:
           - foo-not-this-repo
+        include_archived: false
+
+      # Expose repositories matching other-org/bar-* while using a different profile.
+      # Don't render any custom grains from files in these repositories.
       other-org:
         profile: other_enterprise_org
+        grains_files: {}
         repos:
           - bar-*
-        include_archived: false
-    repos:
-      - foo-*
-    include_archived: true
+
+      # Reset `repos` config to expose all repos.
+      # Showcase custom grains (in detail).
+      tiny-org:
+        # Reset repo selection to include all repos in tiny-org/*.
+        repos: {}
+        # Override the default custom file grains config completely
+        grains_files:
+
+          # When a repo has a `.github/grains.json` file, load it and merge
+          # select keys (by "glob" match) into the repository's grains (no namespace).
+          # You can rename keys by passing a single-key dict
+          # (no globs allowed in source or target).
+          # Explicit renames always win a slot.
+          # Note: inbuilt grains can never be overridden.
+          _:
+            path: .github/grains.json
+            include:
+              - team
+              - status
+              - nested:key
+              - 'include-all-*'
+              - '*-also-these'
+              - 'and-*-these'
+              - 'and:*-these'  # a glob spans nesting levels
+              - some_key: renamed:from:a:deeply:nested:key
+              - some:nested:target: renamed:from:another:deeply:nested:key
+
+          # Try loading several paths and merge their data into the `data` namespace in grains.
+          # If more than one file specifies the same key, the *first* one wins.
+          # This one still excludes any keys that start with an underscore, but shows
+          # exclude/include can be specified as a simple string.
+          data:
+            path:
+              - .github/foo.json
+              - .github/bar.yml
+            exclude: _*
+
+          # Namespaces can be nested.
+          # `include` can be a mapping as a shorthand if most of your
+          # includes are renames. Regular patterns are still supported
+          # by specifying a falsy value.
+          nested:target:copier:
+            path: .copier-answers.yml
+            include:
+              template_version: _commit
+              nested:target: my_question
+              nested_source: my:question
+              just_keep_this_explicit_key:
+              keep_glob_*:stuff: ''
 ```

docs/topics/repo_resources.md

@@ -12,9 +12,12 @@ resource. Compared to the classic
 {py:mod}`github_repo <saltext.github.modules.github_repo>` modules,
 resources offer:
 
-- direct targeting of repositories (`salt 'my-org/my-repo' ...`),
-  including grain-based targeting across all managed repositories
-- per-repository grains (owner, default branch, visibility, ...)
+- direct targeting of repositories (`salt 'my-org/my-repo'` /
+  `salt -C 'T@github:my-org/my-repo'`), including grain-based
+  targeting across all managed repositories (`salt -G 'default_branch:master'`)
+- per-repository grains (owner, default branch, visibility, ...) -
+  a repository resource's grains can optionally even include data files
+  from the repo itself
 - repository-implicit execution and state modules – SLS files describe
   one repository's desired configuration and are applied per resource,
   without threading `repo_name`/`owner` through every call
@@ -66,16 +69,45 @@ with the SRN `github:<owner>/<repo>`.
 ## Targeting
 
 ```bash
-# One repository
+# One repository, by ID
 salt 'my-org/my-repo' test.ping
 
+# One repository, by SRN
+salt -C 'T@github:my-org/my-repo' test.ping
+
 # By resource grain
-salt -G 'github:default_branch:master' repo.info
+salt -G 'default_branch:master' repo.info
 ```
 
-The resource grains are provided under the `github` key: `id`, `owner`,
-`repo`, `full_name`, `private`, `fork`, `default_branch`, `language`,
-`html_url`, `description` and `homepage`.
+The resource grains carry the repository's identity and basic metadata:
+`id`, `owner`, `repo`, `full_name`, `private`, `visibility`, `fork`,
+`archived`, `disabled`, `is_template`, `default_branch`, `language`,
+`license`, `topics`, `custom_properties`, `html_url`, `clone_url`,
+`ssh_url`, `description`, `homepage`, `created_at`, `has_discussions`,
+`has_pages` and `has_projects`.
+
+Custom grains sourced from files inside each repository can be declared
+via {ghconf}`resource:grains_files`, each namespace becoming a grain key
+next to the builtin ones (the special `_` namespace merges into the root
+grains, never overwriting existing keys):
+
+```bash
+salt -G 'copier:template_version:2*' repo.info
+```
+
+:::{important}
+Custom grains are controlled by anyone who can commit to the respective
+repository. Restrict the exposed keys via `include` allowlists and
+keep that trust model in mind when building targeting expressions.
+:::
+
+:::{hint}
+File-based grains and account metadata survive across processes (e.g.
+`salt-call --local` invocations) in a persistent cache, which is
+enabled by default (opt out per profile via {ghconf}`cache`). Cached
+file contents stay valid until the respective repository receives a
+new push.
+:::
 
 ## Execution modules