feat(k8s): add the searxng-check command

README.md

@@ -240,6 +240,15 @@ throwaway pod that carries the label of the service, so the NetworkPolicy applie
 Run it at each bump of the image digest. The token never appears in the output. Exits 1 when
 a target is served, a request without the token is not refused, or the pod opens a
 connection. |
+|  | `devops k8s searxng-check [OPTIONS]` | Check the points of the SearXNG service that a bump of the image digest can change. It sends
+six search requests with format=json through a local port-forward: one for each category of
+the provider, a query that starts with ! and names an engine that is off, a query that starts
+with !! (the redirect is recorded and never followed), and a category that no kept engine
+serves. It judges the keys of each reply, the hits, and the engines that must stay off. It
+also tries three connects from two throwaway pods, so the NetworkPolicy applies to them: a
+pod that is not a client of the service to port 8080, and a pod with the label of the service
+to the metadata address and to a private ClusterIP. Run it at each bump of the image digest.
+Exits 1 when a row fails or a pod opens a connection. |
 |  | `devops k8s create-tls-secret [OPTIONS] <secret_name>` | Create or update a kubernetes.io/tls secret from certificate and private key files. |
 |  | `devops k8s enable-tls [OPTIONS]` | Generate Homelab certificates and apply TLS secrets across Kubernetes cluster namespaces. |
 |  | `devops k8s validate [OPTIONS] <manifest_path>` | Validate Kubernetes YAML manifests against OpenAPI schemas using Kubeconform. |

docs/CLI_REFERENCE.md

@@ -1304,6 +1304,30 @@ devops k8s crawl4ai-check [OPTIONS]
 | `--context` | `string` | - | Kubernetes cluster context name. |
 | `--dry-run` | `boolean` | - | Preview execution plan without mutating external state. |
 
+### `devops k8s searxng-check`
+
+**Check the points of the SearXNG service that a bump of the image digest can change. It sends
+six search requests with format=json through a local port-forward: one for each category of
+the provider, a query that starts with ! and names an engine that is off, a query that starts
+with !! (the redirect is recorded and never followed), and a category that no kept engine
+serves. It judges the keys of each reply, the hits, and the engines that must stay off. It
+also tries three connects from two throwaway pods, so the NetworkPolicy applies to them: a
+pod that is not a client of the service to port 8080, and a pod with the label of the service
+to the metadata address and to a private ClusterIP. Run it at each bump of the image digest.
+Exits 1 when a row fails or a pod opens a connection.**
+
+```bash
+devops k8s searxng-check [OPTIONS]
+```
+
+**Options:**
+
+| Option / Flag | Type | Default | Description |
+|---|---|---|---|
+| `--local-port` | `integer` | `8081` | Local port of the kubectl port-forward to Service searxng. Use a port that no program holds. |
+| `--context` | `string` | - | Kubernetes cluster context name. |
+| `--dry-run` | `boolean` | - | Preview execution plan without mutating external state. |
+
 ### `devops k8s create-tls-secret`
 
 **Create or update a kubernetes.io/tls secret from certificate and private key files.**

docs/ERRORS.md

@@ -121,7 +121,7 @@ Each exception class declares its `error_code` and `exit_code`. A command that a
 | `SecurityError` | `SECURITY_ERROR` | `1` | Security | Base exception for all security, policy, and egress violations. |
 | `ServiceAddressError` | `K8S_ERROR` | `1` | Devops_cli.k8s.service_proxy | Raised when a cluster service address cannot be resolved. |
 | `ServiceNotReachableError` | `DEVOPS_CLI_ERROR` | `1` | Devops_cli.k8s.node_port | A service cannot be reached from outside the cluster. |
-| `ServiceProbeError` | `K8S_SERVICE_PROBE_ERROR` | `1` | K8s | Raised when `devops k8s crawl4ai-check` cannot probe a service: the forward, the throwaway pod |
+| `ServiceProbeError` | `K8S_SERVICE_PROBE_ERROR` | `1` | K8s | Raised when `devops k8s crawl4ai-check` or `devops k8s searxng-check` cannot probe a service: |
 | `ServiceStatusError` | `SERVICE_STATUS_ERROR` | `1` | Telemetry | Raised when querying upstream published service status fails. |
 | `ServiceStoppingError` | `SERVICE_STOPPING` | `1` | Roadmap | Raised when a request ends because the Service is stopping (#2753). |
 | `StackCredentialError` | `K8S_STACK_CREDENTIAL_ERROR` | `1` | K8s | Raised when a stack credential cannot be read, checked, minted, revoked or stored (#2156). |

docs/commands/k8s.md

@@ -557,6 +557,32 @@ devops k8s crawl4ai-check [OPTIONS]
 
 ---
 
+## `devops k8s searxng-check`
+
+**Check the points of the SearXNG service that a bump of the image digest can change. It sends
+six search requests with format=json through a local port-forward: one for each category of
+the provider, a query that starts with ! and names an engine that is off, a query that starts
+with !! (the redirect is recorded and never followed), and a category that no kept engine
+serves. It judges the keys of each reply, the hits, and the engines that must stay off. It
+also tries three connects from two throwaway pods, so the NetworkPolicy applies to them: a
+pod that is not a client of the service to port 8080, and a pod with the label of the service
+to the metadata address and to a private ClusterIP. Run it at each bump of the image digest.
+Exits 1 when a row fails or a pod opens a connection.**
+
+```bash
+devops k8s searxng-check [OPTIONS]
+```
+
+**Options:**
+
+| Option / Flag | Type | Default | Description |
+|---|---|---|---|
+| `--local-port` | `integer` | `8081` | Local port of the kubectl port-forward to Service searxng. Use a port that no program holds. |
+| `--context` | `string` | - | Kubernetes cluster context name. |
+| `--dry-run` | `boolean` | - | Preview execution plan without mutating external state. |
+
+---
+
 ## `devops k8s create-tls-secret`
 
 **Create or update a kubernetes.io/tls secret from certificate and private key files.**

k8s/README.md

@@ -841,7 +841,8 @@ The rule group `crawl4ai` in `monitoring/prometheus-values.yaml` watches the pag
   1. Choose the newest tag that is at least 14 days old (the tag list of `https://hub.docker.com/r/searxng/searxng/tags`).
   2. Set the tag and the digest in `deployment.yaml`.
   3. Refresh `tests/fixtures/searxng/default_engine_names.txt` (the first line names the tag): `docker run --rm --entrypoint cat <image> /usr/local/searxng/searx/settings.yml | python3 -c "import sys, yaml; print(*sorted(e['name'] for e in yaml.safe_load(sys.stdin)['engines']), sep='\n')"`. If a name of `keep_only` is gone, a test fails: read the notes of the release for the new name.
-  4. Run `devops ci test tests/invariants/test_k8s_searxng.py`, then start the image with the controls of the pod and query it once with `format=json`.
+  4. Run `devops ci test tests/invariants/test_k8s_searxng.py`, then run `devops k8s searxng-check` against the new image (see the next bullet) and set `SEARXNG_IMAGE` in `src/devops_cli/k8s/searxng_check.py`.
+- **The check.** `devops k8s searxng-check` tests the running service for the points that a new digest can change. Through a `kubectl port-forward` it sends six `GET /search` requests with `format=json` and no credential: one for each category of the provider (`general`, `it`, `science`), a query `!google ...` that must stay text, a query `!!g ...` whose redirect is recorded and never followed, and a category that no kept engine serves. It fails a row when a reply lacks one of the seven keys (`answers`, `corrections`, `infoboxes`, `query`, `results`, `suggestions`, `unresponsive_engines`), when HTTP 403 shows that `json` is not in `search.formats`, when a category has no hit, or when an engine that must stay off (Brave, Google, Bing, Startpage, Mojeek, Qwant) shows in `results` or in `unresponsive_engines`. It also starts two throwaway pods and tries three connects: a pod that is not a client of the service must not reach port 8080, and a pod with the label of the service must reach neither the metadata address nor a private ClusterIP. The report names each pod and target by a role. A person runs the command at each bump of the digest. Until `SEARXNG_IMAGE` names the new image, the `k8s` row of `devops ci` fails. The query count stays at six, below the about 30 queries after which the upstream sites suspend an engine. If a row fails on the pinned image, do not add code for it: file the evidence as an issue. `devops k8s searxng-check --dry-run` lists every call and makes none.
 - **The log.** `kubectl -n devops logs deploy/searxng` shows the engines that failed. A line `CAPTCHA` from `duckduckgo` or `Suspended: access denied` from `github` means that the engine refused the pod address for a while. The pod answers with the other engines, and the search passes to the next provider when no engine gives a hit. The start prints a warning that `/var/cache/searxng` is not owned by `searxng:searxng`: the group of the volume is 977 (`fsGroup`), and the pod writes to it.
 - **The limits.** Measured with the pinned image under the controls of the pod: it idles at 114 MiB, 20 queries one after another peak at 117 MiB, and 10 at once peak at 122 MiB and 0.8 CPU. The Deployment asks for 192 Mi and 50m, with the limits 384 Mi and 1 CPU. Measure again on the cluster after the first sync.
 

openspec/changes/archive/2026-11-08-3929-searxng-check-command/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-11-08

openspec/changes/archive/2026-11-08-3929-searxng-check-command/design.md

@@ -0,0 +1,28 @@
+# Design
+
+## Decisions
+
+- **A `devops k8s` command, not a live pytest test.**: The test floor fakes the network and `docs/TESTING.md` forbids a skip on the state of the host, so a live test cannot reach the service. The owner chose a command on 2026-11-07. The tests of the command use the process and HTTP fakes, and a person runs the command at each digest bump.
+- **Two modules in `src/devops_cli/k8s/`, one of them the existing shared one.**: `service_probe.py` (#4108) holds the forward and the throwaway pod and stays as it is. `searxng_check.py` holds the rows, `reply_problems`, the engines that stay off and `SEARXNG_IMAGE`. The command module in `commands/k8s/` is a thin adapter: options, tables and exit code.
+- **Six search rows with fixed names.**: `json-general`, `json-it`, `json-science`, `bang-off-engine`, `bang-external` and `unused-category`. The names are the keys of the report and of the tests. Six requests stay far under the limit of about 30 queries after which the upstream sites suspend an engine (`k8s/README.md`).
+- **The categories come from `DEFAULT_SEARXNG_CATEGORIES`.**: The provider sends this value, so the check holds the provider to the pinned image. If a kept engine stops answering in a category, the row of that category has no hit and fails, and the fix is to change the default. The check reads the default and does not copy it.
+- **`reply_problems(status, body, *, expect_hits=True, also_off=())` returns a list of texts.**: An empty list is a pass. `expect_hits` is false for the rows that need no hit, and `also_off` adds the engine that a bang names. The same function judges the three `json-` rows and the off-engine bang, so a missing key, a bad `unresponsive_engines`, a 403 (named as `search.formats`), a missing hit and an engine that stays off each have one place. A text names the key, the category or the engine, and the report prints it.
+- **The engines that stay off are a prefix list.**: `brave`, `braveapi`, `google`, `bing`, `startpage`, `mojeek` and `qwant`, matched by prefix, because `brave` also covers `brave.news`. The list lives once in `searxng_check.py`, and `tests/invariants/test_k8s_searxng.py` imports it. The check reads both `results[].engine` and `unresponsive_engines[][0]`: an engine that is on and suspended shows only in the second.
+- **The external bang is recorded and never followed.**: `http_client` follows redirects by default, so the command passes `follow_redirects=False`. SearXNG answers `!!g` with a redirect to the page of the external site, and the command must not send a request there: the request would leave the cluster under the name of a person. The row records the status and whether a `Location` header exists, and fails only on a fault. The provider already refuses such a query before it sends it.
+- **The unused category is recorded and not asserted as an error.**: The source (`parse_category_form`) falls back to `general` for a category that is not loaded, so 200 with hits is the expected answer. The row fails on a status of 500 or more or on no answer.
+- **Two pods for three connects.**: One pod with a neutral label tries port 8080 of the Service. One pod with the label `app.kubernetes.io/name=searxng` tries the metadata address and the ClusterIP of `roadmap-service`. The first proves that the ingress rule selects the client label and no other. The second proves the egress rule. The pods never become ready (`probe_from_pod`), so the Service sends them no traffic. The neutral label is not the label of `roadmap-service`, the only allowed client, and not the label of `searxng`.
+- **The report names pods and targets by role.**: `<unlabeled-pod>`, `<searxng-pod>`, `<searxng-service>`, `<metadata>` and `<private-cluster-ip>`. The ClusterIPs are read at run time and never printed (rule 10 of AGENTS.md, rule 8 of `SECURITY.md`).
+- **The image check extends the existing check of the `k8s` row.**: `tests/gate/k8s_checks/devops_runtime.py` today asserts only that the image of Deployment `searxng` starts with `docker.io/searxng/searxng:`. It now compares the full image with `SEARXNG_IMAGE`, as it does for Crawl4AI.
+- **The reply fixture comes from the Publish stage.**: A saved reply needs the cluster. The Publish stage runs the command once, saves one reply of the pinned image with the masking of private data, and writes the image into a file next to it. The test of the fixture is part of the same group, so no stage between leaves a red suite.
+- **The command is not an MCP tool.**: It starts pods, so it is not read-only. `--dry-run` has the class `request-plan`.
+- **A changed behavior is not fixed here.**: If a row fails on the pinned image, the person files the evidence as an issue. The change adds no compensating code (AGENTS.md rule 2).
+
+## Rejected alternatives
+
+- **A live pytest test with a marker.**: `docs/TESTING.md` forbids a skip on the state of the host, and the owner rejected it on 2026-11-07.
+- **Follow the redirect of `!!g`.**: The command would send a request to an external site, and a followed redirect hides whether the service redirects.
+- **Assert an error for the unused category.**: The source falls back to `general`, so the assertion would fail on the correct image.
+- **A third pod with the label of `roadmap-service` as a positive control.**: It would run under the label of the one allowed client and add a pod and a policy path that nobody asked for. The forward already proves that the service answers, and an exec that fails for another reason raises `ServiceProbeError` and does not read as `blocked`.
+- **Read the kept engines from the ConfigMap file.**: The command would depend on the path of the repository. The reply shows the engines that run, and the list of those that stay off is short and fixed.
+- **A second copy of the forward and pod code.**: `service_probe.py` was written for this reuse.
+- **A new scanner.**: No open, maintained tool checks a running SearXNG for these points.