Harden the public demo: no metrics or docs, security headers

CHANGELOG.md

@@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Security
+
+- `create_app(..., public=True)` (or `PGQUEUE_ADMIN_PUBLIC=1`) for instances shown to the public:
+  `/metrics` and the API docs (`/docs`, `/redoc`, `/openapi.json`) are not served. The live demo
+  uses it.
+- The demo's nginx config now answers 404 for those paths and sends a Content-Security-Policy,
+  `X-Content-Type-Options`, `X-Frame-Options`, `Referrer-Policy` and `Permissions-Policy`
+  (`deploy/termux/nginx-headers.conf`).
+- The dashboard no longer writes `style` attributes (bar widths are set through the DOM), so a
+  policy without `'unsafe-inline'` works.
+
 ### Added
 
 - `make bench`: a reproducible benchmark with a 1/2/4/8 worker sweep, enqueue and pickup latency

README.md

@@ -243,6 +243,22 @@ The public dashboard exposes **no endpoint that changes state**: in read-only mo
 mounted at all, and the buttons are not rendered. Settings are environment variables, listed in
 [`deploy/termux/env.example`](deploy/termux/env.example).
 
+**What a visitor can and cannot do.** The demo has no input path for code or URLs. Every job
+is built by the demo's own generator from fixed task types and numbers, and the "webhook"
+tasks only sleep and fail on a schedule: nothing makes an outgoing request. The public app
+accepts GET requests only (the dashboard page, static files and the read-only `/api/*` JSON
+and event stream); the only parameters are a job status and a row limit on `/api/jobs`,
+validated and passed to SQL as bind values. In addition the demo runs in **public mode**
+(`create_app(dsn, public=True)` or `PGQUEUE_ADMIN_PUBLIC=1`), which leaves out `/metrics`
+(worker ids, queue names, failure counts) and the API docs (`/docs`, `/redoc`,
+`/openapi.json`) entirely; nginx answers 404 for the same paths as well.
+
+**Retention.** The tables cannot grow without bound: a janitor every 20 s deletes finished
+jobs older than 10 minutes and anything beyond the newest 6000 finished rows, and forgets
+dead workers after 30 s. Live jobs are limited by the generator's backpressure (it pauses
+above 1500 waiting jobs), and `pgqueue_periodic` holds two rows. Tune with
+`DEMO_RETENTION_SECONDS`, `DEMO_MAX_FINISHED` and `DEMO_MAX_BACKLOG`.
+
 ```console
 $ export PGQUEUE_DSN=postgresql://user:[email protected]/pgqueue_demo
 $ python -m demo            # migrates, then serves http://127.0.0.1:8080
@@ -271,6 +287,18 @@ $ deploy/termux/start.sh                                     # try it by hand fi
   disk, disables buffering for the event stream, limits requests and streams per visitor
   (keyed on `CF-Connecting-IP` behind a Cloudflare Tunnel, where every request comes from
   `127.0.0.1`) and caps the streams in total.
+- **Security headers**: `nginx-location.conf` includes `nginx-headers.conf` in every location
+  (nginx drops inherited `add_header` lines once a location sets its own). It sends a
+  Content-Security-Policy with everything restricted to `'self'` (no inline script or style: the
+  dashboard builds its styles through the DOM, and the event stream is a same-origin
+  `EventSource`), `nosniff`, `frame-ancestors 'none'` with `X-Frame-Options: DENY`,
+  `Referrer-Policy` and `Permissions-Policy`. `tests/test_deploy.py` checks the dashboard
+  against the policy.
+- **Updating an existing deployment**: pull, restart the demo (so `PGQUEUE_ADMIN_PUBLIC` takes
+  effect), replace the server's copy of `nginx-location.conf` with the new one (it now
+  includes `nginx-headers.conf`, which must exist at the path in the `include` lines, by
+  default `~/pgqueue/deploy/termux/nginx-headers.conf`), then `nginx -t && nginx -s reload`.
+  Nothing else changes: `nginx-zones.conf` and `~/.config/pgqueue-demo/env` stay as they are.
 - **Public access**: put a Cloudflare Tunnel or Tailscale Funnel in front of nginx; the demo
   itself only listens on `127.0.0.1`.
 - **Database**: the demo uses its own role and database and works with PostgreSQL 16 to 18

demo/server.py

@@ -25,7 +25,7 @@ def build_app(settings: Settings, *, background: bool = True) -> FastAPI:
     With ``background=False`` only the dashboard is served (used by tests and when the
     workload runs elsewhere).
     """
-    app = create_app(settings.dsn, read_only=True, banner=settings.banner)
+    app = create_app(settings.dsn, read_only=True, public=True, banner=settings.banner)
     inner = app.router.lifespan_context
 
     @asynccontextmanager

deploy/termux/env.example

@@ -20,6 +20,8 @@ DEMO_MAX_BACKLOG=1500
 # A worker is SIGKILLed this often (seconds) to show lease recovery; 0 disables it.
 DEMO_CRASH_EVERY=150
 
+# The demo always runs in public mode: no /metrics, no /docs, no /openapi.json.
+
 # Retention: finished jobs older than this, or beyond this many rows, are deleted.
 DEMO_RETENTION_SECONDS=600
 DEMO_MAX_FINISHED=6000

deploy/termux/nginx-headers.conf

@@ -0,0 +1,9 @@
+# Security headers, repeated in every location that sets its own add_header (nginx drops the
+# inherited ones there). The dashboard loads only its own script and stylesheet and opens
+# one same-origin EventSource, so everything is 'self'; a test in tests/test_deploy.py checks
+# the page against this policy.
+add_header Content-Security-Policy "default-src 'self'; script-src 'self'; style-src 'self'; img-src 'self' data:; connect-src 'self'; font-src 'self'; frame-ancestors 'none'; base-uri 'self'; form-action 'self'; object-src 'none'" always;
+add_header X-Content-Type-Options "nosniff" always;
+add_header Referrer-Policy "strict-origin-when-cross-origin" always;
+add_header X-Frame-Options "DENY" always;
+add_header Permissions-Policy "camera=(), microphone=(), geolocation=(), payment=(), usb=(), interest-cohort=()" always;

deploy/termux/nginx-location.conf

@@ -1,23 +1,39 @@
 # Goes inside the server {} block. The dashboard is then at https://<host>/pgqueue/
 # (DEMO_ROOT_PATH=/pgqueue in the env file; the port must match DEMO_PORT).
 
-location = /pgqueue { return 301 /pgqueue/; }
+# Security headers: every location includes nginx-headers.conf itself, because nginx drops
+# inherited add_header lines as soon as a location sets one of its own. Adjust the path to
+# your checkout (as for the alias lines below).
+location = /pgqueue {
+    include /data/data/com.termux/files/home/pgqueue/deploy/termux/nginx-headers.conf;
+    return 301 /pgqueue/;
+}
+
+# Defence in depth: the app (PGQUEUE_ADMIN_PUBLIC, set by the demo) does not serve these at all,
+# but the Prometheus metrics, the OpenAPI schema and the docs pages must never be public.
+location ~ ^/pgqueue/(metrics|docs|redoc|openapi\.json)/?$ {
+    include /data/data/com.termux/files/home/pgqueue/deploy/termux/nginx-headers.conf;
+    return 404;
+}
 
 # Static assets straight from disk: adjust the path to your checkout.
 location /pgqueue/static/ {
     alias /data/data/com.termux/files/home/pgqueue/src/pgqueue/admin/static/;
     expires 1h;
     add_header Cache-Control "public";
+    include /data/data/com.termux/files/home/pgqueue/deploy/termux/nginx-headers.conf;
 }
 
 # Browsers ask for /favicon.ico (and iOS for the touch icon) at the host root, without reading the page.
 location = /favicon.ico {
     alias /data/data/com.termux/files/home/pgqueue/src/pgqueue/admin/static/favicon.ico;
     expires 1d;
+    include /data/data/com.termux/files/home/pgqueue/deploy/termux/nginx-headers.conf;
 }
 location = /apple-touch-icon.png {
     alias /data/data/com.termux/files/home/pgqueue/src/pgqueue/admin/static/apple-touch-icon.png;
     expires 1d;
+    include /data/data/com.termux/files/home/pgqueue/deploy/termux/nginx-headers.conf;
 }
 
 # The live event stream: long-lived, unbuffered, capped per visitor and in total.
@@ -33,6 +49,7 @@ location = /pgqueue/api/stream {
     proxy_cache off;
     proxy_read_timeout 1h;
     gzip off;
+    include /data/data/com.termux/files/home/pgqueue/deploy/termux/nginx-headers.conf;
 }
 
 location /pgqueue/ {
@@ -43,4 +60,5 @@ location /pgqueue/ {
     proxy_set_header Host $host;
     proxy_set_header X-Forwarded-For $remote_addr;
     proxy_set_header X-Forwarded-Proto $scheme;
+    include /data/data/com.termux/files/home/pgqueue/deploy/termux/nginx-headers.conf;
 }

src/pgqueue/admin/api.py

@@ -208,6 +208,7 @@ def create_app(
     password: str | None = None,
     read_only: bool | None = None,
     banner: str | None = None,
+    public: bool | None = None,
 ) -> FastAPI:
     """Build the admin application.
 
@@ -218,11 +219,18 @@ def create_app(
     ``read_only`` (default: ``PGQUEUE_ADMIN_READ_ONLY``) leaves out every
     endpoint that changes state, so the dashboard can be shown to the public.
     ``banner`` is an optional line of text shown at the top of the dashboard.
+
+    ``public`` (default: ``PGQUEUE_ADMIN_PUBLIC``) is for an instance shown to anyone on
+    the internet: ``/metrics`` (worker ids, queue names, failure counts) and the API docs
+    (``/docs``, ``/redoc``, ``/openapi.json``) are not served at all. The dashboard and its
+    ``/api`` endpoints, which the page itself needs, stay.
     """
     username = username or os.environ.get("PGQUEUE_ADMIN_USERNAME")
     password = password or os.environ.get("PGQUEUE_ADMIN_PASSWORD")
     if read_only is None:
         read_only = os.environ.get("PGQUEUE_ADMIN_READ_ONLY", "").lower() in ("1", "true", "yes")
+    if public is None:
+        public = os.environ.get("PGQUEUE_ADMIN_PUBLIC", "").lower() in ("1", "true", "yes")
     engine = create_async_engine(sqlalchemy_url(dsn), pool_size=5, pool_pre_ping=True)
     feed = LiveFeed(engine, read_only=read_only)
 
@@ -232,7 +240,14 @@ async def lifespan(_: FastAPI) -> AsyncIterator[None]:
         await feed.close()
         await engine.dispose()
 
-    app = FastAPI(title="pgqueue admin", version=__version__, lifespan=lifespan)
+    app = FastAPI(
+        title="pgqueue admin",
+        version=__version__,
+        lifespan=lifespan,
+        docs_url=None if public else "/docs",
+        redoc_url=None if public else "/redoc",
+        openapi_url=None if public else "/openapi.json",
+    )
     app.state.engine = engine
     app.state.feed = feed
     app.state.credentials = (username, password) if username and password else None
@@ -258,16 +273,18 @@ async def dashboard(request: Request) -> HTMLResponse:
             {"version": __version__, "read_only": read_only, "banner": banner},
         )
 
-    @app.get(
-        "/metrics",
-        include_in_schema=False,
-        dependencies=[Depends(_authenticate)],
-        response_class=PlainTextResponse,
-    )
-    async def metrics() -> PlainTextResponse:
-        async with engine.connect() as conn:
-            body = await render_metrics(conn)
-        return PlainTextResponse(body, media_type=CONTENT_TYPE_LATEST)
+    if not public:
+
+        @app.get(
+            "/metrics",
+            include_in_schema=False,
+            dependencies=[Depends(_authenticate)],
+            response_class=PlainTextResponse,
+        )
+        async def metrics() -> PlainTextResponse:
+            async with engine.connect() as conn:
+                body = await render_metrics(conn)
+            return PlainTextResponse(body, media_type=CONTENT_TYPE_LATEST)
 
     @app.get("/healthz", include_in_schema=False)
     async def healthz() -> dict[str, str]:

src/pgqueue/admin/static/dashboard.js

@@ -27,6 +27,7 @@
     const node = document.createElement(tag);
     for (const [key, value] of Object.entries(attrs)) {
       if (key === "class") node.className = value;
+      else if (key === "style") Object.assign(node.style, value); // CSSOM, not a style attribute (CSP)
       else if (key.startsWith("on")) node.addEventListener(key.slice(2), value);
       else if (value !== null && value !== undefined) node.setAttribute(key, value);
     }
@@ -244,7 +245,7 @@
           el("span", {}, `failed ${fmt.format(w.failed)}`),
           el("span", {}, w.queues.join(", "))),
         el("div", { class: "busy" },
-          el("div", { class: "track" }, el("div", { class: "fill", style: `width:${pct}%` })),
+          el("div", { class: "track" }, el("div", { class: "fill", style: { width: `${pct}%` } })),
           el("span", { class: "label" }, `${w.in_flight}/${w.concurrency} slots`)));
     });
     $("workers").replaceChildren(...(cards.length ? cards : [el("p", { class: "empty" }, "No workers running")]));
@@ -266,7 +267,7 @@
         el("td", { class: "num" }, seconds((now - new Date(job.started_at)) / 1000)),
         el("td", {},
           el("div", { class: "busy" },
-            el("div", { class: "track" }, el("div", { class: `fill lease ${pct < 35 ? "low" : ""}`, style: `width:${pct}%` })),
+            el("div", { class: "track" }, el("div", { class: `fill lease ${pct < 35 ? "low" : ""}`, style: { width: `${pct}%` } })),
             el("span", { class: "label" }, left > 0 ? `${left.toFixed(0)}s` : "expired"))));
     });
     replaceRows($("leases"), rows, 7, "Nothing is running right now");

tests/test_deploy.py

@@ -0,0 +1,70 @@
+"""The nginx files that ship with the demo, checked against the dashboard they protect."""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parent.parent
+TERMUX = ROOT / "deploy" / "termux"
+ADMIN = ROOT / "src" / "pgqueue" / "admin"
+HEADERS = "include /data/data/com.termux/files/home/pgqueue/deploy/termux/nginx-headers.conf;"
+
+
+def csp() -> dict[str, list[str]]:
+    text = (TERMUX / "nginx-headers.conf").read_text()
+    match = re.search(r'Content-Security-Policy "([^"]+)"', text)
+    assert match
+    directives = {}
+    for part in match.group(1).split(";"):
+        name, *values = part.split()
+        directives[name] = values
+    return directives
+
+
+def test_the_policy_allows_only_the_dashboards_own_resources() -> None:
+    policy = csp()
+    for name in ("default-src", "script-src", "style-src", "connect-src"):
+        assert policy[name] == ["'self'"], name
+    assert policy["frame-ancestors"] == ["'none'"]
+    assert policy["object-src"] == ["'none'"]
+    assert not any("unsafe" in v for values in policy.values() for v in values)
+
+
+def test_the_dashboard_needs_nothing_the_policy_forbids() -> None:
+    html = (ADMIN / "templates" / "dashboard.html").read_text()
+    script = (ADMIN / "static" / "dashboard.js").read_text()
+    css = (ADMIN / "static" / "dashboard.css").read_text()
+
+    assert not re.search(r"<script(?![^>]*\bsrc=)", html), "inline script"
+    assert not re.search(r"<style", html), "inline style element"
+    assert not re.search(r"\sstyle=", html), "inline style attribute"
+    assert not re.search(r"\son[a-z]+=", html), "inline event handler"
+    assert not re.search(r"\sstyle=|\bon[a-z]+=\"", script), "inline style or handler built in JS"
+    assert not re.search(r"\bstyle:\s*[`\"']", script), "style attribute set from a string"
+    assert not re.search(r"setAttribute\(\s*[\"']style", script), "style attribute set from JS"
+    # Links to other sites are fine; anything the page loads or connects to is not.
+    assert not re.search(r"<(?:script|img|link|iframe)[^>]+(?:src|href)=\"https?://", html)
+    assert not re.search(r"(?:fetch|EventSource|import)\(\s*[\"'`]https?://", script)
+    assert not re.search(r"@import|url\(\s*['\"]?https?:", css), "external stylesheet resource"
+    assert not re.search(r"\beval\(|new Function\(", script)
+
+
+@pytest.mark.parametrize("path", ["/metrics", "/docs", "/redoc", "/openapi.json"])
+def test_nginx_answers_404_for_the_internal_endpoints(path: str) -> None:
+    text = (TERMUX / "nginx-location.conf").read_text()
+    match = re.search(r"location ~ (\^[^ ]+) \{[^}]*return 404", text)
+    assert match
+    assert re.match(match.group(1), "/pgqueue" + path)
+    assert re.match(match.group(1), "/pgqueue" + path + "/")
+    assert not re.match(match.group(1), "/pgqueue/api/live")
+
+
+def test_every_location_repeats_the_security_headers() -> None:
+    text = (TERMUX / "nginx-location.conf").read_text()
+    blocks = re.findall(r"^location [^\n]*\{(.*?)^\}", text, re.S | re.M)
+    assert len(blocks) >= 6
+    for block in blocks:
+        assert HEADERS in block

tests/test_metrics.py

@@ -2,7 +2,10 @@
 
 from collections.abc import AsyncIterator
 
+import pytest
 import pytest_asyncio
+from demo.config import Settings
+from demo.server import build_app
 from fastapi import FastAPI
 from prometheus_client import CollectorRegistry
 from prometheus_client.parser import text_string_to_metric_families
@@ -88,3 +91,36 @@ def value(name: str, **labels: str) -> float | None:
         value("pgqueue_worker_attempt_duration_seconds_count", task="ok", outcome="succeeded") == 2
     )
     assert value("pgqueue_worker_concurrency") == worker.concurrency
+
+
+async def test_public_mode_serves_no_metrics_and_no_api_docs(clean_db: str) -> None:
+    app = create_app(clean_db, public=True)
+    client = Client(app)
+    async with app.router.lifespan_context(app):
+        statuses = {
+            path: (await client.get(path)).status
+            for path in ("/metrics", "/docs", "/redoc", "/openapi.json")
+        }
+        dashboard = await client.get("/")
+        live = await client.get("/api/live")
+
+    assert statuses == {"/metrics": 404, "/docs": 404, "/redoc": 404, "/openapi.json": 404}
+    assert dashboard.status == 200
+    assert live.status == 200
+
+
+async def test_public_mode_can_come_from_the_environment(
+    clean_db: str, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    monkeypatch.setenv("PGQUEUE_ADMIN_PUBLIC", "1")
+    app = create_app(clean_db)
+    async with app.router.lifespan_context(app):
+        assert (await Client(app).get("/metrics")).status == 404
+
+
+async def test_the_demo_does_not_expose_metrics_or_docs(clean_db: str) -> None:
+    app = build_app(Settings(dsn=clean_db), background=False)
+    client = Client(app)
+    async with app.router.lifespan_context(app):
+        for path in ("/metrics", "/docs", "/redoc", "/openapi.json"):
+            assert (await client.get(path)).status == 404, path