[Syed-Ali-Abbas-568]

Summary

tutor k8s launch crashes the MongoDB pod on a fresh volume:

"msg":"Error creating journal directory","error":"...Permission denied...\"/data/db/journal\""

Root cause: MongoDB (and MySQL, Redis, Meilisearch) run as non-root, but the PersistentVolume directory a storage provisioner hands them is root-owned. The pod-level fsGroup is meant to bridge that gap, but it's silently skipped on volume types that don't implement it — hostPath / local-path, i.e. what Minikube and kind use. Real cloud block storage (EBS, Azure Disk, GCP PD) implements fsGroup correctly, which is why this has sat open without affecting production users.

Builds on the diagnosis @HammadYousaf01 posted on the issue.

What this does

Adds a non-root init container to each of the four affected Deployments (mongodb, mysql, redis, meilisearch) that fixes volume ownership before the main container starts:

• The logic lives in one script, tutor/templates/apps/permissions/fix-volume-permissions.sh, delivered via kustomize's configMapGenerator — the same mechanism already used for redis.conf and the Caddyfile.
• It checks the volume's group (not owner) before doing anything: on a volume where fsGroup already did its job (real cloud storage ends up root:<gid>), this is a no-op. This matters because the owner never changes on those volumes — an owner-based check would recursively chown a live, working production volume on every single pod restart.
• The container runs as the service's own non-root uid (runAsNonRoot: true), not root. It gets the ability to chown via a file capability baked into the permissions image at build time (setcap cap_chown=ep /bin/busybox), not via the pod's capabilities.add — Kubernetes does not support ambient capabilities (see kubernetes/kubernetes#56374), so a pod-level capability grant alone is silently dropped for a non-root container. This was caught in review — the first version of this PR did use capabilities.add alone and didn't actually work. A file capability is a property of the binary rather than the pod spec, so it's unaffected by that gap, and keeps working with allowPrivilegeEscalation: false. It has to be set on /bin/busybox itself, not a narrower dedicated copy used only for the chown step — also tried and reverted, see commit history for why. This means the openedx-permissions image needs to be rebuilt for this fix to take effect — already-mirrored or already-pulled copies predate the setcap call, and the capability lives in a security.capability xattr that some image re-layering tools strip silently.
• A failed or hung chown is non-fatal: wrapped in timeout 60, and any failure just logs a warning instead of blocking the pod. This bounds a merely-slow/busy chown; it can't do anything about a process stuck in an uninterruptible kernel I/O wait (e.g. a truly dead NFS server) — no userspace mechanism can preempt that. The existing non-k8s permissions fix has no timeout at all today, so this is strictly additive protection, not a complete guarantee.

Known limitation found during testing

The fix fully resolves the reported bug: a fresh, empty volume (the actual precondition in #1304 — a provisioner hands over a brand-new directory before mongod has created anything in it). It does not fully resolve a different scenario: a volume that already has mongod-created files from a prior successful run (e.g. journal/, diagnostic.data/, both 0700) whose ownership later gets reset to root while those subdirectories keep their restrictive mode. CAP_CHOWN permits changing a file's owner regardless of current ownership, but it doesn't grant permission to traverse into a directory you can't read — so the recursive chown fixes the top-level directory but can't descend into those two, and mongod still fails. This is a narrower, different precondition than the reported issue (it requires a volume that already has data, not a fresh one), so I didn't want to silently expand this PR's scope to fix it without flagging it first — happy to take a pass at it if that's wanted.

Test plan

• Verified the non-root + file-capability mechanism end-to-end against a real kind cluster, using the actual fix-volume-permissions.sh script and the exact pod securityContext from deployments.yml (runAsNonRoot, capabilities: {add: [CHOWN], drop: [ALL]}, allowPrivilegeEscalation: false): a fresh root-owned volume gets fixed, and a volume already at root:<gid> (simulating a working fsGroup) is left untouched — same mtime, same ownership.
• Rendered both templates through the real Jinja2 + PyYAML toolchain with every RUN_* flag on and off — valid YAML, correct initContainers/volumes wiring, no regressions when all four services are disabled.
• Full live repro through tutor k8s launch itself, on a real kind cluster, installing this branch and building/loading the updated permissions image: scaled mongodb down, reset its PVC directory on the node to a fresh, empty root:root 0755 (the issue's actual precondition), scaled back up — pod reaches 1/1 Running, mongod startup complete in the logs, no permission errors. Also reconfirmed the no-op case (root:999) in this same real deployment: ownership left unchanged, ready in 10s.
• make test-unit test-format test-types

Fixes #1304

[Faraz32123]

Thanks for digging into this. The diagnosis matches what we saw on #1304, and checking the volume's group rather than its owner is a nice touch for volumes where fsGroup works. A few things before this can go in:

Requested changes

1. Root init container on every k8s deployment. As written, every operator gets a runAsUser: 0 init container on their mongodb/mysql/redis/meilisearch pods, including clusters on EBS/GCE PD/Azure Disk where fsGroup already works. Clusters enforcing Pod Security Admission restricted (or Kyverno/Gatekeeper policies requiring runAsNonRoot) will reject these pods, so a fix for Minikube/kind could take down the databases on production clusters. Could this be opt-in, e.g. a K8S_FIX_VOLUME_PERMISSIONS setting (default false) that wraps the initContainers block (and the ConfigMap), documented for Minikube/kind users?
2. Test plan. The live kind repro and make test-unit test-format test-types are still unchecked, and the description says "marking as draft", but the PR isn't a draft. Could you confirm the live repro before review?

Nits

3. Partial chown after a timeout. I believe busybox chown -R changes the top directory before recursing (worth confirming). If the 60s timeout kills it partway, the top directory already has the new gid, so every later restart reports "nothing to do" and the volume stays partly root-owned. Changing ownership of the contents first and the top directory last (or checking more than the top directory) would avoid that.
4. Hidden failure. On failure the script exits 0, so the main container then crashes with the original "Permission denied" and the real cause is only in the init container logs. Fine as a design choice, but worth a line in the k8s troubleshooting docs.
5. Changelog / air-gapped clusters. The changelog should say that init containers are added to all k8s database deployments (not only Minikube/kind ones), and that air-gapped clusters will also need to mirror DOCKER_IMAGE_PERMISSIONS.

Question: target branch. This is a bug fix, but it currently targets main. Per the contributing guidelines, fixes with no regression risk go to release and changes that could disrupt production operators go to main. Was main deliberate because of the risk in (1)? If it becomes opt-in, would it make sense to retarget release so current users get the fix?

[Syed-Ali-Abbas-568]

/smoke

[github-actions]

Smoke tests requested by @Syed-Ali-Abbas-568, on de58d43:

Mode	Result	Tests
local	✅ passed	24 passed, 0 failed, 1 skipped
k8s	❌ failed	20 passed, 4 failed, 1 skipped

Plugins: indigo mfe, images: openedx. Workflow run (logs are attached to failed runs)

[Syed-Ali-Abbas-568]

/smoke

[github-actions]

Smoke tests requested by @Syed-Ali-Abbas-568, on 6598c8d:

Mode	Result	Tests
local	✅ passed	24 passed, 0 failed, 1 skipped
k8s	✅ passed	24 passed, 0 failed, 1 skipped

Plugins: indigo mfe, images: openedx. Workflow run (logs are attached to failed runs)

[Faraz32123]

Thanks for the updates! Retargeting release, the post-order chown, the troubleshooting docs and the changelog all look good.

One concern with the new non-root approach: Kubernetes doesn't support ambient capabilities (kubernetes/kubernetes#56374), so capabilities.add: ["CHOWN"] on a container running as uid 999/1000 is dropped at execve: chown should fail with EPERM on a root-owned volume, and the script then just logs the warning. I don't think the green k8s smoke test covers this, since kind's local-path volumes are created 0777 and the services start either way. Could you do the live repro from the test plan (volume reset to root:root 0755 on Minikube/kind) and confirm the init container actually changes ownership?

If it does need root, I'd still suggest making the init container opt-in (e.g. K8S_FIX_VOLUME_PERMISSIONS, default false, documented for Minikube/kind), since PSA restricted rejects both root containers and added capabilities. That also makes release an easier target.

Could you also update the PR description? It still mentions a root init container, the draft status and unchecked test items.

[Syed-Ali-Abbas-568]

Hi @Faraz32123 , I tested this on a real cluster and capabilities.add definitely drops at execve when running as non-root. Because of that, chown was just failing silently this whole time.

To fix it, I updated the image build to use setcap cap_chown=ep on busybox instead. Since it's a file capability, it completely avoids the ambient-cap issue. It does mean we need an image rebuild for it to take effect, which I've noted in the PR.

I just re-ran the script on a kind cluster with this fix, and it works perfectly: it fixes fresh root-owned volumes, ignores ones that are already correct, and surprisingly, it still works even with allowPrivilegeEscalation: false.

I decided to skip adding the opt-in flag, since running non-root with this capability should be enough to satisfy Kyverno/Gatekeeper. It still won't pass a fully restricted PSA profile (since that only allows NET_BIND_SERVICE), but our current setup doesn't either, so no regressions there.

I've updated the PR description with all of this as well.

[Faraz32123]

Thanks, the file-capability approach makes sense, and the allowPrivilegeEscalation: false result checks out: no_new_privs only caps the file caps at the pre-exec permitted set, which still contains CHOWN thanks to capabilities.add, so both pieces are needed. I'm fine with skipping the opt-in given the init container is now non-root and baseline-compatible.

Before approving:

1. Could you re-run /smoke on bb77a91? The last run was on 6598c8d, before the image change.
2. Could you tick the full tutor k8s launch repro (fresh mongodb volume reset to root:root 0755)? kind's local-path volumes are 0777, so the green smoke test doesn't really exercise this fix.

Nits:

• setcap on /bin/busybox gives CAP_CHOWN to every applet in the image (sh included) whenever the pod allows it. Setting it on a dedicated copy (e.g. /usr/local/bin/chown-busybox) and calling that from the script would narrow it to the chown.
• File caps live in the security.capability xattr, which some mirroring/re-layering tools strip. Worth a sentence next to the air-gapped note in the changelog, since that would silently bring back the original crash.

[Syed-Ali-Abbas-568]

/smoke

[github-actions]

Smoke tests requested by @Syed-Ali-Abbas-568, on bb77a91:

Mode	Result	Tests
local	✅ passed	24 passed, 0 failed, 1 skipped
k8s	✅ passed	24 passed, 0 failed, 1 skipped

Plugins: indigo mfe, images: openedx. Workflow run (logs are attached to failed runs)

[Syed-Ali-Abbas-568]

I tested out the dedicated binary approach on a real cluster, but unfortunately, it's still throwing EPERM errors. Given how our script is structured, I don't think this method is actually going to work for us.

The issue comes down to allowPrivilegeEscalation: false. With that set, a file capability can only elevate a process up to whatever was already in its permitted set before the exec. Because our script runs through sh, by the time it calls chown, it's doing a second exec from a shell that's already unprivileged. Its permitted set is completely empty, so it doesn't matter which binary we point it to—there are no privileges left to grant.

I ended up reverting back to running setcap directly on /bin/busybox. I re-tested it and everything is working smoothly again. I also made sure to add the xattr warning to the changelog!

[Syed-Ali-Abbas-568]

/smoke

[github-actions]

Smoke tests requested by @Syed-Ali-Abbas-568, on 1936c59:

Mode	Result	Tests
local	✅ passed	24 passed, 0 failed, 1 skipped
k8s	✅ passed	24 passed, 0 failed, 1 skipped

Plugins: indigo mfe, images: openedx. Workflow run (logs are attached to failed runs)

[Faraz32123]

Thanks for the thorough follow-up! Everything I asked for is covered:

• /smoke passes on 1936c59 (local and k8s), and the full tutor k8s launch repro on a fresh root:root 0755 mongodb volume is done, including the no-op root:999 case.
• The explanation for keeping the capability on /bin/busybox makes sense: with allowPrivilegeEscalation: false, the later chown exec can only keep what the sh process already holds, so sh itself has to be the capable file. Thanks for documenting that in the Dockerfile.
• The changelog now covers the security.capability xattr risk for mirrored images.

LGTM 👍