Skip to content

fix(python): read the target's interpreter in austin memory profiles - #137

Merged
blue4209211 merged 2 commits into
mainfrom
fix/austin-attach-target-fs
Oct 3, 2026
Merged

blue4209211 merged 2 commits into
mainfrom
fix/austin-attach-target-fs

Conversation

@mayankpande88

@mayankpande88 mayankpande88 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Python memory profiles (austin) failed for almost every target with:

Cannot determine the version of the Python interpreter. This could be due to the binary not being an actual Python binary, like uWSGI, or a version that this version of Austin does not support.

Cause. austin finds the interpreter, and libpython for a shared build, by the path it reads from /proc/<pid>/maps. It then opens that path in its own mount namespace, not under /proc/<pid>/root. The agent runs in its own container, so that path is one of two things:

  • Missing. Any target whose interpreter isn't at a path the profiler image has (e.g. /usr/local/bin/python3.12) fails with the error above.
  • The profiler's own interpreter. A python:3.14 target was read through the image's python3.14 instead of its own.

Fix. Run austin in a private mount namespace (unshare -m --propagation private) in which the target's interpreter and libpython sit at the paths austin looks up.

  • The kernel won't bind-mount a source from another mount namespace (EINVAL). So each file is first copied out, and the copy is bound. The copy comes from a magic link to the inode the process actually mapped: /proc/<pid>/exe for the interpreter, /proc/<pid>/map_files/<range> for libpython. A file replaced on disk since the process started is therefore still read as mapped.
  • The per-PID copies are removed once austin exits.
  • Neither the mounts nor the shadowing of the image's own Python escape the namespace, so mojo2austin, which runs on that Python, is unaffected.

Two follow-on fixes in the same path:

  • Steady-state processes. austin's memory mode only records allocations that grow the process. A process at steady state therefore produced a header-only artefact that looked like a broken profile. A raw profile now carries # no memory growth observed during the <N>s window: … and is still published. Any other output type fails with that explanation instead of "no stacks found (maybe due low cpu load)".
  • Raw profiles are .txt. GetFileExtension had no austin case, so raw output fell through to .svg (agent-raw-<pid>-1.svg.gz). Consumers that pick a renderer by suffix drew the text as a broken image.
  • Readable errors. They used to be austin's full stderr: an ANSI-coloured ASCII-art banner followed by the message. Now they read austin (PID <n>, <binary>): <the sentence that explains it> (exit status N).

Also removes TestAustin. It profiled a hard-coded PID and then slept for 300 seconds, which added five minutes to every CI run.

Type of change

  • Bug fix
  • CI / build

Test plan

  • Unit: resolving the target files from a fake /proc tree; the unshare wrapper's arguments; cleaning austin's real stderr; sample counting; the no-growth handling for raw vs flamegraph output.
  • New CI job python-austin-e2e (test/e2e/python-austin.sh). It builds the python image and runs the agent from it against real Python containers with --pid=container:<target> --privileged, the same split view (target's PIDs, own filesystem) a debugger pod has. It asserts samples are collected and that the # python: header matches the target's own version.
Engineering detail

Local e2e: built docker/python/Dockerfile from this branch, then ran test/e2e/python-austin.sh against Python 3.12 (Alpine), 3.12 (Debian, shared libpython), 3.14 (Alpine, the same path as the image's own interpreter) and a steady-state 3.12 process. Results:

Target Released image This branch
Python 3.12 (Alpine) fails with the error above 411 samples, # python: 3.12.15
Python 3.12 (Debian) fails with the error above 384 samples, # python: 3.12.15
Python 3.14 (Alpine) fails with the error above 445 samples, # python: 3.14.8 (the target's version, not the image's 3.14.0)
Steady-state 3.12 fails with the error above published with the no-growth note
Non-Python target (/bin/busybox) full banner as the error austin (PID 1, /bin/busybox): Cannot determine the version of the Python interpreter. … (exit status 10)

A process whose interpreter was deleted after it started (/tmp/py3 (deleted) in maps) profiled correctly: 450 samples, # python: 3.12.15. No austin-target-* copy directory was left behind.

The mount namespace was checked from outside after each run: no leaked mounts, and the image's python3.14 still reports 3.14.0.

go build ./..., go vet ./... and golangci-lint (changed files) are clean. go test ./... passes on Linux, except pkg/util/file TestWrite when run as root in a container; that test is untouched here.

Checklist

  • go build ./... and go test ./... pass locally
  • Docs (README, flags) updated if behavior changed: no flag or doc change
  • No breaking changes to the agent's stdout JSON envelope. The only artefact change is the extra # comment line on a no-growth raw profile.

austin finds the interpreter (and libpython, for a shared build) by the
path in /proc/<pid>/maps and opens that path in its own mount namespace.
In the agent's container that path is either missing — "Cannot determine
the version of the Python interpreter" for any target whose interpreter
the image lacks — or the image's own interpreter.

Run austin in a private mount namespace with copies of the target's
interpreter files bound at those paths. A bind straight from
/proc/<pid>/root is refused across mount namespaces, hence the copy.

Also:
- annotate a raw memory profile with no samples as "no memory growth"
  instead of publishing a header-only artefact; other outputs fail with
  that reason
- reduce austin errors to the sentence that explains them, with the PID
  and binary
- drop TestAustin, which profiled a fixed PID and slept 300s
- add a CI job that profiles real Python 3.12/3.14 containers with the
  built image

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Austin Python profiler to run within a private mount namespace using unshare, copying and binding the target's interpreter and libpython files from /proc//root to resolve version determination issues in containerized environments. It also adds robust error parsing, sample counting, and comprehensive tests. The review feedback highlights a critical need to clean up the temporary files in /tmp to prevent disk space exhaustion, and offers valuable suggestions to handle deleted library paths, optimize sample counting by avoiding string allocations, and simplify file appending with deferred closes.

Comment thread internal/agent/profiler/python_austin.go
Comment thread internal/agent/profiler/python_austin.go
Comment thread internal/agent/profiler/python_austin.go
Comment thread internal/agent/profiler/python_austin.go
RamanKharchee
RamanKharchee previously approved these changes Oct 3, 2026
…ean up

Address review:
- read the interpreter through /proc/<pid>/exe and libpython through
  /proc/<pid>/map_files/<range> instead of /proc/<pid>/root<path>, so a
  file replaced on disk since the process started ("(deleted)" in maps)
  is read as mapped rather than from its replacement
- remove the per-PID copy directory once austin exits
- count samples on scanner.Bytes() instead of allocating each line
@blue4209211
blue4209211 merged commit 6116ac0 into main Oct 3, 2026
@blue4209211
blue4209211 deleted the fix/austin-attach-target-fs branch October 3, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants