Fix duplicate XLA compilation of jit(train_step)

[olupton]

Description

The AoT compilation used to extract memory stats previously caused a redundant compilation due to mismatched context managers. Skip the memory stats entirely when AutoPGLE is enabled as there is no nice way of getting the statistics up-front without causing redundant compilation.

Tests

With https://docs.jax.dev/en/latest/config_options.html#jax_log_compiles this can be seen to reduce the number of training step compilations from 2 to 1 without AutoPGLE, or from 3 to 2 with AutoPGLE.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[gemini-code-assist]

Code Review

This pull request removes the legacy mesh context manager from the training loop to prevent double compilation of train_step, and skips ahead-of-time compilation when AutoPGLE is enabled to avoid redundant compilations. The review feedback suggests using getattr when checking jax.config.jax_enable_pgle to prevent potential AttributeError crashes on older JAX versions or unsupported platforms.

[NuojCheng]

@khatwanimohit is the diloco implementation still blocked by this?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!