[tlfua]

Which issue does this PR close?

Part of #6187. This addresses item 2 and the remaining wording in item 1. Item 3 (tests that use the fraction to force spills) is left open; see the note below.

Rationale for this change

The deprecation warning for spark.comet.exec.memoryPool.fraction only checks the SparkConf in the driver plugin, but CometExecIterator.getMemoryConfig reads the value from the session conf for each query. A value set with SET or spark.conf.set, or in an application that registers the session extension without the plugin, takes effect without a warning.

Item 1 is mostly resolved by #7096 and the guide rewrites that followed it. The fair_unified pool now limits each task to min(spark.memory.offHeap.size / running_tasks, pool_size), and both guides and the config doc already describe the fraction as a per-task cap. The user guide still did not say what the cap is.

What changes are included in this PR?

• CometDriverPlugin.warnIfMemoryPoolFractionSet takes the value to check and logs the warning at most once per JVM.
• The driver plugin passes the SparkConf value, as before. CometExecRule passes the session conf value of each query, so SET, spark.conf.set and extension-only sessions are covered too. Because of the once-per-JVM flag, a session that sets the fraction after it has already been warned about gets no second warning.
• The memory tuning guide states that fair_unified limits each task to the fraction times spark.memory.offHeap.size.

The migration guide is unchanged: the 1.1.0 note that the driver logs a warning still holds, and this change adds no behavior change.

Item 3 is not addressed here. The fraction is set to force spills in more places than CometTaskMetricsSuite: CometAggregateSuite, CometExecSuite, CometNativeSuite, CometParquetWriterSuite, CometIcebergWriteActionSuite and CometExecIteratorLifecycleSuite use it too. Whether to move the config to the testing category or replace it in tests is still open in the issue.

How are these changes tested?

CometPluginsMemoryPoolFractionWarningSuite gains a test that sets the fraction with withSQLConf, runs two queries and asserts the warning is logged exactly once. It fails without this change because only the SparkConf was checked. The existing tests now reset the once-per-JVM flag first.

• make format (Maven compile test-compile, scalafix and spotless): passed locally.
• npx prettier --check docs/source/user-guide/latest/tuning/memory.md: passed.
• The JVM test suites were not run locally. The fork CI passed: https://github.com/tlfua/datafusion-comet/actions/runs/42432188048