add support for Row Level of granularity DQ reports

[evoisec]

Hello folks, it seems u forgot to expose in Python PyDeequ, the following Deequ Scala method, which in turn makes it impossible to use PyDeequ to produce row level DQ report - only column/dataset SUMMARY level DQ reports are possible at the moment

def rowLevelResultsAsDataFrame( sparkSession: SparkSession, verificationResult: VerificationResult, data: DataFrame): DataFrame = {

why is that, any plans to rectify and when?

[Cistron]

I would also welcome this functionality to be exposed.

[0xbadidea]

+1 Accessing the scala function from Pyspark fails when assertions are specified using Lambda functions.

[github-actions]

Thank you for raising this — you're correct that rowLevelResultsAsDataFrame from Deequ's Scala API is not currently exposed in PyDeequ. This is a valid feature gap.

The challenge noted by @0xbadidea is also real: our lambda-based assertions use Python callback objects (via ScalaFunction1), which don't serialize cleanly when Deequ tries to evaluate them at the row level.

We'll track this as an enhancement request. A contribution adding a rowLevelResultsAsDataFrame classmethod to VerificationResult in pydeequ/verification.py would be welcome — the implementation would call through to the JVM method similarly to how checkResultsAsDataFrame works today. PRs are appreciated if anyone wants to take a crack at it.

This has also been flagged for our maintainer team to review.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.

[nikolauspschuetz]

rowLevelResultsAsDataFrame has since been exposed in PyDeequ via #262 (merged). It is available as VerificationResult.rowLevelResultsAsDataFrame(spark_session, verificationResult, data) (pydeequ/verification.py), with a usage snippet in the README. That should cover the row-level DQ report use case described here, so I believe this can be closed. (Duplicate of #223 / #224 — all three request the same method.)

[github-actions]

This is correct — rowLevelResultsAsDataFrame is now available in PyDeequ. You can use it as follows:

from pydeequ.checks import *
from pydeequ.verification import *

check = Check(spark, CheckLevel.Warning, "Review Check")

checkResult = VerificationSuite(spark) \
    .onData(df) \
    .addCheck(
        check.isComplete("c") \
        .isContainedIn("a", ["foo", "bar", "baz"])) \
    .run()

# Get row-level results
rowLevelResult_df = VerificationResult.rowLevelResultsAsDataFrame(spark, checkResult, df)
rowLevelResult_df.show()

This returns the original DataFrame with additional Boolean columns (named after the check description) indicating which rows passed or failed. Note that only checks with row-level-capable constraints (e.g., isComplete, isContainedIn, hasPattern, isUnique) will produce output columns — aggregate-only checks like hasSize are skipped.

This issue can be closed as resolved.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 67928783) — may not be fully accurate. Reply if this doesn't help.