Repository navigation
feat(validators): add content_safety wrapper over uli_slur_match and profanity_free - #164
Conversation
…profanity_free Single validator that runs the lexical slur check, then ProfanityFree on the redacted text. Slur-first keeps the rest of the message when the only problem is a listed slur; ProfanityFree has no partial fix and would blank it. - Config exposes only slur_severity; LexicalSlur ignores `languages`. - Fixed error messages so user text isn't echoed into validator logs or rephrase responses (ProfanityFree's message embeds the full input). - Inner validators use NOOP on_fail; only the wrapper's on_fail applies. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ProfanityFree fails anything the alt-profanity-check model scores at 0.5 or more. The model is English-only and weights "ho" (as in "kaise ho") at 0.883 and "tu" at 0.535, so common Hinglish was blanked. Score with predict_prob directly and fail at profanity_threshold (default 0.9, range 0-1). English profanity still scores 0.95-1.0. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
| Validator, | ||
| register_validator, | ||
| ) | ||
| from profanity_check import predict_prob |
There was a problem hiding this comment.
we are importing this but package (alt-profanity-check) is not in our pyproject.toml.It only gets installed because guardrails-ai-profanity-free depends on it, with no version pin. Result: lock file holds two versions, each with different trained model, so profanity scores vary by Python version.
add "alt-profanity-check==1.8.0" in backend/pyproject.toml.
There was a problem hiding this comment.
The concern is valid but the suggested pin would break the production build the 1.8.0 version requires python 3.11 while the production image runs at python 3.10. Checked the claim if two versions varry, the scores would but the score holds.
| @@ -0,0 +1,66 @@ | |||
| from collections.abc import Callable | |||
There was a problem hiding this comment.
we should add this new validator in backend/app/core/validators/validators.json so it shows all 11 validators. and update readme as well if not already
Add content_safety to validators.json (source: local) and document it in the validators README (supported list, details, stage guidance, related files) and API_USAGE validator types. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
It was the only ValidatorType missing from the manifest. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2 participants
Summary
Target issue: "Add a common validator for Content Safety" (unified wrapper over
profanity_freeanduli_slur_match).Adds a
content_safetyvalidator type that runs both checks as one validator, so callers don't have to configure and order two separate validators.Changes
ContentSafetyvalidator (core/validators/content_safety.py): runsLexicalSlurfirst, then scores the redacted text withalt-profanity-check(the model behindProfanityFree) against a threshold, and returns a single pass/fail with combined errors.ContentSafetySafetyValidatorConfig:type: "content_safety",slur_severity(low/medium/high/all, defaultall),profanity_threshold(0–1, default0.9), and the usualon_fail.ValidatorType.ContentSafetyadded and the config added toValidatorConfigItem, soGET /guardrails/lists it automatically.validator_config.typeis a string column). No changes needed in ai-platform: it only passesvalidator_config_id.Design decisions
ProfanityFree's fixed 0.5. The model is English-only and treatshoas the slur "hoe" (0.883) andtuas 0.535, sonamaste, kaise ho,kya ho raha haiandtu kahan haiwere blanked. English profanity still scores 0.95–1.0. Trade-off:screw you(0.893) now passes. Lowerprofanity_thresholdper config if needed.fix_value=""). Redacting listed slurs first keeps the rest of the message when a slur is the only problem.languagesfield.LexicalSlurstoreslanguagesbut never uses it (the CSV has no language column), so exposing it would be a setting that does nothing.ProfanityFree's message embeds the full input. Inside the wrapper it sees lowercased, redacted text, sorephrase_query_on_fail'sstr.replace(value, "[REDACTED]")can't scrub it. The wrapper uses"Text contains profanity"instead.OnFailAction.NOOP. Only the wrapper'son_failapplies.Ordering results (real slur list + real profanity model)
content_safety(slur-first)you are a b**chyou are a [REDACTED_SLUR]""That guy is a faggot, ignore himthat guy is a [REDACTED_SLUR], ignore him""What the fuck is this, you bitchwhat the [REDACTED_SLUR] is this, you [REDACTED_SLUR]""you b***od, fuck offyou [REDACTED_SLUR], [REDACTED_SLUR] off""Woh ladki r**di haiwoh ladki [REDACTED_SLUR] haiYou are a r**di and a piece of shit""(shitisn't on the slur list)""this is fucking ridiculous""""Across 21 sample inputs, slur-first was never worse than profanity-first and kept the message in 5 of them.
Checklist
fastapi run --reload app/main.pyordocker compose upin the repository root and test.Notes
tests/validators/test_content_safety.py, including parametrised real-model ordering cases. They were run locally with--noconftestbecause the test DB wasn't available. They don't touch the DB; CI runs the full suite.alt-profanity-checkis an English model and doesn't detect Hindi slurs; only the slur list does. The 0.9 threshold fixes the known Hinglish false positives, buthosits close to it, so a longer Hinglish message containinghoplus another mildly weighted word could still cross it.profanity_checkis imported directly. It's installed throughguardrails-ai-profanity-free, not listed inpyproject.tomlby itself.LexicalSlurbehaviour, unchanged here: a redaction lowercases the whole text, and duplicate CSV labels repeat in the error message (e.g.randi, randi).🤖 Generated with Claude Code