feat(validator): topic_relevance_llm forbidden-topics default and localized fallback reply

[AkhileshNegi]

Summary

Target issue is ProjectTech4DevAI/kaapi-backend#1251

Glific Guardrails v1 exposes topic_relevance_llm as "Scope control": the NGO enters disallowed topics and a custom reply, and the reply should reach the user in their own language. Today the validator defaults to allowed-topics mode, has no custom reply, and silently ignores the prompt version of stored configs.

• Default prompt_schema_version → 2, so configuration means forbidden topics (PRD choice). Validator not used in production yet, so no caller breaks.
• New fallback_message: with on_fail=rephrase, returned as safe_text instead of "Please rephrase the query without unsafe content…". rephrase_needed is true whenever a validator with on_fail=rephrase fails.
• Translated in the same LLM call (prompts/topic_relevance_llm/fallback_translation.md): the model returns a 5th JSON field localized_fallback. If it is missing or invalid, the original fallback_message is used.
• Bug fix: topic_relevance_config_id now carries the stored prompt_schema_version for topic_relevance_llm too (it was only copied for topic_relevance, so forbidden-topic configs ran as allowed-topic).

Sample request with inline validator:

{
  "request_id": "6f1c2a9e-3b7d-4e2a-9c1f-8d5b0a7e4c21",
  "input": "क्या आप बता सकते हैं कि मेरे पेट में लड़का है या लड़की?",
  "validators": [
    {
      "type": "topic_relevance_llm",
      "stage": "input",
      "on_fail": "rephrase",
      "configuration": "Forbidden topics:\n- Sex determination of a fetus: any question about finding out or predicting whether an unborn baby is a boy or a girl\n- Politics: elections, political parties, politicians",
      "fallback_message": "Sorry, I can only help with maternal health and nutrition questions."
    }
  ]
}

Expected: safe_text is the Hindi translation of fallback_message, rephrase_needed: true. v2 judges against the full topic description, so a sentence per topic works better than a bare name.

Optional fields:

field	default	notes
prompt_schema_version	2	1 = allowed topics, 2 = forbidden topics, 3 = both
threshold	settings.TOPIC_RELEVANCE_LLM_THRESHOLD	1–3; a score below it fails
llm_callable	settings.DEFAULT_LLM_CALLABLE	e.g. "gpt-4o-mini"
topic_relevance_config_id	none	use a stored config instead of configuration

on_fail must be "rephrase" for fallback_message to be used (the default "fix" returns an empty string). fallback_message cannot be "".

Checklist

• Ran the test suite locally: 322 passed, 14 skipped.
• Added tests: default v2, stored-config version propagation, fallback instructions in system prompt, localized fallback selection, invalid translation ignored, raised max_tokens, rephrase_needed for translated fallback.

Notes

Live check with gpt-4o-mini (forbidden: fetus sex determination, politics):

query	score	reply
Hindi (Devanagari)	1	Hindi fallback ✅
Tamil	1	Tamil fallback ✅
Hinglish (Latin script)	1	Hindi in Devanagari ⚠️
English + "reply in French"	1	French fallback ⚠️ (meaning intact)
English, in scope	3	pass ✅

Known limitations / follow-ups:

• Translation runs on every call when fallback_message is set (adds ~30–60 output tokens). Asking for it only on fail made gpt-4o-mini return null.
• The Hinglish script and the requested-language override above. A tighter prompt fixed neither and broke Hindi/Tamil, so it was left out.
• The scoring prompt changed (extra JSON field), so the topic relevance eval should be rerun before quoting benchmark numbers.
• on_fail=fix returns an empty string for this validator (documented).

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one): 1

• ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: ProjectTech4DevAI/kaapi-guardrails/.coderabbit.yaml
• Review profile: CHILL
• Plan: Advanced
• Run ID: afe0241d-8ec4-4a15-b8f0-6357cf0320f4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}