Production-ready AI system for empathy-aware conversational AI with proper organization and deployment structure.
ai/
├── training/ # Model training, fine-tuning, safety, judges
│ ├── configs/ # Hyperparameters, infra, model/stage configs
│ ├── scripts/ # Training scripts (gpu, services, sprint8)
│ ├── tests/ # Training test suite
│ ├── rlhf/ # RLHF / DPO / GRPO trainers
│ ├── sdg/ # Synthetic data generation
│ ├── defense_mechanisms/ # Defense mechanism training
│ ├── coaching_safety/ # Coaching safety modules
│ ├── sliced/ # Stage-sliced datasets (stage1_foundation, stage2_*)
│ ├── utils/ # Shared training utilities
│ ├── checkpoints/ # Model checkpoints
│ └── *.py # Flat trainers/judges/scorers (see note below)
├── data/ # Datasets and data management
│ ├── raw/ # Raw datasets
│ ├── processed/ # Processed datasets
│ ├── curated/ # Curated datasets (DVC-tracked)
│ └── synthetic/ # Synthetic data generation
├── benchmarks/ # Benchmarks
│ ├── *_performance_baseline.json # CPU/memory/inference benchmark baselines
│ └── tests/ # Benchmark tests
├── models/ # Model artifacts and exports
│ ├── checkpoints/ # Saved model checkpoints
│ ├── artifacts/ # Model artifacts
│ ├── exports/ # Exported models for deployment
│ ├── base/ # Base model definitions
│ ├── moe/ # Mixture-of-experts
│ └── pixel/ # Pixel model
├── inference/ # Deployment and inference
│ ├── api/ # API endpoints (cms, mcp_server, memory, techdeck)
│ ├── services/ # Inference services
│ └── deployment/ # Deployment configs (helm, k8s, s3, postgres)
├── pipelines/ # Data and model pipelines
│ ├── data_processing/ # Data processing (academic, journal, youtube, extractors)
│ ├── model_training/ # Training pipelines
│ ├── evaluation/ # Model evaluation
│ ├── orchestration/ # Stage organizers
│ ├── voice/ # Voice / audio pipelines
│ └── edge_case/ # Edge case generation
├── sourcing/ # Data sourcing (academic, journal, youtube)
├── research/ # Research and experimentation
│ ├── notebooks/ # Jupyter notebooks
│ ├── experiments/ # Research experiments
│ ├── analysis/ # Analysis and reports
│ ├── gates/ # Inference gates (consent, crisis, pii)
│ └── reflection/ # Reflection memory modules
├── tools/ # Utilities and tools
│ ├── utilities/ # Utility modules (api, core, data, pipelines, pkg_mera)
│ ├── scripts/ # Shell scripts and command-line tools
│ ├── generators/ # Data generators
│ └── DataDesigner/ # External NeMo Data Designer snapshot
├── qa/ # Quality assurance
│ ├── reports/ # QA reports
│ ├── validation/ # Validation (crisis_detection, diagnosis_arena, therapy_bench)
│ └── testing/ # Test suites
│ └── legacy_tests/ # Archived broken legacy tests
├── annotation/ # Annotation agents and API
├── compliance/ # Compliance (db, security, validators)
├── prompts/ # Prompt templates (agents, clinical, safety, system)
├── configs/ # Config management (envs, models, monitoring, legacy)
├── docs/ # Documentation (api, architecture, guides, ops)
├── scripts/ # Maintenance scripts
├── migrations/ # DB migrations (SQL)
├── experiments/ # Experiment scratch space
├── assets/ # Static assets
└── ops/ # Ops tooling (Dockerfile, Makefile, CI)
Note on flat
training/*.py: ~70 modules (trainers, judges, scorers, SDG pipelines) live at thetraining/root and are heavily cross-imported (training.clinical_validity_scorer,training.pixelated_production_pilot,training.dpo_trainer, etc.). They are importable library code, not runnable scripts, and remain flat by design until a dedicated package split is done.
# Sync dependencies (Python 3.13 via uv)
uv syncuv run pytestuv run python -m training.<entry_point>uv run python -m inference.api.<module>Run all training tests (excluding legacy book PDF converter tests):
uv run pytest training/tests/ --ignore=training/tests/test_book_pdf_converter.py -qCoverage is enforced by CI (.github/workflows/training-safety-coverage.yml) at two tiers:
-
Safety-critical (95 % aggregate, branch coverage):
training.shared_config,training.multilingual_safety_checker,training.clinical_safety_checker, andtraining.reward_score. These four modules gate the clinical decision logic — the 95 % threshold accounts for two unreachableModuleNotFoundErrorfallback lines inmultilingual_safety_checker.pythat cannot be exercised under the editable install used in CI. -
Pilot module (40 %):
training.pixelated_production_pilot. The pilot module is a 1,043-line production SFT pipeline whose body resolves at runtime only with the fulltransformers+trl+peftstack on GPU. Unit tests deliberately cover only the dataclass/CLI/path-safety surface (config, args,HubConfig,RunConfig,_maybe_push_to_hub,safe_path,CheckpointVerificationCallback). ML-pipeline correctness is validated separately via the training runnable and smoke tests.
Run a tier locally (matches what CI runs):
# Safety-critical gate
uv run pytest training/tests/ --ignore=training/tests/test_book_pdf_converter.py \
--cov=training.shared_config \
--cov=training.multilingual_safety_checker \
--cov=training.clinical_safety_checker \
--cov=training.reward_score \
--cov-branch --cov-fail-under=95 -q \
--cov-report=xml:coverage/safety-critical-coverage.xml \
--cov-report=term
# Pilot gate
uv run pytest training/tests/ --ignore=training/tests/test_book_pdf_converter.py \
--cov=training.pixelated_production_pilot \
--cov-branch --cov-fail-under=40 -q \
--cov-report=xml:coverage/pilot-coverage.xml \
--cov-report=termEarlier revisions of this README and the workflow targeted 100 % coverage across
the safety-critical modules. That gate failed in practice because two
ModuleNotFoundError fallback lines in multilingual_safety_checker.py (the
absolute from ai.training.clinical_safety_checker import ... fallback) are
unreachable under an editable install — Python's import machinery always
resolves the relative from .clinical_safety_checker import first when the
training package is on sys.path. The fall-through is a defensive idiom
for distribution layouts where training is not a discoverable package; it
is not a regression target. The 95 % gate documents this honestly.
If the 100 % threshold is required for regulatory reasons in the future, the
test test_fallback_import_path_used will need a custom sys.meta_path finder
to force the relative import to fail deterministically. That's tracked as a
follow-up; do not silently lower the threshold from 95 % to 100 % again.
Lightweight smoke tests for direct preference optimization and group-relative policy optimization:
uv run pytest training/tests/test_dpo_trainer.py training/tests/test_grpo_trainer.py -qThe training pipeline runs on a CPU-only PyTorch stack. The [tool.uv.sources] section in
pyproject.toml pins torch, torchvision, and torchaudio to the pytorch-cpu index to
avoid pulling CUDA binaries that would fail on CPU-only runners.
Set the following environment variable in test/CI environments to disable ML model loading for
safety checkers (enabled by default in conftest.py):
export AI_DISABLE_SAFETY_ML_MODELS=1- Environment configs:
configs/envs/{development,staging,production,testing}/ - Model configs:
configs/models/ - Monitoring configs:
configs/monitoring/
- Training checkpoints:
training/checkpoints/ - Model artifacts:
models/artifacts/ - Exported models:
models/exports/
- Raw datasets:
data/raw/ - Processed datasets:
data/processed/ - Curated datasets:
data/curated/(DVC-tracked) - Synthetic data:
data/synthetic/
- API endpoints:
inference/api/ - Deployment configs:
inference/deployment/ - Service implementations:
inference/services/
- Data Preparation: Use pipelines under
pipelines/data_processing/ - Data Sourcing: Use
sourcing/ - Model Training: Use
training/(flat modules +training/scripts/) - Model Evaluation: Use
pipelines/evaluation/ - Deployment: Use configurations in
inference/deployment/ - Quality Assurance: Use
qa/validation/andqa/testing/
All previous task files and reports are kept in their functional homes
(qa/reports/ for PR/audit data, tools/DataDesigner/ for the external
Data Designer snapshot, qa/testing/legacy_tests/ for broken legacy tests).
# Sync dependencies via uv
uv sync
# Activate virtual environment
source .venv/bin/activate- Documentation:
docs/ - API Reference:
docs/api/ - Architecture Guide:
docs/architecture/