Controls clustering

Groundwork for grouping financial controls by what they do, not by who wrote them.

Every firm writes its control library in its own house style: its own column names, its own taxonomy, its own boilerplate. If you cluster controls from several firms as they are, you mostly get one cluster per firm. This repo prepares the data for clustering by function. It consolidates control libraries from different firms into one schema, then strips out each firm's fingerprints before any clustering happens.

Note

Status: prototype, and the clustering step isn't built. What's here is the consolidation and preprocessing stages, built over two days in July 2025 with Claude Code as a pair programmer. Consolidation works. Preprocessing works in part (see Known issues). It was extracted from a private repo, and the example organisations are fictional.

What it does

firm workbooks ─► consolidate_data.py ─► one schema, stable IDs, domain tags
                                              │
                                              ▼
                                         pipeline.py ─► de-fingerprinted text, ready to cluster

1. Consolidate (src/consolidate_data.py)

• Reads each firm's Excel export using a per-firm column mapping in config/org_configs.json, so adding a firm is a config change, not a code change.
• Maps everything onto one schema: control, risk and process fields, plus source file and sheet.
• Gives every control a stable SHA-256 ID, so references survive a re-run.
• Drops rows with no description, one- or two-word descriptions, obvious test data, and duplicates within a firm.
• Tags each control with a domain (AML, cyber, operational, credit, market, compliance, financial reporting, technology) by keyword score.
• Writes CSV, Parquet and a JSON summary.

2. Preprocess (src/pipeline.py)

• Standardises vocabulary against published terminology (ISO 20022, FIBO and FATF) in config/standardization_mappings.json, and expands abbreviations (KYC, EDD, MLRO and so on).
• Replaces details that identify a firm with placeholders, using regex patterns and spaCy entity recognition: legal suffixes ("Ltd", "plc", "Bank"), people, places and phone numbers.
• Buckets money amounts into bands, so "$50,000" and "$75,000" both become [AMOUNT_MEDIUM].
• Strips control stopwords ("shall", "ensure", "appropriate") and boilerplate phrases.
• Replaces entities on token boundaries, so replacing "US" can't corrupt "obvious".

What it does to a fictional control today:

in:  Acme Capital Ltd shall perform EDD on high risk clients with transfers of $75,000; contact [email protected] or System FinTrack.
out: acme capital [ORGANIZATION] enhanced due diligence on high risk account holder with transaction of [AMOUNT_MEDIUM]; contact ops acme.example or system fintrack.

Why it's built this way

• Rules before models. Term mapping, abbreviations and most fingerprint removal are config and regex, so they're cheap, deterministic and easy to inspect. spaCy handles tokenising and entity recognition.
• Config, not code. Vocabularies, patterns and firm mappings live in JSON, so a domain expert can change them without touching Python.
• Stable IDs. Clusters get reviewed by people, and a review is useless if the IDs change on the next run.

Known issues

The example above shows most of them:

• Firm names survive unless they end in a legal suffix. Text is lowercased before entity recognition runs, so spaCy no longer sees "Acme Capital" as an organisation. Entity recognition needs to move before lowercasing.
• Emails and system names aren't replaced. The cleaning step strips @ before the email pattern runs, and the system-name pattern expects capitals that are already gone.
• Any bare number is treated as an amount, so "within 5 days" becomes [AMOUNT_SMALL]days and the duration bands never apply.
• Boilerplate phrases are only partly removed, because stopword removal runs first and breaks the phrase match.

Fixed while extracting this repo: pattern extraction used findall, which returns capture groups. For the amount pattern that was an empty string, and the pipeline then put [AMOUNT] between every character of every control. tests/test_extract_matches.py covers it.

Not built yet

• The clustering itself. The plan was embeddings over the de-fingerprinted text, with a constraint that a cluster must contain controls from more than one firm.
• Any evaluation of cluster quality.

Run it

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m spacy download en_core_web_sm

# Put firm workbooks matching config/org_configs.json in data/raw/ (git-ignored)
python src/consolidate_data.py data/raw data/processed

python -m pytest tests

The tests test each step on its own, which is why they missed the issues above. 33 pass and 2 are marked as expected failures, each with its reason: abbreviation expansion changes word count, which one test didn't allow for, and removing a phrase between two commas leaves an empty ", ,".

Repo layout

config/   firm column mappings, domain keywords, terminology and preprocessing patterns
docs/     consolidation guide
src/      consolidate_data.py and pipeline.py
tests/    unit and integration tests