Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions notebooks/templates/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ The same functionality is also accesible [in our root directory Makefile](../../
make notebook
```

During generation you'll choose:

1. A **role / document type** (development, validation, or monitoring) that selects which mini-template variants to append.
2. An **inventory record type** (Model by default, Agent, Use Case, Tool, or custom free text) that fills `{record-type}` placeholders in install and next-steps mini-templates.
## Mini-templates

The template generation script/notebook draws from the following mini-templates, should you need to revise them or grab the information from them manually:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
"source": [
"### Key concepts\n",
"\n",
"**record**: A tool tracked in the ValidMind inventory, such as a model. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.\n",
"**record**: A tool tracked in the ValidMind inventory. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.\n",
"\n",
"**model**: [SR 26-2](https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm) (which supersedes SR 11-7) defines a model as a \"complex quantitative method, system, or approach that applies statistical, economic, or financial theories to process input data into quantitative estimates.\" Simple arithmetic, deterministic rule-based processes, or software without statistical, economic, or financial theories underpinning their design or use are generally outside SR 26-2’s definition of a model. Within ValidMind, a model is a type of record tracked in the inventory.\n",
"\n",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
"source": [
"### Key concepts\n",
"\n",
"**record**: A tool tracked in the ValidMind inventory, such as a model. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.\n",
"**record**: A tool tracked in the ValidMind inventory. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.\n",
"\n",
"**model**: [SR 26-2](https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm) (which supersedes SR 11-7) defines a model as a \"complex quantitative method, system, or approach that applies statistical, economic, or financial theories to process input data into quantitative estimates.\" Simple arithmetic, deterministic rule-based processes, or software without statistical, economic, or financial theories underpinning their design or use are generally outside SR 26-2’s definition of a model. Within ValidMind, a model is a type of record tracked in the inventory.\n",
"\n",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
"source": [
"### Key concepts\n",
"\n",
"**record**: A tool tracked in the ValidMind inventory, such as a model. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.\n",
"**record**: A tool tracked in the ValidMind inventory. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.\n",
"\n",
"**model**: [SR 26-2](https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm) (which supersedes SR 11-7) defines a model as a \"complex quantitative method, system, or approach that applies statistical, economic, or financial theories to process input data into quantitative estimates.\" Simple arithmetic, deterministic rule-based processes, or software without statistical, economic, or financial theories underpinning their design or use are generally outside SR 26-2’s definition of a model. Within ValidMind, a model is a type of record tracked in the inventory.\n",
"\n",
Expand Down
35 changes: 33 additions & 2 deletions notebooks/templates/e2e-notebook.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
" - [Designate the filename](#toc1_2__) \n",
" - [Set the title](#toc1_3__) \n",
" - [Set a role / document type](#toc1_4__) \n",
" - [Set an inventory record type](#toc1_5__) \n",
"- [Single-sourcing from mini-templates](#toc2__) \n",
" - [Insert information about ValidMind](#toc2_1__) \n",
" - [Insert setup instructions](#toc2_2__) \n",
Expand Down Expand Up @@ -139,6 +140,34 @@
"tm.select_document()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='toc1_5__'></a>\n",
"\n",
"### Set an inventory record type\n",
"\n",
"Next, you'll be asked to set the inventory record type the notebook deals with. That value is logged and used to fill `{record-type}` placeholders in later mini-templates (for example registration and next-steps instructions):\n",
"\n",
"1. Model (default — press Enter)\n",
"2. Agent\n",
"3. Use Case\n",
"4. Tool\n",
"5. Custom (enter free text)\n",
"\n",
"You can also type a custom record type name directly instead of choosing a numbered option."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tm.select_record_type()"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -199,9 +228,11 @@
"*Install ValidMind* includes information for:\n",
"\n",
"- Installing the ValidMind Library\n",
"- OPTIONAL — Initializing the ValidMind Library (registering a sample model, (skippable) applying a documentation template, retrieving the code snippet, (skippable) previewing the applied template)\n",
"- OPTIONAL — Initializing the ValidMind Library (registering a sample `{record-type}`, (skippable) applying a documentation template, retrieving the code snippet, (skippable) previewing the applied template)\n",
"\n",
"**You'll be prompted to enter a `{template}` here if you choose to include the initialization instructions.** Typing in `SKIP` for those prompts will remove those sections from the generated notebook if they're not relevant.\n",
"\n",
"**You'll be prompted to enter a `{template}` here if you choose to include the initialization instructions.** Typing in `SKIP` for those prompts will remove those sections from the generated notebook if they're not relevant."
"`{record-type}` placeholders from your earlier inventory record type selection are filled in automatically when setup and next-steps mini-templates are appended."
]
},
{
Expand Down
102 changes: 102 additions & 0 deletions notebooks/templates/e2e_template.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,22 @@
"3": "monitoring",
}

RECORD_TYPES = {
"1": "Model",
"2": "Agent",
"3": "Use Case",
"4": "Tool",
}

DEFAULT_RECORD_TYPE = "Model"

INSTALL_CHOICE = {
"1": "install_only",
"2": "install_and_initialize",
}

_selected_document: Optional[str] = None
_selected_record: Optional[str] = None
_selected_install: Optional[str] = None

def ensure_ids(notebook):
Expand Down Expand Up @@ -319,6 +329,96 @@ def select_document():
print(f"Selected: {labels[_selected_document]}")
return _selected_document


def select_record_type(
*,
input_func: Callable[[str], str] = input,
print_func: Callable[[str], None] = print,
record_type_value: Optional[str] = None,
) -> Optional[str]:
"""Requests the inventory record type used to fill `{record-type}` placeholders.

Fixed options are Model, Agent, Use Case, and Tool. Users can also enter a
custom value. Pressing Enter with no input defaults to Model.

Stores the selection in the module-level `_selected_record` variable for
downstream placeholder replacement.

Returns:
The selected record type string.
"""
global _selected_record

if record_type_value is not None:
selected = record_type_value.strip() or DEFAULT_RECORD_TYPE
_selected_record = selected
print_func(f"Selected record type: {_selected_record}")
return _selected_record

choice = input_func(
"Select an inventory record type — "
"[1: Model (default)], "
"[2: Agent], "
"[3: Use Case], "
"[4: Tool], "
"[5: Custom (enter free text)], "
"or press Enter for Model: "
).strip()

if not choice:
selected = DEFAULT_RECORD_TYPE
elif choice in RECORD_TYPES:
selected = RECORD_TYPES[choice]
elif choice == "5":
custom = input_func("Enter a custom inventory record type: ").strip()
if not custom:
print_func(
f"No custom value entered; defaulting to {DEFAULT_RECORD_TYPE}."
)
selected = DEFAULT_RECORD_TYPE
else:
selected = custom
else:
# Allow typing a known label or any free-text record type directly.
selected = choice

_selected_record = selected
print_func(f"Selected record type: {_selected_record}")
return _selected_record


def replace_record_type(
filepath: str,
*,
record_type_value: Optional[str] = None,
print_func: Callable[[str], None] = print,
) -> None:
"""Replace `{record-type}` placeholders in a notebook file.

Uses `record_type_value` when provided; otherwise falls back to the value
stored by `select_record_type()`. Defaults to Model when neither is set.
"""
value = (record_type_value or _selected_record or DEFAULT_RECORD_TYPE).strip()
if not value:
value = DEFAULT_RECORD_TYPE

try:
with open(filepath, "r", encoding="utf-8") as f:
content = f.read()

if "{record-type}" not in content:
return

content = content.replace("{record-type}", value)
with open(filepath, "w", encoding="utf-8") as f:
if not content.endswith("\n"):
content += "\n"
f.write(content)
print_func(f"Replaced {{record-type}} with '{value}' in '{filepath}'")
except Exception as e:
print_func(f"Error replacing record type placeholders in file: {e}")


def add_about(filepath):
"""Appends an about-validmind notebook based on the document type selected via select_document()."""
about_files = {
Expand Down Expand Up @@ -609,6 +709,7 @@ def _append_notebook(filepath, relative_path):
print(f"Error appending notebooks: {e}")
return False

replace_record_type(filepath)
return True


Expand Down Expand Up @@ -744,6 +845,7 @@ def add_copyright(filepath: str) -> None:
if filepath:
set_title(filepath)
select_document()
select_record_type()
add_about(filepath)
select_install()
add_install(filepath)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,23 +45,22 @@
"id": "register-model",
"metadata": {},
"source": [
"#### Register sample model\n",
"#### Register sample {record-type}\n",
"\n",
"Let's first register a sample record (model) for use with this notebook.\n",
"Let's first register a sample record ({record-type}) for use with this notebook.\n",
"\n",
"1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/access/log-in-to-validmind.html).\n",
"\n",
"2. In the left sidebar, select **Inventory**.\n",
"\n",
"3. Select `Model` by clicking on **{Record} Inventory**, where `{Record}` is the currently active type of record. (Learn more: [Register records in the inventory](https://docs.validmind.ai/guide/inventory/register-records-in-inventory.html))\n",

"4. Click **+ Register Model**.\n",
"\n",
"5. Enter the model details and click **Next >** to continue to assignment of inventory record stakeholders.\n",
"\n",
"6. Select your own name under the **Record Owner** drop-down.\n",
"\n",
"7. Click **Register Model** to add the model to your inventory."
"3. Select `{record-type}` by clicking on **{Record} Inventory**, where `{Record}` is the currently active type of record. (Learn more: [Register records in the inventory](https://docs.validmind.ai/guide/inventory/register-records-in-inventory.html))\n",
"4. Click **+ Register {record-type}**.\n",
"\n",
"5. Enter the {record-type} details and click **Next >** to continue to assignment of inventory record stakeholders.\n",
"\n",
"6. Select your own name under the **Record Owner** drop-down.\n",
"\n",
"7. Click **Register {record-type}** to add the {record-type} to your inventory."
]
},
{
Expand All @@ -71,11 +70,11 @@
"source": [
"#### Apply documentation template\n",
"\n",
"Once you've registered your model, let's select a documentation template. A template predefines sections for your documentation and provides a general outline to follow, making the documentation process much easier.\n",
"Once you've registered your {record-type}, let's select a documentation template. A template predefines sections for your documentation and provides a general outline to follow, making the documentation process much easier.\n",
"\n",
"1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n",
"1. In the left sidebar that appears for your {record-type}, click **Documents** and select **Development**.\n",
"\n",
" If you cannot locate your Development document, make sure Development type documents are enabled for model records and create a new document. (**Learn more:** [Manage documents](https://docs.validmind.ai/guide/templates/manage-documents.html#add-record-documents))\n",
" If you cannot locate your Development document, make sure Development type documents are enabled for {record-type} records and create a new document. (**Learn more:** [Manage documents](https://docs.validmind.ai/guide/templates/manage-documents.html#add-record-documents))\n",
"\n",
"2. Under **Template**, select `{template}`.\n",
"\n",
Expand All @@ -91,12 +90,11 @@
"\n",
"Initialize the ValidMind Library with the *code snippet* unique to each record per document, ensuring your test results are uploaded to the correct record and automatically populated in the right document in the ValidMind Platform when you run the Library.\n",
"\n",
"1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **Document** drop-down menu.\n",
"1. On the left sidebar that appears for your {record-type}, select **Getting Started** and select `Development` from the **Document** drop-down menu.\n",
"\n",
"2. Click **Copy snippet to clipboard**.\n",
"\n",
"3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/quickstart/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:",
"\n",
"3. Next, [load your {record-type} identifier credentials from an `.env` file](https://docs.validmind.ai/developer/quickstart/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:\n",
"If your organization authenticates with OAuth 2.0 / OIDC instead of API keys, see [Authenticate with OIDC device flow](https://docs.validmind.ai/developer/quickstart/authenticate-with-oidc-device-flow.html) or try the [OIDC device-flow quickstart notebook](https://docs.validmind.ai/notebooks/quickstart/quickstart_model_documentation_oidc_device_flow.html).\n"
]
},
Expand All @@ -107,7 +105,7 @@
"metadata": {},
"outputs": [],
"source": [
"# Load your model identifier credentials from an `.env` file\n",
"# Load your {record-type} identifier credentials from an `.env` file\n",
"\n",
"%load_ext dotenv\n",
"%dotenv .env\n",
Expand All @@ -132,9 +130,9 @@
"source": [
"### Preview the documentation template\n",
"\n",
"Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n",
"Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your {record-type}.\n",
"\n",
"You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:"
"You will upload documentation and test results unique to your {record-type} based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:"
]
},
{
Expand Down
Loading
Loading