Repository navigation
Add a Colab notebook for training models from log files #112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,250 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Train an IMPSY model\n", | ||
| "\n", | ||
| "[](https://colab.research.google.com/github/cpmpercussion/impsy/blob/main/notebooks/impsy_training.ipynb)\n", | ||
| "\n", | ||
| "This notebook trains an [IMPSY](https://charlesmartin.au/impsy) model from your `.log` files and gives you a `.tflite` model to perform with. Use it if you don't have a computer that can run IMPSY's training yourself.\n", | ||
| "\n", | ||
| "Log files from IMPSY, IMPSYpi, the IMPSY AUv3 app and IMPSY Web all work. Their names end in `-{dimension}d-mdrnn.log`, e.g. `2026-09-01T12-00-00-9d-mdrnn.log`.\n", | ||
| "\n", | ||
| "You don't need a GPU: training a small model on about 10 minutes of data takes a few minutes on Colab's standard CPU runtime.\n", | ||
| "\n", | ||
| "Run each cell in order with the ▶ button (or Shift-Enter)." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## 1. Install IMPSY\n", | ||
| "\n", | ||
| "This takes a minute or two. If Colab asks you to restart the session afterwards, restart it and carry on from step 2." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "%pip install -q impsy" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## 2. Upload your log files\n", | ||
| "\n", | ||
| "Choose one or more `.log` files, or a `.zip` of them. Outside Colab, put your log files in a `logs` folder next to this notebook instead." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import zipfile\n", | ||
| "from pathlib import Path\n", | ||
| "\n", | ||
| "LOGS_DIR = Path(\"logs\")\n", | ||
| "LOGS_DIR.mkdir(exist_ok=True)\n", | ||
| "\n", | ||
| "try:\n", | ||
| " from google.colab import files\n", | ||
| "\n", | ||
| " uploaded = files.upload()\n", | ||
| " for name, content in uploaded.items():\n", | ||
| " if name.endswith(\".zip\"):\n", | ||
| " with zipfile.ZipFile(Path(name)) as z:\n", | ||
| " for member in z.namelist():\n", | ||
| " if member.endswith(\".log\") and not member.startswith(\"__MACOSX\"):\n", | ||
| " (LOGS_DIR / Path(member).name).write_bytes(z.read(member))\n", | ||
| " Path(name).unlink()\n", | ||
| " elif name.endswith(\".log\"):\n", | ||
| " Path(name).rename(LOGS_DIR / Path(name).name)\n", | ||
| " else:\n", | ||
| " print(f\"Skipping {name}: not a .log or .zip file\")\n", | ||
| "except ImportError:\n", | ||
| " print(\"Not running in Colab: using the log files already in\", LOGS_DIR.resolve())" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## 3. Check your logs\n", | ||
| "\n", | ||
| "This counts your log files by dimension (the number of values in each log plus one for time). If you have logs of more than one dimension, it uses the one with the most files; set `DIMENSION` in the next step to choose a different one." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import re\n", | ||
| "from collections import Counter\n", | ||
| "\n", | ||
| "dimensions = Counter()\n", | ||
| "for log in sorted(LOGS_DIR.glob(\"*.log\")):\n", | ||
| " match = re.search(r\"-(\\d+)d-mdrnn\\.log$\", log.name)\n", | ||
| " if match:\n", | ||
| " dimensions[int(match.group(1))] += 1\n", | ||
| " else:\n", | ||
| " print(f\"Ignoring {log.name}: its name should end in -{{dimension}}d-mdrnn.log\")\n", | ||
| "\n", | ||
| "for dim, count in sorted(dimensions.items()):\n", | ||
| " print(f\"{dim}d: {count} log file(s)\")\n", | ||
| "assert dimensions, \"No usable log files found, go back to step 2 and upload some.\"" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## 4. Choose training settings\n", | ||
| "\n", | ||
| "- **Model size:** `xs` or `s` is a good start. Bigger models (`m`, `l`, `xl`) can learn more but need much more data and take longer to train.\n", | ||
| "- **Max epochs:** training usually stops earlier, when the model stops improving for *patience* epochs.\n", | ||
| "- **Dimension:** leave at 0 to use the most common dimension from step 3." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "cellView": "form" | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "MODEL_SIZE = \"s\" # @param [\"xxs\", \"xs\", \"s\", \"m\", \"l\", \"xl\"]\n", | ||
| "MAX_EPOCHS = 100 # @param {type:\"integer\"}\n", | ||
| "PATIENCE = 10 # @param {type:\"integer\"}\n", | ||
| "DIMENSION = 0 # @param {type:\"integer\"}\n", | ||
| "\n", | ||
| "if DIMENSION == 0:\n", | ||
| " DIMENSION = dimensions.most_common(1)[0][0]\n", | ||
| "assert DIMENSION in dimensions, (\n", | ||
| " f\"There are no {DIMENSION}d logs. Choose one of: \"\n", | ||
| " + \", \".join(f\"{d}\" for d in sorted(dimensions))\n", | ||
| ")\n", | ||
| "print(f\"Training a {MODEL_SIZE} model on {DIMENSION}d data.\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## 5. Make a dataset\n", | ||
| "\n", | ||
| "This collects all of your logs of the chosen dimension into one dataset file. It's good to have more than 10,000 interactions, but you can train on less. The model learns from runs of 50 events, so only logs with more than 51 events are used for training." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import numpy as np\n", | ||
| "from impsy.dataset import generate_dataset\n", | ||
| "from impsy.train import SEQ_LEN\n", | ||
| "\n", | ||
| "Path(\"datasets\").mkdir(exist_ok=True)\n", | ||
| "dataset_file = generate_dataset(DIMENSION, source=\"logs\", destination=\"datasets\")\n", | ||
| "assert dataset_file is not None, (\n", | ||
| " f\"No data found in the {DIMENSION}d logs. Logs need rows recorded from \"\n", | ||
| " \"your interface (with 'interface' in the second column).\"\n", | ||
| ")\n", | ||
| "\n", | ||
| "with np.load(dataset_file, allow_pickle=True) as loaded:\n", | ||
| " long_enough = [p for p in loaded[\"perfs\"] if len(p) > SEQ_LEN + 1]\n", | ||
| "print(f\"{len(long_enough)} log(s) long enough to train on.\")\n", | ||
| "assert long_enough, (\n", | ||
| " f\"None of your logs have more than {SEQ_LEN + 1} events, \"\n", | ||
| " \"so there's nothing to train on. Record some longer logs.\"\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## 6. Train\n", | ||
| "\n", | ||
| "You'll see the loss for each epoch; lower is better. The `val_loss` is measured on data the model hasn't trained on, and training stops once it stops improving." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from impsy.train import train_mdrnn\n", | ||
| "\n", | ||
| "Path(\"models\").mkdir(exist_ok=True)\n", | ||
| "output = train_mdrnn(\n", | ||
| " DIMENSION,\n", | ||
| " dataset_file,\n", | ||
| " MODEL_SIZE,\n", | ||
| " early_stopping=True,\n", | ||
| " patience=PATIENCE,\n", | ||
| " num_epochs=MAX_EPOCHS,\n", | ||
| " batch_size=64,\n", | ||
| " save_location=\"models\",\n", | ||
| ")\n", | ||
| "tflite_file = Path(output[\"tflite_file\"])\n", | ||
| "print(\"Trained model:\", tflite_file)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## 7. Download your model\n", | ||
| "\n", | ||
| "Download the `.tflite` file and copy it to your IMPSY setup:\n", | ||
| "\n", | ||
| "- **IMPSY / IMPSYpi:** put it in the `models` folder (or upload it on the web UI's Models page) and set `file` under `[model]` in `config.toml`, along with the matching `dimension` and `size`.\n", | ||
| "- **IMPSY AUv3 and IMPSY Web:** load the `.tflite` file in the app." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "try:\n", | ||
| " from google.colab import files\n", | ||
| "\n", | ||
| " files.download(str(tflite_file))\n", | ||
| "except ImportError:\n", | ||
| " print(\"Your model is at\", tflite_file.resolve())" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "colab": { | ||
| "provenance": [] | ||
| }, | ||
| "kernelspec": { | ||
| "display_name": "Python 3", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "name": "python" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 0 | ||
| } | ||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Constraint and lock are fixed in d99cabc (
psutil>=6.0.0,<8, locked at 7.2.2). The notebook installs from PyPI, so this reaches Colab with the next impsy release, which should go out soon after this merges.