From 5481bf2f548bcc1736bcc7b811c8fa0b034fc94c Mon Sep 17 00:00:00 2001 From: Charles Martin Date: Sat, 26 Sep 2026 18:07:11 +1000 Subject: [PATCH 1/3] Add a Colab notebook for training models from log files notebooks/impsy_training.ipynb installs impsy from PyPI, takes uploaded .log files (or a .zip), makes a dataset, trains a model and downloads the .tflite. It uses the train_mdrnn/generate_dataset APIs from the released 1.1.0 so it works before the next release, and runs outside Colab too using a local logs folder. Linked from the README and the get-started page. Part of #97. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016oUxTsghWZWfnCnnkKcuNv --- README.md | 2 + docs/get-started.md | 2 + notebooks/impsy_training.ipynb | 233 +++++++++++++++++++++++++++++++++ 3 files changed, 237 insertions(+) create mode 100644 notebooks/impsy_training.ipynb diff --git a/README.md b/README.md index ef65200..ed39acf 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,8 @@ Every time you use IMPSY's `run` command, a new log file is created so that you There's two steps for training: Generate a dataset file, and train the predictive model. +> **No computer to train on?** The [IMPSY training notebook](https://colab.research.google.com/github/cpmpercussion/impsy/blob/main/notebooks/impsy_training.ipynb) runs the same steps on Google Colab: upload your `.log` files, and download a trained `.tflite` model. You'll need a Google account. + Use the `dataset` command: poetry run ./start_impsy.py dataset --dimension (N+1) diff --git a/docs/get-started.md b/docs/get-started.md index 8ee2df0..3322b75 100644 --- a/docs/get-started.md +++ b/docs/get-started.md @@ -45,6 +45,8 @@ IMPSY's prediction model is a small mixture density recurrent network. Training See the main repository for the current training command, hyperparameter defaults, and tips for tuning the mixture parameters. +If you don't have a computer that can run IMPSY, the [IMPSY training notebook](https://colab.research.google.com/github/cpmpercussion/impsy/blob/main/notebooks/impsy_training.ipynb) trains a model on Google Colab from your uploaded `.log` files. + ## 5. Run the instrument Once you have a trained model, IMPSY can run in *call-and-response* mode (the model answers when you pause), *polyphony* mode (you and the model play simultaneously), or *battle* mode (the model plays autonomously alongside you) — plus a *user-only* mode for logging data without predictions. See the [configuration reference]({{ '/config/' | relative_url }}) for details. On a desktop machine you can iterate quickly; on a Pi, you get a self-contained intelligent instrument that boots straight into performance mode. diff --git a/notebooks/impsy_training.ipynb b/notebooks/impsy_training.ipynb new file mode 100644 index 0000000..57a586c --- /dev/null +++ b/notebooks/impsy_training.ipynb @@ -0,0 +1,233 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Train an IMPSY model\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cpmpercussion/impsy/blob/main/notebooks/impsy_training.ipynb)\n", + "\n", + "This notebook trains an [IMPSY](https://charlesmartin.au/impsy) model from your `.log` files and gives you a `.tflite` model to perform with. Use it if you don't have a computer that can run IMPSY's training yourself.\n", + "\n", + "Log files from IMPSY, IMPSYpi, the IMPSY AUv3 app and IMPSY Web all work. Their names end in `-{dimension}d-mdrnn.log`, e.g. `2026-09-01T12-00-00-9d-mdrnn.log`.\n", + "\n", + "You don't need a GPU: training a small model on about 10 minutes of data takes a few minutes on Colab's standard CPU runtime.\n", + "\n", + "Run each cell in order with the ▶ button (or Shift-Enter)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Install IMPSY\n", + "\n", + "This takes a minute or two. If Colab asks you to restart the session afterwards, restart it and carry on from step 2." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q impsy" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Upload your log files\n", + "\n", + "Choose one or more `.log` files, or a `.zip` of them. Outside Colab, put your log files in a `logs` folder next to this notebook instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import zipfile\n", + "from pathlib import Path\n", + "\n", + "LOGS_DIR = Path(\"logs\")\n", + "LOGS_DIR.mkdir(exist_ok=True)\n", + "\n", + "try:\n", + " from google.colab import files\n", + "\n", + " uploaded = files.upload()\n", + " for name, content in uploaded.items():\n", + " if name.endswith(\".zip\"):\n", + " with zipfile.ZipFile(Path(name)) as z:\n", + " for member in z.namelist():\n", + " if member.endswith(\".log\") and not member.startswith(\"__MACOSX\"):\n", + " (LOGS_DIR / Path(member).name).write_bytes(z.read(member))\n", + " Path(name).unlink()\n", + " elif name.endswith(\".log\"):\n", + " Path(name).rename(LOGS_DIR / Path(name).name)\n", + " else:\n", + " print(f\"Skipping {name}: not a .log or .zip file\")\n", + "except ImportError:\n", + " print(\"Not running in Colab: using the log files already in\", LOGS_DIR.resolve())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Check your logs\n", + "\n", + "This counts your log files by dimension (the number of values in each log plus one for time). If you have logs of more than one dimension, it uses the one with the most files; set `DIMENSION` in the next step to choose a different one." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import re\n", + "from collections import Counter\n", + "\n", + "dimensions = Counter()\n", + "for log in sorted(LOGS_DIR.glob(\"*.log\")):\n", + " match = re.search(r\"-(\\d+)d-mdrnn\\.log$\", log.name)\n", + " if match:\n", + " dimensions[int(match.group(1))] += 1\n", + " else:\n", + " print(f\"Ignoring {log.name}: its name should end in -{{dimension}}d-mdrnn.log\")\n", + "\n", + "for dim, count in sorted(dimensions.items()):\n", + " print(f\"{dim}d: {count} log file(s)\")\n", + "assert dimensions, \"No usable log files found, go back to step 2 and upload some.\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. Choose training settings\n", + "\n", + "- **Model size:** `xs` or `s` is a good start. Bigger models (`m`, `l`, `xl`) can learn more but need much more data and take longer to train.\n", + "- **Max epochs:** training usually stops earlier, when the model stops improving for *patience* epochs.\n", + "- **Dimension:** leave at 0 to use the most common dimension from step 3." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form" + }, + "outputs": [], + "source": [ + "MODEL_SIZE = \"s\" # @param [\"xxs\", \"xs\", \"s\", \"m\", \"l\", \"xl\"]\n", + "MAX_EPOCHS = 100 # @param {type:\"integer\"}\n", + "PATIENCE = 10 # @param {type:\"integer\"}\n", + "DIMENSION = 0 # @param {type:\"integer\"}\n", + "\n", + "if DIMENSION == 0:\n", + " DIMENSION = dimensions.most_common(1)[0][0]\n", + "print(f\"Training a {MODEL_SIZE} model on {DIMENSION}d data.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Make a dataset\n", + "\n", + "This collects all of your logs of the chosen dimension into one dataset file. It's good to have more than 10,000 interactions, but you can train on less." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from impsy.dataset import generate_dataset\n", + "\n", + "Path(\"datasets\").mkdir(exist_ok=True)\n", + "dataset_file = generate_dataset(DIMENSION, source=\"logs\", destination=\"datasets\")\n", + "assert dataset_file is not None, \"The dataset is empty; check your logs contain 'interface' rows.\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 6. Train\n", + "\n", + "You'll see the loss for each epoch; lower is better. The `val_loss` is measured on data the model hasn't trained on, and training stops once it stops improving." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from impsy.train import train_mdrnn\n", + "\n", + "Path(\"models\").mkdir(exist_ok=True)\n", + "output = train_mdrnn(\n", + " DIMENSION,\n", + " dataset_file,\n", + " MODEL_SIZE,\n", + " early_stopping=True,\n", + " patience=PATIENCE,\n", + " num_epochs=MAX_EPOCHS,\n", + " batch_size=64,\n", + " save_location=\"models\",\n", + ")\n", + "tflite_file = Path(output[\"tflite_file\"])\n", + "print(\"Trained model:\", tflite_file)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 7. Download your model\n", + "\n", + "Download the `.tflite` file and copy it to your IMPSY setup:\n", + "\n", + "- **IMPSY / IMPSYpi:** put it in the `models` folder (or upload it on the web UI's Models page) and set `file` under `[model]` in `config.toml`, along with the matching `dimension` and `size`.\n", + "- **IMPSY AUv3 and IMPSY Web:** load the `.tflite` file in the app." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " from google.colab import files\n", + "\n", + " files.download(str(tflite_file))\n", + "except ImportError:\n", + " print(\"Your model is at\", tflite_file.resolve())" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} From d99cabcfeb21ce446828d3f16b390e4b1143e925 Mon Sep 17 00:00:00 2001 From: Charles Martin Date: Sat, 26 Sep 2026 18:31:02 +1000 Subject: [PATCH 2/3] Allow psutil 7 psutil was capped at <7 only because the original Poetry caret constraint (^6.0.0) was carried over in the PEP 621 migration. IMPSY only uses cpu_count() and virtual_memory(), which are unchanged in 7. The cap conflicts with current IPython (psutil>=7), e.g., when installing impsy in Colab. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016oUxTsghWZWfnCnnkKcuNv --- poetry.lock | 50 +++++++++++++++++++++++++++----------------------- pyproject.toml | 2 +- 2 files changed, 28 insertions(+), 24 deletions(-) diff --git a/poetry.lock b/poetry.lock index f62cb7f..fdaab29 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1420,34 +1420,38 @@ files = [ [[package]] name = "psutil" -version = "6.1.1" -description = "Cross-platform lib for process and system monitoring in Python." +version = "7.2.2" +description = "Cross-platform lib for process and system monitoring." optional = false -python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7" +python-versions = ">=3.6" groups = ["main"] files = [ - {file = "psutil-6.1.1-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:9ccc4316f24409159897799b83004cb1e24f9819b0dcf9c0b68bdcb6cefee6a8"}, - {file = "psutil-6.1.1-cp27-cp27m-manylinux2010_i686.whl", hash = "sha256:ca9609c77ea3b8481ab005da74ed894035936223422dc591d6772b147421f777"}, - {file = "psutil-6.1.1-cp27-cp27m-manylinux2010_x86_64.whl", hash = "sha256:8df0178ba8a9e5bc84fed9cfa61d54601b371fbec5c8eebad27575f1e105c0d4"}, - {file = "psutil-6.1.1-cp27-cp27mu-manylinux2010_i686.whl", hash = "sha256:1924e659d6c19c647e763e78670a05dbb7feaf44a0e9c94bf9e14dfc6ba50468"}, - {file = "psutil-6.1.1-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:018aeae2af92d943fdf1da6b58665124897cfc94faa2ca92098838f83e1b1bca"}, - {file = "psutil-6.1.1-cp27-none-win32.whl", hash = "sha256:6d4281f5bbca041e2292be3380ec56a9413b790579b8e593b1784499d0005dac"}, - {file = "psutil-6.1.1-cp27-none-win_amd64.whl", hash = "sha256:c777eb75bb33c47377c9af68f30e9f11bc78e0f07fbf907be4a5d70b2fe5f030"}, - {file = "psutil-6.1.1-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:fc0ed7fe2231a444fc219b9c42d0376e0a9a1a72f16c5cfa0f68d19f1a0663e8"}, - {file = "psutil-6.1.1-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:0bdd4eab935276290ad3cb718e9809412895ca6b5b334f5a9111ee6d9aff9377"}, - {file = "psutil-6.1.1-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b6e06c20c05fe95a3d7302d74e7097756d4ba1247975ad6905441ae1b5b66003"}, - {file = "psutil-6.1.1-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:97f7cb9921fbec4904f522d972f0c0e1f4fabbdd4e0287813b21215074a0f160"}, - {file = "psutil-6.1.1-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:33431e84fee02bc84ea36d9e2c4a6d395d479c9dd9bba2376c1f6ee8f3a4e0b3"}, - {file = "psutil-6.1.1-cp36-cp36m-win32.whl", hash = "sha256:384636b1a64b47814437d1173be1427a7c83681b17a450bfc309a1953e329603"}, - {file = "psutil-6.1.1-cp36-cp36m-win_amd64.whl", hash = "sha256:8be07491f6ebe1a693f17d4f11e69d0dc1811fa082736500f649f79df7735303"}, - {file = "psutil-6.1.1-cp37-abi3-win32.whl", hash = "sha256:eaa912e0b11848c4d9279a93d7e2783df352b082f40111e078388701fd479e53"}, - {file = "psutil-6.1.1-cp37-abi3-win_amd64.whl", hash = "sha256:f35cfccb065fff93529d2afb4a2e89e363fe63ca1e4a5da22b603a85833c2649"}, - {file = "psutil-6.1.1.tar.gz", hash = "sha256:cf8496728c18f2d0b45198f06895be52f36611711746b7f30c464b422b50e2f5"}, + {file = "psutil-7.2.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:2edccc433cbfa046b980b0df0171cd25bcaeb3a68fe9022db0979e7aa74a826b"}, + {file = "psutil-7.2.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e78c8603dcd9a04c7364f1a3e670cea95d51ee865e4efb3556a3a63adef958ea"}, + {file = "psutil-7.2.2-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1a571f2330c966c62aeda00dd24620425d4b0cc86881c89861fbc04549e5dc63"}, + {file = "psutil-7.2.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:917e891983ca3c1887b4ef36447b1e0873e70c933afc831c6b6da078ba474312"}, + {file = "psutil-7.2.2-cp313-cp313t-win_amd64.whl", hash = "sha256:ab486563df44c17f5173621c7b198955bd6b613fb87c71c161f827d3fb149a9b"}, + {file = "psutil-7.2.2-cp313-cp313t-win_arm64.whl", hash = "sha256:ae0aefdd8796a7737eccea863f80f81e468a1e4cf14d926bd9b6f5f2d5f90ca9"}, + {file = "psutil-7.2.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:eed63d3b4d62449571547b60578c5b2c4bcccc5387148db46e0c2313dad0ee00"}, + {file = "psutil-7.2.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7b6d09433a10592ce39b13d7be5a54fbac1d1228ed29abc880fb23df7cb694c9"}, + {file = "psutil-7.2.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fa4ecf83bcdf6e6c8f4449aff98eefb5d0604bf88cb883d7da3d8d2d909546a"}, + {file = "psutil-7.2.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e452c464a02e7dc7822a05d25db4cde564444a67e58539a00f929c51eddda0cf"}, + {file = "psutil-7.2.2-cp314-cp314t-win_amd64.whl", hash = "sha256:c7663d4e37f13e884d13994247449e9f8f574bc4655d509c3b95e9ec9e2b9dc1"}, + {file = "psutil-7.2.2-cp314-cp314t-win_arm64.whl", hash = "sha256:11fe5a4f613759764e79c65cf11ebdf26e33d6dd34336f8a337aa2996d71c841"}, + {file = "psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486"}, + {file = "psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979"}, + {file = "psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9"}, + {file = "psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e"}, + {file = "psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8"}, + {file = "psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc"}, + {file = "psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988"}, + {file = "psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee"}, + {file = "psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372"}, ] [package.extras] -dev = ["abi3audit", "black", "check-manifest", "coverage", "packaging", "pylint", "pyperf", "pypinfo", "pytest-cov", "requests", "rstcheck", "ruff", "sphinx", "sphinx_rtd_theme", "toml-sort", "twine", "virtualenv", "vulture", "wheel"] -test = ["pytest", "pytest-xdist", "setuptools"] +dev = ["abi3audit", "black", "check-manifest", "colorama", "coverage", "packaging", "psleak", "pylint", "pyperf", "pypinfo", "pyreadline3", "pytest", "pytest-cov", "pytest-instafail", "pytest-xdist", "pywin32", "requests", "rstcheck", "ruff", "setuptools", "sphinx", "sphinx_rtd_theme", "toml-sort", "twine", "validate-pyproject[all]", "virtualenv", "vulture", "wheel", "wheel", "wmi"] +test = ["psleak", "pytest", "pytest-instafail", "pytest-xdist", "pywin32", "setuptools", "wheel", "wmi"] [[package]] name = "pycodestyle" @@ -2096,4 +2100,4 @@ dev = ["pytest", "setuptools"] [metadata] lock-version = "2.1" python-versions = ">=3.11,<3.14" -content-hash = "01841000ea45f2aaa03acc60e7e7239fa307c7fd38980bbb1c886fcb9bdf9f22" +content-hash = "d9c32f8466cf57054fe1acc5bf02aef681462cc765b02c1e0d7866753d050d97" diff --git a/pyproject.toml b/pyproject.toml index 86435e9..cf51010 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,7 +18,7 @@ dependencies = [ "python-rtmidi>=1.5.8,<2", "click>=8.1.7,<9", "flask>=3.0.3,<4", - "psutil>=6.0.0,<7", + "psutil>=6.0.0,<8", ] [project.urls] From 0c4aba3070b54d31c5e807ef92833ba0e24f12e9 Mon Sep 17 00:00:00 2001 From: Charles Martin Date: Sat, 26 Sep 2026 19:10:38 +1000 Subject: [PATCH 3/3] Check notebook dimension and log lengths before training The notebook now says which dimensions are available when DIMENSION has no logs, and stops before training when no log is longer than the training sequence, instead of failing inside Keras. The get-started page no longer says training has no cloud component, now that it links to Colab. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016oUxTsghWZWfnCnnkKcuNv --- docs/get-started.md | 2 +- notebooks/impsy_training.ipynb | 21 +++++++++++++++++++-- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/docs/get-started.md b/docs/get-started.md index 3322b75..41703a9 100644 --- a/docs/get-started.md +++ b/docs/get-started.md @@ -41,7 +41,7 @@ The web interface makes this straightforward: hit record, perform, save the log. ## 4. Train a mixture density RNN -IMPSY's prediction model is a small mixture density recurrent network. Training runs locally: no cloud component, no account, no required GPU for modestly sized models. Training a usable instrument on a few minutes of data takes minutes, not hours. +IMPSY's prediction model is a small mixture density recurrent network. Training runs on your own computer, with no account and no GPU needed for modestly sized models. Training a usable instrument on a few minutes of data takes minutes, not hours. See the main repository for the current training command, hyperparameter defaults, and tips for tuning the mixture parameters. diff --git a/notebooks/impsy_training.ipynb b/notebooks/impsy_training.ipynb index 57a586c..99eae51 100644 --- a/notebooks/impsy_training.ipynb +++ b/notebooks/impsy_training.ipynb @@ -132,6 +132,10 @@ "\n", "if DIMENSION == 0:\n", " DIMENSION = dimensions.most_common(1)[0][0]\n", + "assert DIMENSION in dimensions, (\n", + " f\"There are no {DIMENSION}d logs. Choose one of: \"\n", + " + \", \".join(f\"{d}\" for d in sorted(dimensions))\n", + ")\n", "print(f\"Training a {MODEL_SIZE} model on {DIMENSION}d data.\")" ] }, @@ -141,7 +145,7 @@ "source": [ "## 5. Make a dataset\n", "\n", - "This collects all of your logs of the chosen dimension into one dataset file. It's good to have more than 10,000 interactions, but you can train on less." + "This collects all of your logs of the chosen dimension into one dataset file. It's good to have more than 10,000 interactions, but you can train on less. The model learns from runs of 50 events, so only logs with more than 51 events are used for training." ] }, { @@ -150,11 +154,24 @@ "metadata": {}, "outputs": [], "source": [ + "import numpy as np\n", "from impsy.dataset import generate_dataset\n", + "from impsy.train import SEQ_LEN\n", "\n", "Path(\"datasets\").mkdir(exist_ok=True)\n", "dataset_file = generate_dataset(DIMENSION, source=\"logs\", destination=\"datasets\")\n", - "assert dataset_file is not None, \"The dataset is empty; check your logs contain 'interface' rows.\"" + "assert dataset_file is not None, (\n", + " f\"No data found in the {DIMENSION}d logs. Logs need rows recorded from \"\n", + " \"your interface (with 'interface' in the second column).\"\n", + ")\n", + "\n", + "with np.load(dataset_file, allow_pickle=True) as loaded:\n", + " long_enough = [p for p in loaded[\"perfs\"] if len(p) > SEQ_LEN + 1]\n", + "print(f\"{len(long_enough)} log(s) long enough to train on.\")\n", + "assert long_enough, (\n", + " f\"None of your logs have more than {SEQ_LEN + 1} events, \"\n", + " \"so there's nothing to train on. Record some longer logs.\"\n", + ")" ] }, {