Tenzor is a deep learning framework written from scratch on top of NumPy. It implements reverse-mode automatic differentiation and an nn package with layers, losses and optimizers. The API follows PyTorch. A three-layer MLP trained with Tenzor reaches 97.65% test accuracy on MNIST after two epochs.
Tenzor was written to learn how PyTorch works internally. The goal was to understand more than the gradient computation. It also covers how layers, losses and optimizers are built on top of automatic differentiation. This includes what backward(), Linear.forward() and optimizer.step() do. The design follows PyTorch.
Requirements: Python 3.12 and NumPy. PyTorch and torchvision are only needed for the tests and for loading MNIST in the demo.
git clone https://github.com/philipp-sz/tenzor_python.git
cd tenzor_python
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtA training step looks the same as in PyTorch:
import numpy as np
from tenzor import Tensor
from tenzor.nn import Sequential, Linear, ReLU, CrossEntropyLoss, SGD
model = Sequential(Linear(784, 128), ReLU(), Linear(128, 10))
criterion = CrossEntropyLoss()
optimizer = SGD(model.parameters(), lr=0.01)
x = Tensor(np.random.randn(32, 784))
y = Tensor(np.random.randint(0, 10, size=32))
model.train()
loss = criterion(model(x), y)
optimizer.zero_grad()
loss.backward()
optimizer.step()demo.ipynb trains two MLPs on MNIST. pytest runs the tests.
- Tensor operations:
Add,Mul,Pow,MatMul,GetItem,Sum,Mean,Min,Max,Maximum,Minimum,Exp,Log,Reshape,Transpose,Squeeze,Unsqueeze, with broadcasting inAddandMul. - Layers:
Linear,Flatten,Sequential,Dropout,LayerNorm,BatchNorm,Embedding,MultiHeadAttention. - Activations:
ReLU,LeakyReLU,Sigmoid,Tanh,Softmax. - Losses:
MSELoss,MAELoss,CrossEntropyLoss. - Optimizers:
SGDwith momentum,Adagrad,RMSprop,Adam,AdamW. - 96 tests. Forward results and gradients are compared against PyTorch. Optimizers are tested for convergence.
demo.ipynb trains on MNIST for two epochs with batch size 64 and seed 0. Losses are the mean cross-entropy over all samples of the epoch or the test set.
| Model | Optimizer | Train loss, epoch 1 | Train loss, epoch 2 | Test loss | Test accuracy |
|---|---|---|---|---|---|
Linear(784, 128), ReLU, Linear(128, 10) |
SGD, lr 0.01 |
0.4713 | 0.2600 | 0.2264 | 93.45% |
3 hidden layers (512, 256, 128) with BatchNorm, LeakyReLU, Dropout(0.2) |
AdamW, lr 0.001 |
0.2413 | 0.1197 | 0.0765 | 97.65% |
Tensor wraps a NumPy array and adds requires_grad, grad and grad_fn. Every operation is a Function subclass with a forward and a backward method. Function.apply() runs forward and, if any input requires gradients, stores the inputs and sets the result's grad_fn. The graph is built while the code runs (define-by-run).
Tensor.backward() sorts the graph topologically with a depth-first search that visits each node once. It then calls each grad_fn.backward() in reverse order and adds the returned gradients to the inputs' .grad. A node that is reached over two paths therefore receives the sum of both gradients. For example, with a of shape (2, 2) and b of shape (2,), (a * b + b).sum().backward() gives b.grad = a.sum(axis=0) + 2.
Add and Mul accept NumPy broadcasting in the forward pass. In the backward pass the gradient is summed over every broadcast axis, so it has the shape of the original input. A bias of shape (10,) added to a batch of shape (100, 10) receives the sum of the 100 row gradients.
Module.__setattr__ registers every Tensor attribute as a parameter and every Module attribute as a submodule. parameters() collects them recursively, and train() and eval() set the mode on all submodules. Linear stores its weight as (in_features, out_features), computes x @ W + b and uses He initialization (standard deviation sqrt(2 / in_features)). Dropout uses inverted dropout. BatchNorm keeps running mean and variance for evaluation.
CrossEntropyLoss computes the log-softmax with the log-sum-exp form, x - max(x) - log(sum(exp(x - max(x)))). This avoids taking the log of a softmax that has underflowed to 0. AdamW applies weight decay directly to the parameters after the Adam update. Added to the gradient instead, the decay term would be divided by the adaptive denominator sqrt(v), so parameters with large past gradients would be regularized less.
tenzor/
tensor.py Tensor, Function and all operations
nn/layers.py Module, layers, activations, losses
nn/optimizers.py optimizers
tests/ comparisons against PyTorch
demo.ipynb MNIST training
- Runs on the CPU only and is not optimized for speed.
- No
Conv2d, and noDatasetorDataLoader. The demo loads MNIST with torchvision. EmbeddingandMultiHeadAttentionare tested in the forward pass only. Their gradients are not compared against PyTorch, and neither was used in a training run.
MIT, see LICENSE.
