Skip to content

About

A small deep learning framework built from scratch on top of NumPy, implementing reverse-mode automatic differentiation and a PyTorch-like nn API.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

20 Commits

Folders and files

Repository files navigation

Tenzor

Tenzor is a deep learning framework written from scratch on top of NumPy. It implements reverse-mode automatic differentiation and an nn package with layers, losses and optimizers. The API follows PyTorch. A three-layer MLP trained with Tenzor reaches 97.65% test accuracy on MNIST after two epochs.

Predictions of the larger MLP on MNIST test images

Background

Tenzor was written to learn how PyTorch works internally. The goal was to understand more than the gradient computation. It also covers how layers, losses and optimizers are built on top of automatic differentiation. This includes what backward(), Linear.forward() and optimizer.step() do. The design follows PyTorch.

Usage

Requirements: Python 3.12 and NumPy. PyTorch and torchvision are only needed for the tests and for loading MNIST in the demo.

git clone https://github.com/philipp-sz/tenzor_python.git
cd tenzor_python
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

A training step looks the same as in PyTorch:

import numpy as np
from tenzor import Tensor
from tenzor.nn import Sequential, Linear, ReLU, CrossEntropyLoss, SGD

model = Sequential(Linear(784, 128), ReLU(), Linear(128, 10))
criterion = CrossEntropyLoss()
optimizer = SGD(model.parameters(), lr=0.01)

x = Tensor(np.random.randn(32, 784))
y = Tensor(np.random.randint(0, 10, size=32))

model.train()
loss = criterion(model(x), y)
optimizer.zero_grad()
loss.backward()
optimizer.step()

demo.ipynb trains two MLPs on MNIST. pytest runs the tests.

Features

  • Tensor operations: Add, Mul, Pow, MatMul, GetItem, Sum, Mean, Min, Max, Maximum, Minimum, Exp, Log, Reshape, Transpose, Squeeze, Unsqueeze, with broadcasting in Add and Mul.
  • Layers: Linear, Flatten, Sequential, Dropout, LayerNorm, BatchNorm, Embedding, MultiHeadAttention.
  • Activations: ReLU, LeakyReLU, Sigmoid, Tanh, Softmax.
  • Losses: MSELoss, MAELoss, CrossEntropyLoss.
  • Optimizers: SGD with momentum, Adagrad, RMSprop, Adam, AdamW.
  • 96 tests. Forward results and gradients are compared against PyTorch. Optimizers are tested for convergence.

Results

demo.ipynb trains on MNIST for two epochs with batch size 64 and seed 0. Losses are the mean cross-entropy over all samples of the epoch or the test set.

Model Optimizer Train loss, epoch 1 Train loss, epoch 2 Test loss Test accuracy
Linear(784, 128), ReLU, Linear(128, 10) SGD, lr 0.01 0.4713 0.2600 0.2264 93.45%
3 hidden layers (512, 256, 128) with BatchNorm, LeakyReLU, Dropout(0.2) AdamW, lr 0.001 0.2413 0.1197 0.0765 97.65%

How it works

Tensor and autograd

Tensor wraps a NumPy array and adds requires_grad, grad and grad_fn. Every operation is a Function subclass with a forward and a backward method. Function.apply() runs forward and, if any input requires gradients, stores the inputs and sets the result's grad_fn. The graph is built while the code runs (define-by-run).

Tensor.backward() sorts the graph topologically with a depth-first search that visits each node once. It then calls each grad_fn.backward() in reverse order and adds the returned gradients to the inputs' .grad. A node that is reached over two paths therefore receives the sum of both gradients. For example, with a of shape (2, 2) and b of shape (2,), (a * b + b).sum().backward() gives b.grad = a.sum(axis=0) + 2.

Broadcasting

Add and Mul accept NumPy broadcasting in the forward pass. In the backward pass the gradient is summed over every broadcast axis, so it has the shape of the original input. A bias of shape (10,) added to a batch of shape (100, 10) receives the sum of the 100 row gradients.

Modules

Module.__setattr__ registers every Tensor attribute as a parameter and every Module attribute as a submodule. parameters() collects them recursively, and train() and eval() set the mode on all submodules. Linear stores its weight as (in_features, out_features), computes x @ W + b and uses He initialization (standard deviation sqrt(2 / in_features)). Dropout uses inverted dropout. BatchNorm keeps running mean and variance for evaluation.

Losses and optimizers

CrossEntropyLoss computes the log-softmax with the log-sum-exp form, x - max(x) - log(sum(exp(x - max(x)))). This avoids taking the log of a softmax that has underflowed to 0. AdamW applies weight decay directly to the parameters after the Adam update. Added to the gradient instead, the decay term would be divided by the adaptive denominator sqrt(v), so parameters with large past gradients would be regularized less.

Project structure

tenzor/
  tensor.py          Tensor, Function and all operations
  nn/layers.py       Module, layers, activations, losses
  nn/optimizers.py   optimizers
tests/               comparisons against PyTorch
demo.ipynb           MNIST training

Limitations

  • Runs on the CPU only and is not optimized for speed.
  • No Conv2d, and no Dataset or DataLoader. The demo loads MNIST with torchvision.
  • Embedding and MultiHeadAttention are tested in the forward pass only. Their gradients are not compared against PyTorch, and neither was used in a training run.

License

MIT, see LICENSE.

About

A small deep learning framework built from scratch on top of NumPy, implementing reverse-mode automatic differentiation and a PyTorch-like nn API.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages