mnistidx

A Go package for reading MNIST IDX files. This library provides a simple API for sequentially reading handwritten digit images and their corresponding labels from MNIST image and label streams.

About MNIST

The MNIST database is a large dataset of handwritten digits commonly used for training and testing machine learning algorithms. It contains:

• Training set: 60,000 images with labels
• Test set: 10,000 images with labels
• Image size: 28×28 pixels (grayscale)
• Labels: Digits 0-9

Features

• Simple API: Read image-label pairs sequentially with a single reader
• Streaming-friendly: Reads data on demand instead of buffering the full dataset
• Typed content: Uses image.ImageContent and label.LabelContent for clarity
• Validation: Ensures image count and label count match before reading

Installation

go get github.com/boolka/mnistidx

Quick Start

package main

import (
	"bytes"
	"fmt"
	"io"

	"github.com/boolka/mnistdb/pkg/mnistdb"
	"github.com/boolka/mnistidx/pkg/mnistidx"
)

func main() {
	idx, err := mnistidx.NewIDX(
		bytes.NewReader(mnistdb.TrainImages),
		bytes.NewReader(mnistdb.TrainLabels),
	)
	if err != nil {
		panic(err)
	}

	buf := idx.NewBuffer()

	count := 0
	for {
		label, err := idx.Read(buf)
		if err == io.EOF {
			break
		}
		if err != nil {
			panic(err)
		}

		fmt.Printf("Image %d: Label=%d\n", count, label)
		count++
	}

	fmt.Printf("Total images read: %d\n", count)
}

Converting to image.Image

After reading image bytes with idx.Read, you can convert the returned buffer into a standard Go image.Image with GetNRGBAImg:

img := buf.GetNRGBAImg(idx.ImagesHeader)
fmt.Println("Image bounds:", img.Bounds())

Example 1: Reading First 10 Images

package main

import (
	"bytes"
	"io"

	"github.com/boolka/mnistdb/pkg/mnistdb"
	"github.com/boolka/mnistidx/pkg/mnistidx"
)

func main() {
	idx, _ := mnistidx.NewIDX(
		bytes.NewReader(mnistdb.TrainImages),
		bytes.NewReader(mnistdb.TrainLabels),
	)

	buf := idx.NewBuffer()

	for i := 0; i < 10; i++ {
		label, err := idx.Read(buf)
		if err == io.EOF {
			break
		}
		println("Label:", label)
	}
}

Example 2: Image Statistics

package main

import (
	"bytes"
	"io"

	"github.com/boolka/mnistdb/pkg/mnistdb"
	"github.com/boolka/mnistidx/pkg/mnistidx"
)

func main() {
	idx, _ := mnistidx.NewIDX(
		bytes.NewReader(mnistdb.TrainImages),
		bytes.NewReader(mnistdb.TrainLabels),
	)

	buf := idx.NewBuffer()
	labelCounts := [10]int{}

	for {
		label, err := idx.Read(buf)
		if err == io.EOF {
			break
		}
		labelCounts[label]++
	}

	for digit, count := range labelCounts {
		println("Digit", digit, "count:", count)
	}
}

Example 3: Building a Dataset with Batches

package main

import (
	"bytes"
	"fmt"
	"io"

	"github.com/boolka/mnistdb/pkg/mnistdb"
	"github.com/boolka/mnistidx/pkg/mnistidx"
)

func main() {
	idx, _ := mnistidx.NewIDX(
		bytes.NewReader(mnistdb.TrainImages),
		bytes.NewReader(mnistdb.TrainLabels),
	)

	batchSize := 32
	imageBuf := idx.NewBuffer()
	batch := make([]byte, batchSize*len(imageBuf))
	labels := make([]int8, batchSize)

	for batchIdx := 0; ; batchIdx++ {
		for i := 0; i < batchSize; i++ {
			label, err := idx.Read(imageBuf)
			if err == io.EOF {
				fmt.Printf("Loaded %d batches\n", batchIdx)
				return
			}
			copy(batch[i*len(imageBuf):(i+1)*len(imageBuf)], imageBuf)
			labels[i] = int8(label)
		}
		fmt.Printf("Loaded batch %d with %d images\n", batchIdx, batchSize)
	}
}

Output:

Loaded batch 0 with 32 images
Loaded batch 1 with 32 images
Loaded batch 2 with 32 images
...
Loaded 1875 batches

API Reference

MnistIDX

The main type for reading MNIST IDX format files.

Constructor

func NewIDX(imagesReader, labelsReader io.Reader) (*MnistIDX, error)

Creates a new MNIST IDX reader from image and label readers.

• Returns: *MnistIDX or an error if:
  • image or label file format is invalid
  • image and label counts don't match

Fields

ImagesHeader *image.ImageHeader
LabelsHeader *label.LabelHeader

ImageHeader fields:

• MN: Magic number (2051)
• ImagesCount: Number of images in the file
• ImgRows: Number of rows per image
• ImgCols: Number of columns per image

LabelHeader fields:

• MN: Magic number (2049)
• LabelsCount: Number of labels in the file

Methods

func (i *MnistIDX) NewBuffer() image.ImageContent

Returns a correctly sized image buffer for one image.

func (i *MnistIDX) Read(buf image.ImageContent) (label.LabelContent, error)

Reads the next image into buf and returns the corresponding label.

• Parameters:
  • buf: buffer returned by NewBuffer()
• Returns:
  • label: digit label (0-9)
  • error: io.EOF when all images are read, or any read error

Requirements

• Go 1.23.4 or later
• github.com/boolka/mnistdb for sample/test MNIST data

License

MIT License - see LICENSE file for details.