Tutorial 3

In this tutorial, we are going to directly train a simple SNN with a single hidden layer using EventProp on the MNIST dataset, converted to a latency spike code.

Clearly, this is far from a state of the art architecture, but it still achieves 96% accuracy on MNIST.

Install

Download wheel file

[1]:

if "google.colab" in str(get_ipython()):
    !gdown 1V_GzXUDzcFz9QDIpxAD8QNEglcSipssW
    !pip install pygenn-5.0.0-cp310-cp310-linux_x86_64.whl
    %env CUDA_PATH=/usr/local/cuda

    !rm -rf /content/ml_genn
    !git clone https://github.com/genn-team/ml_genn.git --branch genn_5 -c advice.detachedHead=false
    !pip install ./ml_genn/ml_genn

Downloading...
From: https://drive.google.com/uc?id=1V_GzXUDzcFz9QDIpxAD8QNEglcSipssW
To: /content/pygenn-5.0.0-cp310-cp310-linux_x86_64.whl

0% 0.00/8.29M [00:00<?, ?B/s]

100% 8.29M/8.29M [00:00<00:00, 139MB/s]

Processing ./pygenn-5.0.0-cp310-cp310-linux_x86_64.whl Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from pygenn==5.0.0) (1.25.2) Requirement already satisfied: deprecated in /usr/local/lib/python3.10/dist-packages (from pygenn==5.0.0) (1.2.14) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from pygenn==5.0.0) (5.9.5) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated->pygenn==5.0.0) (1.14.1) pygenn is already installed with the same version as the provided wheel. Use –force-reinstall to force an installation of the wheel. env: CUDA_PATH=/usr/local/cuda Cloning into ‘ml_genn’… remote: Enumerating objects: 8248, done. remote: Counting objects: 100% (555/555), done. remote: Compressing objects: 100% (330/330), done. remote: Total 8248 (delta 278), reused 300 (delta 222), pack-reused 7693 Receiving objects: 100% (8248/8248), 37.43 MiB | 19.40 MiB/s, done. Resolving deltas: 100% (5510/5510), done. Processing ./ml_genn/ml_genn

Preparing metadata (setup.py) … done

Requirement already satisfied: pygenn<6.0.0,>=5.0.0 in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (5.0.0) Requirement already satisfied: enum-compat in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (0.0.3) Requirement already satisfied: tqdm>=4.27.0 in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (4.66.2) Requirement already satisfied: deprecated in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (1.2.14) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from pygenn<6.0.0,>=5.0.0->ml-genn==2.1.0) (1.25.2) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from pygenn<6.0.0,>=5.0.0->ml-genn==2.1.0) (5.9.5) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated->ml-genn==2.1.0) (1.14.1) Building wheels for collected packages: ml-genn

Building wheel for ml-genn (setup.py) … done Created wheel for ml-genn: filename=ml_genn-2.1.0-py3-none-any.whl size=109412 sha256=c42c80643c3f51294cd7cdde73a3dc7a0c77b1963cc4a967c1603cd7ff30597c Stored in directory: /tmp/pip-ephem-wheel-cache-crg955fq/wheels/3f/cf/27/0e9dec4bb1be2afac4b38c2dfb4ce0bc164ce1ecb32b6f91b8

Successfully built ml-genn Installing collected packages: ml-genn

Attempting uninstall: ml-genn
Found existing installation: ml-genn 2.1.0 Uninstalling ml-genn-2.1.0:

Successfully uninstalled ml-genn-2.1.0

Successfully installed ml-genn-2.1.0 </pre>

0% 0.00/8.29M [00:00<?, ?B/s]

100% 8.29M/8.29M [00:00<00:00, 139MB/s]

Processing ./pygenn-5.0.0-cp310-cp310-linux_x86_64.whl Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from pygenn==5.0.0) (1.25.2) Requirement already satisfied: deprecated in /usr/local/lib/python3.10/dist-packages (from pygenn==5.0.0) (1.2.14) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from pygenn==5.0.0) (5.9.5) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated->pygenn==5.0.0) (1.14.1) pygenn is already installed with the same version as the provided wheel. Use –force-reinstall to force an installation of the wheel. env: CUDA_PATH=/usr/local/cuda Cloning into ‘ml_genn’{ldots} remote: Enumerating objects: 8248, done. remote: Counting objects: 100% (555/555), done. remote: Compressing objects: 100% (330/330), done. remote: Total 8248 (delta 278), reused 300 (delta 222), pack-reused 7693 Receiving objects: 100% (8248/8248), 37.43 MiB | 19.40 MiB/s, done. Resolving deltas: 100% (5510/5510), done. Processing ./ml_genn/ml_genn

Preparing metadata (setup.py) {ldots} done

Requirement already satisfied: pygenn<6.0.0,>=5.0.0 in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (5.0.0) Requirement already satisfied: enum-compat in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (0.0.3) Requirement already satisfied: tqdm>=4.27.0 in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (4.66.2) Requirement already satisfied: deprecated in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (1.2.14) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from pygenn<6.0.0,>=5.0.0->ml-genn==2.1.0) (1.25.2) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from pygenn<6.0.0,>=5.0.0->ml-genn==2.1.0) (5.9.5) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated->ml-genn==2.1.0) (1.14.1) Building wheels for collected packages: ml-genn

Building wheel for ml-genn (setup.py) {ldots} done Created wheel for ml-genn: filename=ml_genn-2.1.0-py3-none-any.whl size=109412 sha256=c42c80643c3f51294cd7cdde73a3dc7a0c77b1963cc4a967c1603cd7ff30597c Stored in directory: /tmp/pip-ephem-wheel-cache-crg955fq/wheels/3f/cf/27/0e9dec4bb1be2afac4b38c2dfb4ce0bc164ce1ecb32b6f91b8

Successfully built ml-genn Installing collected packages: ml-genn

Attempting uninstall: ml-genn
Found existing installation: ml-genn 2.1.0 Uninstalling ml-genn-2.1.0:

Successfully uninstalled ml-genn-2.1.0

Successfully installed ml-genn-2.1.0 end{sphinxVerbatim}

0% 0.00/8.29M [00:00<?, ?B/s]

100% 8.29M/8.29M [00:00<00:00, 139MB/s]

Processing ./pygenn-5.0.0-cp310-cp310-linux_x86_64.whl Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from pygenn==5.0.0) (1.25.2) Requirement already satisfied: deprecated in /usr/local/lib/python3.10/dist-packages (from pygenn==5.0.0) (1.2.14) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from pygenn==5.0.0) (5.9.5) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated->pygenn==5.0.0) (1.14.1) pygenn is already installed with the same version as the provided wheel. Use –force-reinstall to force an installation of the wheel. env: CUDA_PATH=/usr/local/cuda Cloning into ‘ml_genn’… remote: Enumerating objects: 8248, done.[K remote: Counting objects: 100% (555/555), done.[K remote: Compressing objects: 100% (330/330), done.[K remote: Total 8248 (delta 278), reused 300 (delta 222), pack-reused 7693[K Receiving objects: 100% (8248/8248), 37.43 MiB | 19.40 MiB/s, done. Resolving deltas: 100% (5510/5510), done. Processing ./ml_genn/ml_genn

Preparing metadata (setup.py) … [?25l[?25hdone

Requirement already satisfied: pygenn<6.0.0,>=5.0.0 in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (5.0.0) Requirement already satisfied: enum-compat in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (0.0.3) Requirement already satisfied: tqdm>=4.27.0 in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (4.66.2) Requirement already satisfied: deprecated in /usr/local/lib/python3.10/dist-packages (from ml-genn==2.1.0) (1.2.14) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from pygenn<6.0.0,>=5.0.0->ml-genn==2.1.0) (1.25.2) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from pygenn<6.0.0,>=5.0.0->ml-genn==2.1.0) (5.9.5) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated->ml-genn==2.1.0) (1.14.1) Building wheels for collected packages: ml-genn

Building wheel for ml-genn (setup.py) … [?25l[?25hdone Created wheel for ml-genn: filename=ml_genn-2.1.0-py3-none-any.whl size=109412 sha256=c42c80643c3f51294cd7cdde73a3dc7a0c77b1963cc4a967c1603cd7ff30597c Stored in directory: /tmp/pip-ephem-wheel-cache-crg955fq/wheels/3f/cf/27/0e9dec4bb1be2afac4b38c2dfb4ce0bc164ce1ecb32b6f91b8

Successfully built ml-genn Installing collected packages: ml-genn

Attempting uninstall: ml-genn
Found existing installation: ml-genn 2.1.0 Uninstalling ml-genn-2.1.0:

Successfully uninstalled ml-genn-2.1.0

Successfully installed ml-genn-2.1.0

Install MNIST package

[2]:

!pip install mnist

Collecting mnist
  Using cached mnist-0.2.2-py2.py3-none-any.whl (3.5 kB)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from mnist) (1.25.2)
Installing collected packages: mnist
Successfully installed mnist-0.2.2

Build model

Import standard modules and required mlGeNN classes

[3]:

import mnist
import numpy as np
import matplotlib.pyplot as plt

from ml_genn import InputLayer, Layer, SequentialNetwork
from ml_genn.callbacks import Checkpoint
from ml_genn.compilers import EventPropCompiler, InferenceCompiler
from ml_genn.connectivity import Dense
from ml_genn.initializers import Normal
from ml_genn.neurons import LeakyIntegrate, LeakyIntegrateFire, SpikeInput
from ml_genn.optimisers import Adam
from ml_genn.serialisers import Numpy
from ml_genn.synapses import Exponential

from ml_genn.utils.data import (calc_latest_spike_time, linear_latency_encode_data)

from ml_genn.compilers.event_prop_compiler import default_params

##Parameters

Define some model parameters

[4]:

NUM_INPUT = 28 * 28
NUM_HIDDEN = 128
NUM_OUTPUT = 10
BATCH_SIZE = 128

Latency encoding

There are numerous ways to encode images using spikes but here we are going to emit a single spike for each neuron at a time proportional the each pixel’s grayscale.

[5]:

train_spikes = linear_latency_encode_data(mnist.train_images(), 20.0)

Network definition

Because our network is entirely feedforward, we can define it as a SequentialNetwork where each layer is automatically connected to the previous layer. As we have converted the MNIST dataset to spikes, we will use a SpikeInput to inject these directly into the network. For our hidden layer we are going to use standard Leaky integrate-and-fire neurons. Finally, we are going to use a non-spiking output layer and read classifications out of this by determening the maximum of the output neurons’ averaged membrane voltages.

[6]:

# Create sequential model
serialiser = Numpy("latency_mnist_checkpoints")
network = SequentialNetwork(default_params)
with network:
    # Populations
    input = InputLayer(SpikeInput(max_spikes=BATCH_SIZE * NUM_INPUT),
                                  NUM_INPUT)
    hidden = Layer(Dense(Normal(mean=0.078, sd=0.045)),
                   LeakyIntegrateFire(v_thresh=1.0, tau_mem=20.0,
                                      tau_refrac=None),
                   NUM_HIDDEN, Exponential(5.0))
    output = Layer(Dense(Normal(mean=0.2, sd=0.37)),
                   LeakyIntegrate(tau_mem=20.0, readout="avg_var"),
                   NUM_OUTPUT, Exponential(5.0))

Compilation

In mlGeNN, in order to turn an abstract network description into something that can actually be used for training or inference you use a compiler class. Here, we use the EventPropCompiler to train with EventProp and specify batch size and how many timesteps to evaluate each example for as well as choosing our optimiser and loss function. Because this is a classification task, we want to use cross-entropy loss and, because our labels are specified in this way (rather than e.g. one-hot encoded), we use the sparse catgorical variant.

[7]:

compiler = EventPropCompiler(example_timesteps=20,
                         losses="sparse_categorical_crossentropy",
                         optimiser=Adam(1e-2), batch_size=BATCH_SIZE)
compiled_net = compiler.compile(network)

Training

Now we will train the model for 10 epochs using our compiled network. To verify its performance we take 10% of the training data as a validation split and add an additional callback to checkpoint weights every epoch.

[8]:

with compiled_net:
    # Evaluate model on numpy dataset
    callbacks = ["batch_progress_bar", Checkpoint(serialiser)]
    compiled_net.train({input: train_spikes},
                       {output: mnist.train_labels()},
                       num_epochs=15, shuffle=True,
                       validation_split=0.1,
                       callbacks=callbacks)

Evaluate

Load weights checkpointed from last epoch:

[9]:

network.load((14,), serialiser)

Create an InferenceCompiler and compile network for inference:

[10]:

compiler = InferenceCompiler(evaluate_timesteps=20,
                             reset_in_syn_between_batches=True,
                             batch_size=BATCH_SIZE)
compiled_net = compiler.compile(network)

Encode test set using the same log-latency encoding and evaluate it:

[11]:

test_spikes = linear_latency_encode_data(mnist.test_images(), 20.0)
with compiled_net:
    compiled_net.evaluate({input: test_spikes},
                          {output: mnist.test_labels()})