Overview / Background / Motivation

Overview

BackPropagationVisualization is a web application which shows the weights and activation values in a basic neural network, and allows you to observe changes in these values as the network is trained. The neural network learns how to apply basic logic functions AND, OR, and XOR, and hence has just 2 input units and a single output unit, with a single layer of 5 hidden units. The training algorithm uses gradient descent with backpropagation, and the only form of regularization being adjustment of the learning rate... avoiding more advanced techniques like early stopping, momentum, weight decay, etc... to keep the training process and the code simple and easy to follow.

The neural network is implemented as part of a .NET class library (the class BasicFeedForwardNeuralNetwork from my project SimpleML). The backend of the application is built as an ASP.NET Web API, whilst the frontend/presentation is written in AngularJS, using Google Material Design components.

BackPropagationVisualization Screen Shot

Background and Motivation

As there's been a lot of 'buzz' around machine learning over the past few years, I took two Coursera courses... Andrew Ng's Machine Learning primer, and then later the more specialized Neural Networks course taught by Geoffrey Hinton. Feed forward neural networks were the topic in these courses that stood out for me... the foundation of many complex and cutting edge applications of machine learning (image recognition, self-driving cars, etc...), but with somewhat mysterious inner-workings based on fairly complex maths (at least from the perspective of a software engineer). Both courses involved assignments which used feed forward neural networks to recognise handwritten digits, using the MNIST database as a training source. The capability and application of these networks was fascinating... being able to recognise new handwritten digits using only existing digits as a source (rather than having to use bespoke code to recognise unique features of the digits). But, in both courses the networks involved large matrices of input/hidden/output units and weights, and operations in the backpropagation process were somewhat abstracted and hidden by bulk operations on these matrices. So whilst the overall result/effect of the training was easy to comprehend, the inner workings of the network during that training was not.

BackPropagationVisualization implements a much simpler use case... taking 2 binary values as input and learning to apply AND, OR, and XOR logic functions to produce a single binary output. As a result, the network contains just 2 input units, a single output unit, and 5 hidden units. Also, as there are only 4 possible permutations of input values ( {False,False}; {False,True}; {True,False}; {True,True} ), there are only 4 training cases. This results in a much simpler neural network than those in the Coursera courses...

Neural Network Input Units Hidden Units Output Units Training Examples
BackPropagationVisualization 2 5 1 4
'Machine Learning' Programming Exercise 4 400 25 10 5000
'Neural Networks' Assignment 3 256 7 10 1000

The BasicFeedForwardNeuralNetwork class implementing the neural network uses arrays and for loops rather than matrices and matrix operations, so while maybe not as performant, it's simpler to understand and follow.

An additional motivation was to build the front end of the application in one of the rich, new (at the time) javascript libraries available, so the user interface is written using AngularJS.