Explaining neural network training concepts through an interactive visualization

Using a neural network for complementary color prediction

Abstract

In this article we explore an interactive data visualization as a mean to help users understand neural network training concepts. The visualization let the users interact with the neural network training procedure, changing parameters and viewing results as they are shown on the browser. As most machine learning and neural network methods are opaque and hard to understand by the public in general, we explore this visualization as a way to teach the techniques and concepts around neural network model training, visually exposing the behavior of the training algorithm to the users playing with the visualization, while the network is being trained. In this article we describe one particular exploration, using the problem of predicting the complementary color of a RGB color, implemented as a neural network model.

Motivation

We believe that the best way to understand neural network concepts is by interacting with real-life neural network implementations. In this paper, we present an interactive web application, where it is possible to train a neural network to predict a complementary color in RGB, change its parameters, see the network converge, start over and run the model over and over again, adjusting parameters at will. Hopefully the user will eventually develop insights about neural network training concepts after some time playing with the application. By the way, the parameters of the demo are a bit off to start with, so it is up to the user to play with the parameters and improve them. There is an explanation of the interface here, and the application is live at the end of the article.

Neural Networks

Neural network concepts are neither easily understood, nor easily explained. Most explanations start with the McCulloch/Pitts neuron, build the idea of layers and weights, define backpropagation and may, or may not, end the discussion with learning rates, batch size, epochs, activation functions and other intricacies of neural network implementations.

But when an student or researcher has to implement a neural network, he or she will have to manipulate the very last layer of abstraction, as almost every library deliver layers, weights and backpropagation as well wrapped black boxes. It is during the very first neural network implementation that the beginner will have a better grasp of learning rates, batch sizes, activation units, network architecture, epochs and other concepts.

Keeping this student or researcher in mind, we developed a web based application, on top of tensorflow.js , a javascript inception of TensorFlow library and a spin-off of deeplearn.js (now Tensorflow.js core ). In this application the user can play with some of the model parameters (more options and functionality will appear in later versions) of an application and see the model training and predictions on a web browser, as it happens.

Visualizations and Explanations

Visualizations enable users to better understand and explore large sets of data or processes.

Different types of users require different types of explanations and learning strategies. We divide users into three categories, according to how knowledgeable they are:

  1. Layperson: has no knowledge of the process or algorithms being explained. This covers most of the general public;
  2. Knowledgeable: knows the concepts and algorithms, is familiar with the techniques involved;
  3. Designer: this is a person involved in the design of the solution. Although a knowledgeable individual, the designer might also need to better understand the system, in order to fine tune parameters.

This case was developed with knowledgeable in mind but a layperson can also explore it, as the demo assumes little prior knowledge of Neural Networks.

The model

To demonstrate a real neural network in action we selected the problem of predicting a complementary color from the RGB color space. The idea was adapted from a deeplearn.js (now Tensorflow.js core) tutorial, called Complementary Color Prediction, developed by Chi Zeng, from Google. The original example was developed in TypeScript but as the library was eventually deprecated, the code, apparently, is not available anymore. Below you can see the table output of the example developed by Chi Zheng. The original code automatically triggered the neural network training and updated the table below accordingly, there was no options to initialize, adjust or tweak the network training parameters.

Initially we've built a D3.js visualization for the neural network output and corrected an small issue with activation units. Later, when deeplearn.js moved to tensorflow.js, we rewrote the whole application in JavaScript with tensorflow.js and built the interactive demo described in this article.

Why complementary colors?

The main reasons for choosing a model to predict complementary colors are:

Implementation

The model was implemented in JavaScript with an architecture of 3 hidden layers. One batch of data/label pairs is generated randomically for each training round, as shown in the code below.

async function train1Batch() { // Reduce the learning rate by 85% every 42 steps //currentLearningRate = initialLearningRate * Math.pow(0.85, Math.floor(step/42)); //model.optimizer.learningRate = currentLearningRate; const batchData = generateData(batchSize); const dataTensor = color2tensor(batchData[0]); const labelTensor = color2tensor(batchData[1]); const history = await model.fit(dataTensor, labelTensor, {batchSize: batchSize, epochs: epochs }); cost = history.history.loss[0]; tf.dispose(dataTensor, labelTensor); return step++; }

The labels are generated using an implementation of a complementary color algorithm based on an answer by Edd in Stack Overflow. The full code is rather long and is available at our application repository in github . The computeComplementaryColor function converts an RGB color input to HSL , compute the complementary color in this color space and converts the result back to RGB.

At each training round the model runs for the configured number of epochs with the same batch data. After a user defined number of training rounds, the model is called again to predict a set of test colors and the results can be visually compared, on the fly, with the reference RGB labels generated by Edd's algorithm.

The output of each training round is displayed on a color wheel using D3.js , as shown in the mock-up below.

Start Reset Learning Rate Batch Size Epochs Render interval Step Limit Cost Target
Starts/stops the network training
Resets the network, looses all training but keeps parameter values
Learning Rate controls how much the weights of the network are adjusted
Batch size is the amount of data sent to the network per training round
Epochs is the amount of times the network trains with the same batch
Number of network training runs before displaying the learned colors
Number of learning rounds before stopping
Stops learning if cost is smaller than Cost Target

244, 67, 54233, 30, 99156, 39, 176103, 58, 18363, 81, 18133, 150, 24376, 175, 80139, 195, 74205, 220, 57255, 235, 59255, 193, 7255, 152, 0255, 87, 34121, 85, 72158, 158, 15896, 125, 1390, 0, 054, 231, 24430, 233, 16459, 176, 39138, 183, 58181, 163, 63243, 126, 33175, 76, 171130, 74, 19572, 57, 22059, 79, 2557, 69, 2550, 103, 25534, 202, 25572, 108, 121158, 158, 158139, 110, 960, 0, 0Original colors(RGB)Computedcomplementarycolors (RGB)Predicted colors(RGB)Latest 200rendered colorpredictionsColor referenceStep 100Cost 0.02244873Time 0.245318, 161, 1898, 188, 14574, 204, 68135, 186, 60177, 165, 66237, 130, 46164, 52, 160122, 64, 18077, 68, 20657, 71, 22239, 79, 23235, 102, 22523, 148, 20284, 127, 157134, 133, 122144, 129, 11283, 125, 140

At the very bottom of the graphics there is 5 colors horizontal bar, showing the last 200 model predictions, and a reference color for comparison. As the model converges the right side of each horizontal bar should be as close as possible to the corresponding reference color.

Neural network architecture

The neural network has an input layer for the RGB colors, 3 hidden fully connected layers with 64, 32 and 16 relu units, respectively, and the output layer, for the model predictions. The code below shows how the neural network was created.

function modelInit() { //Add input layer // First layer must have an input shape defined. model.add(tf.layers.dense({units: 3, inputShape: [3]})); // Afterwards, TF.js does automatic shape inference. model.add(tf.layers.dense({units: 64, activation: 'relu' })); // Afterwards, TF.js does automatic shape inference. model.add(tf.layers.dense({units: 32, activation: 'relu' })); // Afterwards, TF.js does automatic shape inference. model.add(tf.layers.dense({units: 16, activation: 'relu' })); // Afterwards, TF.js does automatic shape inference. model.add(tf.layers.dense({units: 3 })); }

Optimizer, metric and loss

We used a momentum optimizer with mean squared error loss and accuracy metric. The loss function was implemented manually, as seen on the code below, instead of using the library provided Mean Squared Error loss function. This was a design choice that allowed us to change/tweak the loss function at will during development time. On a future version we plan to provide the user with a selection of loss functions, as a configuration option.

function loss(predictions, labels) { // Subtract our labels (actual values) from predictions, square the results, // and take the mean. Inputs are tensors. return tf.tidy(() => { const meanSquareError = predictions.sub(labels).square().mean(); return meanSquareError; }); }

Common Misconception

We found out that a common difficulty in understanding machine learning applications arise from a simple misconception about the very framework of a machine learning application. Most people look at the field through the lenses of a traditional application, where data is fed to an algorithm that produces an answer.

To understand machine learning applications, one must first understand how they come to life: a LARGE amount of data is fed to a training algorithm and it produces, as output, an instance of a prediction algorithm trained with the input data that was provided to the machine learning algorithm. This output can then take data as input and produce predictions in accordance with the training process. There are NO guarantees that the predictions will be fully correct but they will, hopefully, be as good as possible.

An analogy for the layperson

Suppose that you are given a card with a color at random and you must give back a card with a color too, it can be the same color or a different one. In the beginning, as you have no training, the best you can do is to give back a card at random. Being a deterministic algorithm, every time someone gives you a card with a color X, you will produce as a response, the same Y color. So if I give you a red card and you give me back a green card, next time I give you a red card you will give me back a green card again. At this juncture you will produce bogus answers, but answers you will produce!

Now suppose that you are given set of n card pairs, with the correct input/output pairings. You can then use these pairs to correct your card selections. But there is a catch, whenever you correct the predictions for one card (for example, blue is the correct answer to red), all the other predictions are affected in a way, they change a little bit. What you try to do then, is to minimize the error of the color pairs you got as input.

The idea is that, after this training and adjusting procedure, you may be able to make good guesses on color pairs you saw, and, hopefully, you will be able to make good guesses on the colors you never saw!

Now, you can adjust the color predictions in small baby steps, changing the output colors just a little bit (remember that when you change the answer for one color, all the other answers are slightly affected), or you can adjust the colors in big steps, with a big effect on your other predictions. Or you can do something in between these two extremes.

Every time you get a color pair list, you adjust the color predictions again. When will you stop this training process? Maybe you will stop after training for 1.000 rounds, or when your data has finished, or you will stop when the error in all the colors you saw is smaller than a certain value (meaning that you’re doing good predictions on this data). It is up to you.

If you are still following us, then you, likely, understood what were you doing with the color cards. We can now throw in a bit of nomenclature:

Of course there are other important concepts in neural network training, but the five above are the ones you will see at work in the interactive application.

The demo

The application we present is based in one of the tutorials for the late deeplearn.js library (now tensorflow.js, a javascript inception of TensorFlow and a spinoff of deeplearn.js). That tutorial, called Complementary Color Prediction, was developed by Chi Zeng, from Google.

We developed a D3.js visual interface for the CCP tutorial in typescript and later, when deeplearn.js moved to tensorflow.js, we rewrote the whole code in javascript and added a bit of interactivity on top of it.

244, 67, 54233, 30, 99156, 39, 176103, 58, 18363, 81, 18133, 150, 24376, 175, 80139, 195, 74205, 220, 57255, 235, 59255, 193, 7255, 152, 0255, 87, 34121, 85, 72158, 158, 15896, 125, 1390, 0, 054, 231, 24430, 233, 16459, 176, 39138, 183, 58181, 163, 63243, 126, 33175, 76, 171130, 74, 19572, 57, 22059, 79, 2557, 69, 2550, 103, 25534, 202, 25572, 108, 121158, 158, 158139, 110, 960, 0, 0200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200200, 200, 200Original colors(RGB)Computedcomplementarycolors (RGB)Predicted colors(RGB)Latest 200rendered colorpredictionsColor reference

In the figure above we show the starting stage of the demo. The inner ring of the display is a set of colors that we will use to predict its complementary colors . In the middle ring we show the corresponding complementary colors, for reference and control. In the outer ring we display the model predicted complementary colors, for each color in the original color set (the inner ring). In the beginning the outer ring is gray, as no color was predicted yet. When running, the outer ring will be updated with the model predicted colors, and the user will be able to compare the predictions with the reference complementary colors in the middle ring.

It is possible to play with the demo using the controls below:

Start Reset Learning Rate Batch Size Epochs Render interval Step Limit Cost Target
Starts/stops the network training
Resets the network, looses all training but keeps parameter values
Learning Rate controls how much the weights of the network are adjusted
Batch size is the amount of data sent to the network per training round
Epochs is the amount of times the network trains with the same batch
Number of network training runs before displaying the learned colors
Number of learning rounds before stopping
Stops learning if cost is smaller than Cost Target

The five sliders on the right side let the user adjust the values for:

Start Reset Learning Rate Batch Size Epochs Render interval Step Limit Cost Target
Starts/stops the network training
Resets the network, looses all training but keeps parameter values
Learning Rate controls how much the weights of the network are adjusted
Batch size is the amount of data sent to the network per training round
Epochs is the amount of times the network trains with the same batch
Number of network training runs before displaying the learned colors
Number of learning rounds before stopping
Stops learning if cost is smaller than Cost Target

In the beginning the user can adjust any of the five sliders, choosing whatever combination he/she like. The user has to click on the start button to start the training. The colors on the outer ring will change, as the training goes on. Each new update shows the predictions of the neural network algorithm for the complementary colors at that stage of the training. The use can compare the results with the colors in the middle ring. When the training is running, the Start button becomes a red Stop button, as shown in the figure below. The user can click on the stop button to halt the model training.

Stop Reset Learning Rate Batch Size Epochs Render interval Step Limit Cost Target
Starts/stops the network training
Resets the network, looses all training but keeps parameter values
Learning Rate controls how much the weights of the network are adjusted
Batch size is the amount of data sent to the network per training round
Epochs is the amount of times the network trains with the same batch
Number of network training runs before displaying the learned colors
Number of learning rounds before stopping
Stops learning if cost is smaller than Cost Target

After the training starts, the user won't be able to change neither the Learning Rate nor the Batch Size, so these parameter must be wisely chosen. The training may stop for three reasons:

  1. The user clicked on Stop.
  2. The Step Limit was reached.
  3. The Cost Target was reached.

If the training was manually halted, then the user can restart it by clicking on start again. If the Step Limit was reached, the user can increase the step limit, if possible, and restart the training. If the Cost Target was reached, the user can decrease the cost target, if possible, and restart the training. Whenever the training is halted, the user can Reset it by clicking on the Reset button (notice that the button is disabled when the training is running). A reset will clear all the training data (meaning that the model will "forget" the predictions made so far), but will keep the values chosen for the training session. This way the user can incrementally fine tune the training and explore the impact of each parameter change.

Notice that the initial parameters are not optimal, so the user must play with them a little bit before finding out a good workable combination of parameter input. The user is encouraged to try different combinations, play with them and, with luck, acquire some insights on how these parameters influence the training and the model predictions. One important thing to notice about machine learning training is that there is a random effect going on. The user will see it in action by running the model with the same parameters more then once. He/She won't get the same results twice! The demo the be found at the end of the article.

Apendix

Acknowledgements

First and foremost we would like to thank the TensorFlowJS team, this is an amazing library that opens new doors to machine learning explorations on the browser. Tensorflow.js code base can be found here. A special thanks goes to Chi Zeng, whose code inspired this application.

Start Reset Learning Rate Batch Size Epochs Render interval Step Limit Cost Target
Starts/stops the network training
Resets the network, looses all training but keeps parameter values
Learning Rate controls how much the weights of the network are adjusted
Batch size is the amount of data sent to the network per training round
Epochs is the amount of times the network trains with the same batch
Number of network training runs before displaying the learned colors
Number of learning rounds before stopping
Stops learning if cost is smaller than Cost Target