Simple Highway Network example from scratch.

This is a very basic Highway Network example comparing its performances with a Feedforward Neural Network.

This program implements the highway network described in [Srivastava 2015] . It compares the performances of a highway network with that of a feedforward neural network on a similar problem which simply consists of distinguishing between 3 data points. It confirms the fact that highway networks continue to have good performances even for higher numbers of hidden layers, as shown in the following image.

Every step of the gradient calculation is provided in the grad_highway_layer. function