| Neural Network Toolbox™ | |
| Provide feedback about this page |
Simple Neuron
A neuron with a single scalar input and no bias appears on the left below.
The scalar input p is transmitted through a connection that multiplies its strength by the scalar weight w to form the product wp, again a scalar. Here the weighted input wp is the only argument of the transfer function f, which produces the scalar output a. The neuron on the right has a scalar bias, b. You can view the bias as simply being added to the product wp as shown by the summing junction or as shifting the function f to the left by an amount b. The bias is much like a weight, except that it has a constant input of 1.
The transfer function net input n, again a scalar, is the sum of the weighted input wp and the bias b. This sum is the argument of the transfer function f. (Radial Basis Networks, discusses a different way to form the net input n.) Here f is a transfer function, typically a step function or a sigmoid function, that takes the argument n and produces the output a. Examples of various transfer functions are in Transfer Functions. Note that w and b are both adjustable scalar parameters of the neuron. The central idea of neural networks is that such parameters can be adjusted so that the network exhibits some desired or interesting behavior. Thus, you can train the network to do a particular job by adjusting the weight or bias parameters, or perhaps the network itself will adjust these parameters to achieve some desired end.
All the neurons in the Neural Network Toolbox™ software have provision for a bias, and a bias is used in many of the examples and is assumed in most of this toolbox. However, you can omit a bias in a neuron if you want.
As previously noted, the bias b is an adjustable (scalar) parameter of the neuron. It is not an input. However, the constant 1 that drives the bias is an input and must be treated as such when you consider the linear dependence of input vectors in Linear Filters.
Transfer Functions
Many transfer functions are included the Neural Network Toolbox software. A complete list of them can be found in the reference pages. Three of the most commonly used functions are shown below.
The hard-limit transfer function shown above limits the output of the neuron to either 0, if the net input argument n is less than 0, or 1, if n is greater than or equal to 0. This function is used in Perceptrons, to create neurons that make classification decisions.
The toolbox has a function, hardlim, to realize the mathematical hard-limit transfer function shown above. Try the following code:
It produces a plot of the function hardlim over the range -5 to +5.
All the mathematical transfer functions in the toolbox can be realized with a function having the same name.
The linear transfer function is shown below.
Neurons of this type are used as linear approximators in Linear Filters.
The sigmoid transfer function shown below takes the input, which can have any value between plus and minus infinity, and squashes the output into the range 0 to 1.
This transfer function is commonly used in backpropagation networks, in part because it is differentiable.
The symbol in the square to the right of each transfer function graph shown above represents the associated transfer function. These icons replace the general f in the boxes of network diagrams to show the particular transfer function being used.
For a complete listing of transfer functions and their icons, see the reference pages. You can also specify your own transfer functions.
You can experiment with a simple neuron and various transfer functions by running the demonstration program nnd2n1.
Neuron with Vector Input
A neuron with a single R-element input vector is shown below. Here the individual element inputs
and the weighted values are fed to the summing junction. Their sum is simply Wp, the dot product of the (single row) matrix W and the vector p.
The neuron has a bias b, which is summed with the weighted inputs to form the net input n. This sum, n, is the argument of the transfer function f.
This expression can, of course, be written in MATLAB® code as
However, you will seldom be writing code at this level, for such code is already built into functions to define and simulate entire networks.
Abbreviated Notation
The figure of a single neuron shown above contains a lot of detail. When you consider networks with many neurons, and perhaps layers of many neurons, there is so much detail that the main thoughts tend to be lost. Thus, the authors have devised an abbreviated notation for an individual neuron. This notation, which is used later in circuits of multiple neurons, is shown.
Here the input vector p is represented by the solid dark vertical bar at the left. The dimensions of p are shown below the symbol p in the figure as Rx1. (Note that a capital letter, such as R in the previous sentence, is used when referring to the size of a vector.) Thus, p is a vector of R input elements. These inputs postmultiply the single-row, R-column matrix W. As before, a constant 1 enters the neuron as an input and is multiplied by a scalar bias b. The net input to the transfer function f is n, the sum of the bias b and the product Wp. This sum is passed to the transfer function f to get the neuron's output a, which in this case is a scalar. Note that if there were more than one neuron, the network output would be a vector.
A layer of a network is defined in the previous figure. A layer includes the combination of the weights, the multiplication and summing operation (here realized as a vector product Wp), the bias b, and the transfer function f. The array of inputs, vector p, is not included in or called a layer.
Each time this abbreviated network notation is used, the sizes of the matrices are shown just below their matrix variable names. This notation will allow you to understand the architectures and follow the matrix mathematics associated with them.
As discussed in Transfer Functions, when a specific transfer function is to be used in a figure, the symbol for that transfer function replaces the f shown above. Here are some examples.
You can experiment with a two-element neuron by running the demonstration program nnd2n2.
| Provide feedback about this page |
| Neuron Model and Network Architectures | Network Architectures | |
| © 1984-2008- The MathWorks, Inc. - Site Help - Patents - Trademarks - Privacy Policy - Preventing Piracy - RSS |
