Model Representation of Neural Network

Anatomical neurons are cells that are present in the brain in millions. A neuron has a cell body, a number of input wires, called dendrites and an output wire called axon.

In a simplistic way a neuron is a computational unit that receive some input via dendrites, does some computation and then outputs something via the axon to other neurons in the brain.

A neuron implemented on the computer has a very simple model that mimics the architecture of an anatomical neuron. We’re a going to model a neuron as just a logistic unit. The yellow node represents the body of the neuron, which is fed input through its dendrites, and produces an output $h_\theta(x)$ that is produced by the neuron body, though its activation function and transported forward by the neuron axon. Where $h_\theta(x)=\frac{1}{1+e^{-\theta^Tx}}$

A simpler representation is sometimes used to depict a neural network

where $x_0$ (sometimes called the bias unit) is usually omitted in favor of another representation:

and the parameters, or weights ($w$) are accompanied by the bias $b$.

Until now we represented single neurons; a neural network is a group of different neurons connected together. The input nodes are grouped in what is called the input layer ($x$), which is always the first layer of the neural network. The final layer is called the output layer, since it computes the final value of our hypothesis. And all layers in between the input and the output layers are called hidden layers. They are called hidden layers because we can’t observes the values computed by these nodes.

The computational entities in a neural networks are:

• $a_i^{[l]}$ activation neuron/unit $i$ in layer $l$
• $W^{[l]}$ matrix of weights controlling the function mapping from layer $l$ to layer $l+1$
• $b^{[l]}$ the bias vectors

And the computation in the network

Forward propagation

The flow of the computation in the network in Figure 10 from input (left) to prediction (right), called forward propagation, is just like that in logistic regression but a lot more times. In fact, each unit in layer $l$ is densely connected (namely is connected to all units in layer $l+1$) and we will have to compute a logistic regression for each connection.

So, for example, the computations that we will have to execute from the input layer to the first layer will be:

That is to say that we compute our hidden units in the first layer as a $3\times 3$ matrix of parameters $W^{[l]}_{ij}$, weighting the connection from unit $j$ in layer $l-1$ to unit $i$ in layer $l$.

Vectorization

First step of vectorization

Let’s see vectorization for $\eqref{eq:neuralnet}$: this process can be then applied to any other layer. In $\eqref{eq:neuralnet}$ we have the equations $\eqref{eq:l1unit1vect}$ are the operations required to calculate $a^{[1]}_1$ from input $x_1, x_2, x_3$ (Figure 11, panel A).

Similarly, $\eqref{eq:l1unit2vect}$ are the operations required to calculate $a^{[1]}_2$ from input $x_1, x_2, x_3$ (Figure 11, panel B)

Finally, $\eqref{eq:l1unit3vect}$ are the operations required to calculate $a^{[1]}_3$ from input $x_1, x_2, x_3$ (Figure 11, panel C)

Second step of vectorization

Given the set of equation that describe the activtion of the first layer from the input layer, let’s see how to calculate $z^{[1]}$ as a vector:

Let’s first take the vectors $w^{[1]T}_i$ and stack them into a matrix $W^{[1]}$.

$w^{[1]}_i$ are column vectors, so their transpose are row vectors that are stacked vertically.

And now we can calculate $a^{[1]}$

So, summarizing what we have written above and extending it to the second layer we have

or more generally

Third step of vetorization across multiple examples

The process in $\eqref{eq:vectanneqs}$ must be repeated for each training example $x^{(m)}$ and will produce $m$ outputs $a^{(m)[2]} = \hat{y}^{(m)}$

In a non-vectorized implementation you would have something along the lines of:

for i in len(examples):
    z1[i] = w1 @ x[i] + b[i]
    a1[i] = sigmoid(z1[i])
    z2[i] = w2 @ x([i] + b[i]
    a2[i] = sigmoid(z1[i])

Given our vector of training examples:

So this means that our vectorized implementation becomes

with $Z^{[1]}$ and $A^{[1]}$ represent the $z$-values and $a$-values of the first layer of the neural network:

where $m$ is the number of training examples and $n^{[1]}$ is the number of nodes (hidden units) in the first layer of the neural networks.

Comments