Neural Networks – Some Basics

Que -1 How does a Human Brain Neural Network typically look like and how is it modelled in Artificial Neural Networks ?
Ans -1 The Brain consists of individual cells called Neurons. The typical neurons looks like below:-

A neuron takes inputs from Dendrites and gives out put through Axon terminals. The Nucleus based on the inputs received fires electrochemical pulses through Axon which are connected to dendrites of other neurons.

The point of connection between the Axons of one neuron and Dendrites of other neuron is called a synapse.

From each dendrite the neuron receives a signal x_i and the total net input signal to the neuron is a linear combination of these input signals x_i

Each synapse ( the connecting point between two neurons) either amplifies or dampens the signal received from the axon of other neurons by a factor w_i. The nucleus applies a function over the input signals and either produces an electrochemical signal and passes to the next neuron or produces no signal at all.

Model of neuron

The net input signal is linear combination of w_ix_i and bias which neuron applies :-
y = b + w₁x₁ + w₂x₂ + w₃x₃ + …………… + w_nx_n

The x₁,x₂,x₃ may be treated equivalent to features or independent variables in regression.
The y may be treated equivalent to the dependent variable y in regression

Que -2 What are activation functions why are they required ?
Ans -2 In the above equation if you see f_(net) results in a plane summation of the bias and weight matrix. This is a linear relationship. For linear transformation we may not require an activation function altogether like the case of simple linear regression.

But, as you may have studied in Advanced Linear Regression it is not necessary that the output y is linearly related to input. So when an ANN will try to model the relationship between the input features that are not linearly related to the output function, in order to introduce this non –linearity in the equation an activation function like ReLU or Swish( details afterwards) may be chosen which gives an output which applies a complex operation on the sum of weights and bias.

The activation function is also dependent on the type of problem. For eg. A simple linear regression may require an activation function which does linear summation and gives the resultant output as a simple sum value, whereas a logistic regression with binary classifier may require the activation function to give output as a zero or one representing whether the input does not belong to a particular class or whether it belongs to the particular class resp.

A simple binary classifier activation function

A more smoothened Sigmoid function representing 0 or 1

Que -3 Anatomy of Artificial Neural Networks

Ans -3
Since we need to do parallel processing instead of a single neuron we take into consideration a whole layer of neurons.
There are three type of layers:-
Input Layer – > which specifies the inputs given
Hidden layers -> which apply various transformations on the inputs
Out put layer -> which produces the final output

Que -4 When you train an artificial neural network, how does it store the information gained during training

Ans -4 In the Ans- 1 we saw that the Neural Network had synapses or connections that applied what is called weight on the input signal. During training inputs are provided and some random weights are assigned to each connection ( or synapse) which applies to the input. The neuron then produces an output . The error in prediction is determined based on what was the actual output and what was the expected output. Based on the error a feedback is sent to the layer of neurons to adjust the weights before next prediction. This feedback is called backward propagation.

Based on feedback the weights are changed in a certain way. Then again fresh input is supplied and error is determined. If the error has increased from last time a different strategy is applied to tune weights. If the error has decreased the same strategy is applied to further change the weights. When back propagation happens not just the errors but also the gradient is supplied back to neurons.

The combination of weights and biases in various layers represents the training knowledge stored by the Artificial Neural Network. Just like a computer works on binary language to represent knowledge, the ANN works on combination of weights and biases to store knowledge or information.

Que -5 If the building of a neural network starts from random weights, random no of neurons and random strategy, do I have to everytime go through the same process to build a neural network ?

Ans -5 Researchers have, over the years, worked on the principle of artificial neural networks and they have come up with certain architectures or combination of layers that are useful in determining solution to particular class of problems. For Eg, If you face a problem in computer Vision you would use a CNN or Convolutional Neural Network to find the solution to the problem. The layers in this architecture are more or less fixed and you are supposed to just tune some hyper parameters in order to get the right output.