Neural Network Backpropagation

Backpropagation: The feedforward backpropagation network is a neural model that does not have feedback connections, it uses the errors during training, To minimize the squared error between the output and target values. Errors in the output determine measures of hidden layer output errors, which are used as a basis to adjust the connection weights between the input and hidden layers. Adjusting the two sets of weights between the layers and recalculating the outputs is an iterative process that is carried on until the errors fall below a tolerance level. A learning rate parameters scale the adjustments to weights. A momentum parameter can be used in scaling the adjustments from a previous iteration and adding the adjustments in the current iteration. The backpropagation network creates a map form input vectors to output vectors. a set of input and output vectors are use to train the network first. Once training is completed, the weights are set and the network can be used to find for new inputs. The dimension of the input vector determines the number of neurons in the input layer, and the number of neurons in the output layer is determined by the dimension of the outputs. The architecture of the backpropagation network, is multilayer feedforward network one layer of input units, one hidden units and one of outputs units, The input layer is connected to the hidden layer and the output layer is connected to the output layer by means of interconnection weights.

Training: The backpropagation network undergoes supervised training, with a finite number of inputs and a desired output pattern. An input pattern is presented at the input layer. The neurons here pass the pattern activations to the next layer neurons, which are in a hidden layer. The outputs of the hidden layer neurons are obtained by using a bias, and also a threshold function with the activations determined by the weights and the inputs. These hidden layer outputs become inputs to the output neurons, which process the inputs using an optional bias and a threshold function. The final output of the network is determined by the activations from the output layer. The output computed and input pattern are compared, a function of this error for each component of the pattern is determined, and adjustment to weights of connections between the hidden and output layer is computed. A similar computation, still based on the error in the output, is made for the connection weights the input and hidden layers. The procedure is repeated with each pattern pair assigned for training the network. Each pass through all the training patterns is called a cycle or an EPOCH. This process is repeated until the error is within a prescribed tolerance.

----------------------------------------------------------------------------------

Adjustment of Weights of Connections from a Neuron in the Hidden Layer

----------------------------------------------------------------------------------

input pattern: 	[ 1.1, 2.4, 3.2, 5.1, 3.9 ]
target output: 	[ 0.52, 0.25, 0.75, 0.97 ]

weights given for the second hidden layer neuron by the vector:

[ –0.33, 0.07, –0.45, 0.13, 0.37 ]

The activation will be the quantity:

(−0.33 * 1.1) + (0.07 * 2.4) + (−0.45 * 3.2) + (0.13 * 5.1) + (0.37 * 3.9) = 0.471

Add to this an optional bias of, say, 0.679, to give 1.15. If we use the sigmoid function given by:

1 / ( 1+ exp(−x)), with x = 1.15, we get the output of this hidden layer neuron as:	0.7595

----------------------------------------------------------------------------------

Computed output pattern also.

actual: 	[ 0.61, 0.41, 0.57, 0.53 ]
desired:	[ 0.52, 0.25, 0.75, 0.97 ] 

component−wise differences are given in the vector, desired − actual:

[ −0.09, −0.16, 0.18, 0.44 ]

We use these to form another vector where each component is a product of the error component, corresponding computed pattern component, and the complement of the latter "actual" with respect to 1.

error-vector:

[ (1 - 0.61), (1 - 0.41), (1 - 0.57), (1 - 0.53) ] =>	
[ 0.39, 0.59, 0.43, 0.47 ] =>	
[ (0.61 * 0.39 * −0.09), (0.41 * 0.59 * −0.16), (0.57 * 0.43 * 0.18), (0.53 * 0.47 * 0.44) ] => 

output:

[ -0.02, -0.04, 0.04, 0.11 ]

desired–actual vector, which is the error vector multiplied by the actual output vector, gives you a value of error reflected back at the output of the hidden layer. This is scaled by a value of (1 − output vector), which is the first derivative of the output activation function for numerical stability The errors needs to be carried further. We need now the weights on the connections between the second neuron in the hidden layer that we are concentrating on, and the different output neurons. the weights are given by:

[ 0.85, 0.62, –0.10, 0.21 ]

The error of the second neuron in the hidden layer is calculated as below, using its output.

error:

0.7595 * (1 − 0.7595) * ((0.85 * −0.02) + (0.62 * −0.04) + ( −0.10 * 0.04) + (0.21 * 0.11)) = −0.0041

Again, here we multiply the error (−0.02) from the output of the current layer, by the output value (0.7595) and the value (1 − 0.7595). We use the weights on the connections between neurons to work backwards through the network. Next, we need the learning rate parameter for this layer;

let us set it as:    0.2
learning rate:	     0.7595 * 0.2 = 0.1519

We multiply this by each component of the output of the second neuron in the hidden layer.

vector:

[ –0.02, –0.04, 0.04, 0.11] 
[ –0.02, –0.04, 0.04, 0.11] 
[ (–0.003 * 0.1519), (–0.006 * 0.1519), (0.006 * 0.1519), (0.017 * 0.1519)]

This result is a vector that gives the adjustments to the weights on the connections that go from the second neuron in the hidden layer to the output neurons.

[ –0.003, –0.006, 0.006, 0.017 ]
[ (0.85 + –0.003), (0.62 + –0.006), (–0.10 + 0.006), (0.21 + 0.017) ]

After these adjustments are added, the weights to be used in the next cycle on the connections between the second neuron in the hidden layer and the output neurons become:

[ 0.847, 0.614, –0.094, 0.227 ]

----------------------------------------------------------------------------------

Adjustment of Weights of Connections from a Neuron in the Input Layer.

----------------------------------------------------------------------------------

To determine the adjustments for the weights on connections between the input and hidden layers, we need the errors determined for the outputs of hidden layer neurons, a learning rate parameter, and the activations of the input neurons, which are just the input values for the input layer.

learning rate: 
0.15	[ (0.15 * 1.1 * −0.0041), 
(0.15 * 2.4 * −0.0041), 
(0.15 * 3.2 * −0.0041), 
(0.15 * 5.1 * −0.0041), 
(0.15 * 3.9 * −0.0041) ]
[ -0.001, -0.001, –0.002, -0.003, -0.002 ]
[ (–0.33 + -0.001), (0.07 + -0.001), 
(–0.45 + –0.002), (0.13 + -0.003), (0.37 + -0.002) ]

weight adjustments:		
[ -0.331, 0.069, -0.452, 0.127, 0.368 ]

----------------------------------------------------------------------------------

Adjustments to Threshold Values or Biase.

---------------------------------------------------------------------------------

Adjustment for the threshold value of a neuron in the output layer is obtained by multiplying the calculated error (not just the difference) in the output at the output neuron and the learning rate parameter used in the adjustment calculation for weights at this layer. Adjustments to the threshold values of the four output neurons are given by the vector:

[ error vector	] * [ learning rate ]	[ –0.02, –0.04, 0.04, 0.11 ] * [ 0.2 ] =>		
[ –0.004, –0.008, 0.008, 0.022 ]

These adjustments are added to the current levels of threshold values at the output neurons. The adjustment to the threshold value of a neuron in the hidden layer is obtained similarly by multiplying the learning rate with the computed error in the output of the hidden layer neuron. Therefore, for the second neuron in the hidden layer, the adjustment to its threshold value is calculated as 0.15 * –0.0041, which is –0.0006. learning rate * computed error:

0.15 * –0.0041 = –0.0006		

Add this to the current threshold value of 0.679 to get 0.6784, which is to be used for this neuron in the next training pattern for the neural network.

0.679 + –0.0006 = 0.6784

Image Processing:

It can be said that the image processing is the use of techniques or tools with which they can be analyzing to identify shadows, colors and relationships that can not be perceived by the human eye or improvements in image processing is used to solve identification problems, for example in forensic medicine, facial tracking and recognition, in the creation of meteorological maps from satellite images. These are bitmap-formatted images that are captured with digital cameras.

Face Detection:

The main function of a face detector is to look for the existence of a face in an image of certain size, and in the case that it is find the the position. Locating and tracking human faces is basic for face recognition and/or facial expressions analysis. In order to locate a human face, the system needs to capture an image using a camera and process the image, search the image for important features and then use these features to determine the location of the face.For face detection there are various algorithms including skin color based algorithms. Color is an important feature of human faces. Using skin color as a feature for tracking a face has several advantages. The processing of color is much faster than looking for other facial features.Under certain lighting conditions, color is invariant. This property makes motion estimation much easier because only a translation model is needed for motion estimation. However, color is not a physical phenomenon; it is a perceptual phenomenon that is related to the spectral characteristics of visible electromagnetic wavelengths which are perceived by the retina. Tracking human faces using color as a feature has several problems like the color representation of a face obtained by a camera is influenced by many factors like ambient light, object movement, etc. different cameras produce significantly different color values even for the same person under the same lighting conditions and skin color differs from person to person. In order to use color as a feature for face tracking, we have to solve these problems. A disadvantage of the color cue is its sensitivity to illumination color changes and, especially in the case of RGB, sensitivity to illumination intensity. One way to increase tolerance toward intensity changes in images is to transform the RGB image into a color space whose intensity and chromaticity are separate and use only chromaticity part for detection. In this project I have presented a comparative study of three well known skin color face localization/detection algorithms and have given a new algorithm based on skin color classification in RGB and YCbCr color models.

RGB MODEL:

The RGB color space consists of the three additive primaries: red, green and blue. Spectral components of these colors combine to produce a resultant color. Black is at the origin "0 absence of color". White is the opposite "255 all the colors". The gray scale follows the line from black to white. In a 24-bit color graphics system with 8 bits per color channel, red is (255, 0, 0). The RGB model simplifies the design of computer graphics systems but is not ideal for all applications. The red, green and blue color components are highly correlated. This makes it difficult to execute some image processing algorithms.

MODEL:

YCbCr color space has been defined in response to increasing demands for digital algorithms in handling video information, and has since become a widely used model in a digital video. It belongs to the family of television transmission color spaces. The family includes others such as YUV and YIQ. YCbCr is a digital color system, while YUV and YIQ are analog spaces for the respective PAL and NTSC systems. These color spaces separate RGB into luminance and chrominance information and are useful in compression applications. The Recommendation 601 specifies 8 bit (0 to 255) coding of YCbCr, whereby the luminance component Y has an excursion of 219 and an offset of +16. This coding places black at code 16 and white at code 235. In doing so, it reserves the extremes of the range for signal processing foot room and headroom. On the other hand, the chrominance components Cb and Cr have excursions of +112 and offset of +128, producing a range from 16 to 240 inclusively

Face detection based on skin color using RGB color space:

Crowley and Coutaz said one of the simplest algorithms for detecting skin pixels is to use skin color algorithm. The perceived human color varies as a function of the relative direction to the illumination. The pixels for skin region can be detected using a normalized color histogram, and can be further normalized for changes in intensity on dividing by luminance. And thus converted an [R, G, B] vector is converted into an [r, g] vector of normalized color which provides a fast means of skin detection. This gives the skin color region which localizes face. As in, the output is a face detected image which is from the skin region. This algorithm fails when there are some more skin region like legs, arms, etc.

Face detection based on skin color using YCbCr color space:

We have implemented a skin color classification algorithm with color statistics gathered from YCbCr color space. Studies have found that pixels belonging to skin region exhibit similar Cb and Cr values. Furthermore, it has been shown that skin color model based on the Cb and Cr values can provide good coverage of different human races. The thresholds be chosen as [Cr1, Cr2] and [Cb1, Cb2], a pixel is classified to have skin tone if the values [Cr, Cb] fall within the thresholds The skin color distribution gives the face portion in the color image. This algorithm is also having the constraint that the image should be having face as the skin region.

Eye detection Illumination-based Method:

From a segment face skin image, two eye maps a created, one for chrominance component. The chrominance eye map is based on the fact that high Cb and low Cr values can usually be found around the eyes. The values of Cb and Cr are normalized to a range of (0-255), the this eye map can be calculated from:

EyeMapC = 1/3 { (Cr)² + (Cb)² + (Cb/Cr) } 1/3 

is a scalin factor to ensure that the eye map stays within the range. To construct the luminance component, morphological operators (erosion and dilation) are designed to emphasise the brighter and dark pixels in this component around the eye regions. So the eye map from the luminance component is calculated from:

EyeMapL = Dilation Y(x, y) / Erosion Y(x, y)

Mouth detection:

To detect mouth region, the mouth region contains stronger red component and weaker a blue component in facial regions (Cr > Cb), in the Cr/Cb feature, but it has a high response in Cr2. The Mouth Map is constructed as.

MouthMap Cr² { Cr² -nCr/Cb } where n = 0.95 * avrg(Cr²) /avrg(Cr/Cb)Cr2 and Cr/Cb are normalized to the range of (0 - 255), morphological operators (erosion and dilation) are apply her two. BPNN Backpropagation Neural Network with heuristic rule: This model for skin detection uses a BP ANN and heuristic rule based on YCbCr. The goal of the algorithm is to increase the classification reliability of of skin detection with different lighting conditions. In this project the BP ANN was trained using a resized image of 60x60 (a dataset of 3600 pixels) of various skin tones per image.the pixels of image are converted from RGB to YCbCr color space, and the YCbCr are used to train a three layer neural network.The pixels of this image are clasified into skin and non-skin pixels depending on the output of the neural network an the heuristic rule, which classify a pixel as skin if the following condition is true.

(132 < Cr > 173) ^ (76 < Cb > 126)