![\"\"](\"http:\/\/blog.stratio.com\/wp-content\/uploads\/2017\/09\/DeepLearning1.jpg\")
Figure 1: Deep Learning generated image. <\/span><\/em><\/span><\/h6>\nIn this first filmed session, we start by defining neural networks as a machine learning model <\/span>inspired by the human brain, which arise as a way to create highly non-linear features in an intelligent way. We then show how predictions are computed via forward propagation. After that, we define the error function and the gradient descent algorithm to minimize it, which for neural networks makes use of an additional algorithm called backpropagation. Note that training a neural network requires a lot of training data in order to obtain a good approximation of the gradient at each point of the parameter space (and because there are a lot of parameters in a network, it is a high-dimensional space). The backpropagation algorithm leads to a much more efficient computation of the gradient compared to a direct method.<\/span><\/p>\n
\nNeural Networks for supervised learning<\/b><\/h2>\n
The remainder of this post focuses on how to use a neural network for supervised learning problems. We start with a dataset with D input features composed of examples (rows) which we treat as column vectors <\/span>x <\/b>= (x<\/span>1<\/span>, x<\/span>2<\/span>, \u2026, x<\/span>D<\/span>)<\/span>T<\/span> that we will use to teach our network. Note that the target column which indicates what the correct output should be for each example (needed by the network to learn) is not included in this vector. The motivation of neural networks is the need for an inherently non-linear model which is able, for instance, to perform well in classification problems where the decision boundary between the classes is non linear. Although linear models fed with non-linear transformation of the features (like powers or products between features) also exhibit non-linear boundaries, this rapidly becomes unfeasible when the number of features of the data grow.<\/span><\/p>\n![\"\"](\"http:\/\/blog.stratio.com\/wp-content\/uploads\/2017\/09\/Interconnected_human_neurons3.jpg\")
\n Figure 2: two interconnected human neurons <\/span><\/i><\/span><\/p>\nNeural networks are inspired by how the human brain works: it is composed of simple neurons, each carrying a simple computation with its inputs and transmitting its output to another neuron (Fig. 2). Similarly, neural networks consist of simple units called artificial neurons (Fig. 3(a)) which apply some non-linear function <\/span>g<\/span><\/i> (called <\/span>activation function<\/span><\/i>)<\/span> to a linear combination of its inputs. The coefficients <\/span>?<\/b> of the linear combination represent what the neuron has learned. The most commonly used activation function is the sigmoid.<\/span><\/p>\n![\"\"](\"http:\/\/blog.stratio.com\/wp-content\/uploads\/2017\/09\/artificial_neuron.jpg\")
Figure 3: an artificial neuron with three inputs (left) and the sigmoid activation function (right)<\/span><\/i><\/span><\/span><\/h6>\nA neural network is composed of multiple interconnected artificial neurons organized in layers, such that the outputs (a.k.a. \u201cactivations\u201d) of the neurons of one layer act as inputs for the neurons of the next layer. Therefore, what the neurons in the inner layers are using as <\/span>features<\/span><\/i> (i.e. the activations from previous layer) are indeed non-linear combinations of the original input features. Such combinations possibly represent higher-level concepts (let\u2019s call them <\/span>mysterious features<\/span><\/i>), instead of the raw features present in our data.<\/span><\/p>\nMultilayer perceptron and forward propagation to make predictions<\/b><\/h2>\n
If we number the neurons of the network, the output (predictions) of a fully connected network like the one displayed in Fig. 4 can be computed as shown in the image – this process is sometimes known as <\/span>forward propagation<\/span><\/i>. The notation and operations seem a bit ugly (a.k.a. \u201cmore technical\u201d), but feel free to skip the figure if you are not interested in the maths. It is nothing more than applying the equation of Fig. 3 repeatedly for each neuron in the hidden layer and in the output layer. In the figure we are collecting together the parameters of each layer in matrices B<\/span>(j)<\/span>. We call the set of all of these matrices <\/span>B<\/i><\/b> – that\u2019s why the output of the network is a function of the inputs named <\/span>h<\/span><\/i>B<\/i><\/b>.<\/span><\/p>\n![\"\"](\"http:\/\/blog.stratio.com\/wp-content\/uploads\/2017\/09\/fully_connected_network.jpg\")
<\/p>\n
Figure 4: a fully connected network (multilayer perceptron) with one hidden layer and three inputs for examples with D = 3 features.<\/span><\/i><\/span><\/h6>\nIn case we are facing a binary classification problem, the class is determined by re-coding the output to one of the classes, either setting a threshold on the output (binary classification) or having as many neurons in the output layer as classes in our problem (multiclass classification). The output class computed by the network corresponds to the index of the neuron of the output layer which yields the highest output value.<\/span><\/p>\nTraining algorithm: cost function and gradient descent via backpropagation<\/b><\/h2>\n
In any Machine Learning model, fitting a model means finding the best values of its parameters. The best model is one whose parameter values minimize the total error with respect to the actual outputs on our training data. Since the error depends on the parameters chosen, it is a function of the parameters, called the <\/span>cost function <\/span><\/i>J. The most common error measure is the MSE (Mean Squared Error): <\/span>![\"\"](\"http:\/\/blog.stratio.com\/wp-content\/uploads\/2017\/09\/formula_DeepLearning.png\")
where h<\/span>?0, ?1, \u2026, ?P<\/span> is the equation of the model (be it a neural network or any other machine learning model) depending on P+1 parameters, and <\/span>m<\/span><\/i> is the number of training examples. <\/span>
\n<\/span><\/p>\nIf the cost function is convex, it only has one global optimum, which is the global minimum. It can be found by computing the partial derivatives of J with respect to every variable (gradient vector ?J), and solving an equation system to find the point where the derivatives are all equal to 0 simultaneously (exact solution). Unfortunately, this can be difficult when the model equation is complicated (which yields a non-convex error function with many <\/span>local optima<\/span><\/i>), has many variables (large equation system) or is not derivable. That\u2019s why we often use some approximate, iterative optimization algorithm for this task, which does not guarantee the global minimum but generally performs well. Gradient descent is the most common, although there is a wide family of techniques which share the approximate, iterative nature (getting closer and closer to the solution at each time step <\/span>t<\/span><\/i>) and the need for evaluating the partial derivatives of J at different parameter points ?<\/span>(t)<\/span>.<\/span><\/p>\n![\"\"](\"http:\/\/blog.stratio.com\/wp-content\/uploads\/2017\/09\/plot_error_function2.jpg\")
<\/p>\n
Fig 5: a plot of an error function J for two parameters and the iterative process of reaching a minimum (called \u201cconvergence\u201d). Note the starting point heavily affects the solution obtained.<\/span><\/i><\/span><\/h6>\n
\nNeural Networks for supervised learning<\/b><\/h2>\n
The remainder of this post focuses on how to use a neural network for supervised learning problems. We start with a dataset with D input features composed of examples (rows) which we treat as column vectors <\/span>x <\/b>= (x<\/span>1<\/span>, x<\/span>2<\/span>, \u2026, x<\/span>D<\/span>)<\/span>T<\/span> that we will use to teach our network. Note that the target column which indicates what the correct output should be for each example (needed by the network to learn) is not included in this vector. The motivation of neural networks is the need for an inherently non-linear model which is able, for instance, to perform well in classification problems where the decision boundary between the classes is non linear. Although linear models fed with non-linear transformation of the features (like powers or products between features) also exhibit non-linear boundaries, this rapidly becomes unfeasible when the number of features of the data grow.<\/span><\/p>\n Neural networks are inspired by how the human brain works: it is composed of simple neurons, each carrying a simple computation with its inputs and transmitting its output to another neuron (Fig. 2). Similarly, neural networks consist of simple units called artificial neurons (Fig. 3(a)) which apply some non-linear function <\/span>g<\/span><\/i> (called <\/span>activation function<\/span><\/i>)<\/span> to a linear combination of its inputs. The coefficients <\/span>?<\/b> of the linear combination represent what the neuron has learned. The most commonly used activation function is the sigmoid.<\/span><\/p>\n A neural network is composed of multiple interconnected artificial neurons organized in layers, such that the outputs (a.k.a. \u201cactivations\u201d) of the neurons of one layer act as inputs for the neurons of the next layer. Therefore, what the neurons in the inner layers are using as <\/span>features<\/span><\/i> (i.e. the activations from previous layer) are indeed non-linear combinations of the original input features. Such combinations possibly represent higher-level concepts (let\u2019s call them <\/span>mysterious features<\/span><\/i>), instead of the raw features present in our data.<\/span><\/p>\n If we number the neurons of the network, the output (predictions) of a fully connected network like the one displayed in Fig. 4 can be computed as shown in the image – this process is sometimes known as <\/span>forward propagation<\/span><\/i>. The notation and operations seem a bit ugly (a.k.a. \u201cmore technical\u201d), but feel free to skip the figure if you are not interested in the maths. It is nothing more than applying the equation of Fig. 3 repeatedly for each neuron in the hidden layer and in the output layer. In the figure we are collecting together the parameters of each layer in matrices B<\/span>(j)<\/span>. We call the set of all of these matrices <\/span>B<\/i><\/b> – that\u2019s why the output of the network is a function of the inputs named <\/span>h<\/span><\/i>B<\/i><\/b>.<\/span><\/p>\n In case we are facing a binary classification problem, the class is determined by re-coding the output to one of the classes, either setting a threshold on the output (binary classification) or having as many neurons in the output layer as classes in our problem (multiclass classification). The output class computed by the network corresponds to the index of the neuron of the output layer which yields the highest output value.<\/span><\/p>\n In any Machine Learning model, fitting a model means finding the best values of its parameters. The best model is one whose parameter values minimize the total error with respect to the actual outputs on our training data. Since the error depends on the parameters chosen, it is a function of the parameters, called the <\/span>cost function <\/span><\/i>J. The most common error measure is the MSE (Mean Squared Error): <\/span> If the cost function is convex, it only has one global optimum, which is the global minimum. It can be found by computing the partial derivatives of J with respect to every variable (gradient vector ?J), and solving an equation system to find the point where the derivatives are all equal to 0 simultaneously (exact solution). Unfortunately, this can be difficult when the model equation is complicated (which yields a non-convex error function with many <\/span>local optima<\/span><\/i>), has many variables (large equation system) or is not derivable. That\u2019s why we often use some approximate, iterative optimization algorithm for this task, which does not guarantee the global minimum but generally performs well. Gradient descent is the most common, although there is a wide family of techniques which share the approximate, iterative nature (getting closer and closer to the solution at each time step <\/span>t<\/span><\/i>) and the need for evaluating the partial derivatives of J at different parameter points ?<\/span>(t)<\/span>.<\/span><\/p>\n
\n Figure 2: two interconnected human neurons <\/span><\/i><\/span><\/p>\n Figure 3: an artificial neuron with three inputs (left) and the sigmoid activation function (right)<\/span><\/i><\/span><\/span><\/h6>\nMultilayer perceptron and forward propagation to make predictions<\/b><\/h2>\n
<\/p>\n Figure 4: a fully connected network (multilayer perceptron) with one hidden layer and three inputs for examples with D = 3 features.<\/span><\/i><\/span><\/h6>\n
Training algorithm: cost function and gradient descent via backpropagation<\/b><\/h2>\n
where h<\/span>?0, ?1, \u2026, ?P<\/span> is the equation of the model (be it a neural network or any other machine learning model) depending on P+1 parameters, and <\/span>m<\/span><\/i> is the number of training examples. <\/span>
\n<\/span><\/p>\n<\/p>\n Fig 5: a plot of an error function J for two parameters and the iterative process of reaching a minimum (called \u201cconvergence\u201d). Note the starting point heavily affects the solution obtained.<\/span><\/i><\/span><\/h6>\n