The Gradient descent algorithm we saw in an earlier post just gives you an idea of how gradient descent works. We’ve seen that

$$W \leftarrow W - \alpha E’(W)\\
$$

This is just a rough formula to get you started. In practice what $E’(W)$ is calculating is the average of all the gradients due to every point in your training data. This is known as Vanilla Gradient Descent or Batch Gradient Descent. Large datasets with millions of instances are easily available nowadays. This presents a challenge to the vanilla gradient descent. The weights are updated after calculating the average of all the data points available, which significantly increases training time.

Let’s see some optimizations to keep these under control.

In the last post on overfitting and underfitting. We’ve seen that to achieve a perfect classifier is a tedious task. In this post let’s see some common tricks and methods to control the generalization of a network.

Just to recap what we’ve seen so far,

• Part 1: Neuron and the perceptron algorithm
• Part 2: Multilayer Perceptrons and Activation Function
• Part 3: Cost Function and Gradient Descent
• Part 4: Forward and Backward Propagation
• Part 5: Loss function and Cross-entropy

In the gradient descent post, you’ve seen what an error function is. What the characteristics of a good loss function are. Let’s take a deep dive into how we achieve this.

Till now we’ve seen

• How a Multilayer Perceptron works
• How Gradient Descent helps Multilayer Perceptron

Now let’s see some tricks that extend these ideas to Deep neural networks.

In the last post we’ve discussed how a Multilayer Perceptron works. Now let’s see how to search/update a set of weights to achieve a good classifer.

The XOR problem

In the previous post you’ve seen how a perceptron works. Now let’s dive into some more interesting problems in deep learning. What follows is the classic XOR problem. Develop a method for the correct classification of the following points.

This is the first part of the series “Deep Learning Fundamentals”. The goal of this series is to explore the mechanisms of artificial neural networks. The focus is more on presenting an intuitive way of understanding neural networks. So, you can expect an emphasis on how and why things work rather than what does the job. More often than not I’ll try to use simple math without focusing on notation. Let’s jump into the fundamental unit of most of the neural networks - “a neuron”.