Cross entropy

Cross entropy is a derivative-based function as it uses the derivative of a specially designed equation, which is given as follows:

Cross entropy allows the network to learn faster when the difference between the expected and actual output is greater. In other words, the bigger the error, the faster it helps the network learn. We will get our heads around this using some simple code.

Like before, for now, you can use an online alternative if you do not have Python already installed, at https://www.jdoodle.com/python-programming-online. We will cover the installation and setup in Chapter 2, Creating a Real-Estate Price Prediction Mobile App. Follow these steps to see how a network learns using cross entropy:

First, let's import the math library so that we can use the log function:

from numpy import log

Next, let's define a function called cross_enrtopy, based on the preceding formula:

def cross_entropy(y,a): 
    return -1 *(y*log(a)+(1-y)*log (1-a))

For example, consider a single neuron with just one sample, (n=1). Say the expected output is 0 (y=0) and the neuron outputs 0.01 (a=0.01):

cross_entropy(0, 0.01)

The output is as follows:

0.010050335853501451

Since the expected and actual output values are very small, the resultant cost is very small.

Similarly, if the expected and actual output values are very large, then the resultant cost is still small:

cross_entropy(1000,999.99)

The output is as follows:

0.010050335853501451

Similarly, if the expected and actual output values are far apart, then the resultant cost is large:

cross_entropy(0,0.9)

The output is as follows:

2.3025850929940459

Therefore, the larger the difference in expected versus actual output, the faster the learning becomes. Using cross entropy, we can get the error of the network, and at the same time, the magnitude of the weights and bias is irrelevant, helping the network learn faster.