When training a neural network, the correctness of the network's output must be assessed. 1 As we know the expected correct output of training data, the output of training can be compared. The cost function measures the difference between actual and training outputs. A cost of zero between the actual and expected outputs would signify that the network has been training as would be possible; this would clearly be ideal.
Matthew, Mayo. “Deep Learning Key Terms, Explained.” KDnuggets, KDnuggets, 10AD, 2016, http://www.kdnuggets.com/2016/10/deep-learning-key-terms-explained.html (1)