How do we know if our neural network is doing a good job? We need a score to measure its performance. A Loss Function (or Cost Function) quantifies the error between the network’s prediction and the actual target.
Different tasks require different loss functions. Here, represents the predicted value (our activation output ) and represents the true target.
- Mean Absolute Error (MAE): often used for regression when outliers shouldn’t be penalized too heavily.
- Cross-Entropy Loss: The standard for classification tasks like image classification and also used for predicting the next token in GPT models.
