AI Practitioner

What is Entropy in Deep Learning? In  deep learning , entropy usually refers to  information entropy , a concept from information theory introduced by Claude Shannon in 1948. It measures the  uncertainty or randomness  in a probability distribution. Intuition: If you're very  uncertain  about something (like predicting the class of an image), the  entropy is high . If you're  confident  about your prediction (for example, the model is very sure it’s a cat), the  entropy is low . Mathematical Definition (Shannon Entropy) Given a probability distribution $p = [p_1, p_2, ..., p_n]$, the  entropy H(p)  is calculated as: This formula sums the "surprise" or "information content" from each possible outcome. Entropy in Deep Learning: Where and Why? In deep learning, entropy is most commonly used in the  loss function , particularly: Cross Entropy Loss Used for  classification problems . Compares the  tr...