Skip to main content

Understanding Activation Functions in Deep Learning

Activation functions are a core component of deep learning models, especially neural networks. They determine whether a neuron should be activated or not, introducing non-linearity into the model. Without activation functions, a neural network would behave like a linear regression model, no matter how many layers it had.

Role of Activation Functions

  1. Non-linearity: Real-world data is non-linear. Activation functions allow the model to learn complex patterns.
  2. Enabling deep learning: Deep networks rely on stacking layers. Without non-linearity, stacking layers is meaningless.
  3. Gradient flow: The function affects how gradients propagate during backpropagation, impacting training speed and performance.

Evolution of Activation Functions

Let’s walk through a timeline of activation functions and why newer ones replaced earlier ones.

1. Sigmoid Function

  • Formula: $\sigma(x)=\frac{1}{1+e^{-x}}(x)$
  • Range: (0, 1)
  • Problems:
    • Vanishing gradients for large/small inputs
    • Outputs not centered at 0
  • Good for binary classification

2. Tanh (Hyperbolic Tangent)

  • Formula: $\tanh(x)=\frac{e^x-e^{-1}}{e^x+e^{-x}}(x)$
  • Range: (-1, 1)
  • Zero-centered output
  • Still suffers from vanishing gradients

3. ReLU (Rectified Linear Unit)

  • Formula: $ReLU(x)=max(0,x)$
  • Simple and efficient
  • Sparse activation (some neurons deactivate)
  • Dying ReLU problem (neurons output zero for all inputs)

4. Leaky ReLU

  • Formula: $LeakyReLU(x)=max(\alpha x,x),\text{where alpha is small(e.g., 0.01)}$
  • Solves dying ReLU by allowing small negative slope
  • Still not perfect, requires tuning

5. ELU (Exponential Linear Unit)

  • Formula: $ELU(x)=\left\{\begin{matrix} x & if x> 0 \\ \alpha(e^x-1) & if x\leq 0 \\ \end{matrix}\right.$​
  • Smooth at 0
  • Can produce negative outputs, pushing mean activations closer to zero
  • Slightly more expensive to compute

6. Swish (by Google)

  • Formula: $Swish(x)=x\cdot \sigma(x)$
  • Smooth, non-monotonic
  • Often outperforms ReLU in deeper models
  • Slightly more computational overhead

7. Mish (newer, similar to Swish)

  • Formula: $Mish(x)=x\cdot \tanh(ln(1+e^x))$
  • Promotes smoother gradients
  • Shows better generalization in some tasks
  • Complex and computationally heavier

Comparison Diagram of Activation Functions

Here’s a plot comparing their shapes:


Reference

  • Sheng Shen, Zhewei Yao, Amir Gholami, Michael W. Mahoney, Kurt Keutzer. (2020). PowerNorm: Rethinking Batch Normalization in Transformers. arXiv:2003.07845 [cs].
  • Prajit Ramachandran, Barret Zoph, Quoc V. Le. (2017). Swish: A Self-Gated Activation Function. arXiv:1710.05941 [cs.NE].
  • Diganta Misra. (2019). Mish: A Self Regularized Non-Monotonic Neural Activation Function. arXiv:1908.08681 [cs.LG].
Tags

Comments

Post a Comment

Popular

Understanding SentencePiece: A Language-Independent Tokenizer for AI Engineers

In the realm of Natural Language Processing (NLP), tokenization plays a pivotal role in preparing text data for machine learning models. Traditional tokenization methods often rely on language-specific rules and pre-tokenized inputs, which can be limiting when dealing with diverse languages and scripts. Enter SentencePiece—a language-independent tokenizer and detokenizer designed to address these challenges and streamline the preprocessing pipeline for neural text processing systems. What is SentencePiece? SentencePiece is an open-source tokenizer and detokenizer developed by Google, tailored for neural-based text processing tasks such as Neural Machine Translation (NMT). Unlike conventional tokenizers that depend on whitespace and language-specific rules, SentencePiece treats the input text as a raw byte sequence, enabling it to process languages without explicit word boundaries, such as Japanese, Chinese, and Korean. This approach allows SentencePiece to train subword models di...
Post a Comment
Read more

Mastering the Byte Pair Encoding (BPE) Tokenizer for NLP and LLMs

Byte Pair Encoding (BPE) is one of the most important and widely adopted subword tokenization algorithms in modern Natural Language Processing (NLP), especially in training Large Language Models (LLMs) like GPT. This guide provides a deep technical dive into how BPE works, compares it with other tokenizers like WordPiece and SentencePiece, and explains its practical implementation with Python code. This article is optimized for AI engineers building real-world models and systems. 1. What is Byte Pair Encoding? BPE was originally introduced as a data compression algorithm by Gage in 1994. It replaces the most frequent pair of bytes in a sequence with a single, unused byte. In 2015, Sennrich et al. adapted BPE for NLP to address the out-of-vocabulary (OOV) problem in neural machine translation. Instead of working with full words, BPE decomposes them into subword units that can be recombined to represent rare or unseen words. 2. Why Tokenization Matters in LLMs Tokenization is th...
Post a Comment
Read more

Using Gemini API in LangChain: Step-by-Step Tutorial

What is LangChain and Why Use It? LangChain  is an open-source framework that simplifies the use of  Large Language Models (LLMs)  like OpenAI, Gemini (Google), and others by adding structure, tools, and memory to help build real-world applications such as chatbots, assistants, agents, or AI-enhanced software. Why Use LangChain for LLM Projects? Chainable Components : Easily build pipelines combining prompts, LLMs, tools, and memory. Multi-Model Support : Work with Gemini, OpenAI, Anthropic, Hugging Face, etc. Built-in Templates : Manage prompts more effectively. Supports Multi-Turn Chat : Manage complex interactions with memory and roles. Tool and API Integration : Let the model interact with external APIs or functions. Let's Walk Through the Code: Gemini + LangChain I will break the code into  4 main parts , each showcasing different features of LangChain and Gemini API. Part 1: Basic Gemini API Call Using LangChain import os from dotenv import load_dotenv load_dot...
Post a Comment
Read more