Skip to main content

Understanding Activation Functions in Deep Learning

Activation functions are a core component of deep learning models, especially neural networks. They determine whether a neuron should be activated or not, introducing non-linearity into the model. Without activation functions, a neural network would behave like a linear regression model, no matter how many layers it had.

Role of Activation Functions

  1. Non-linearity: Real-world data is non-linear. Activation functions allow the model to learn complex patterns.
  2. Enabling deep learning: Deep networks rely on stacking layers. Without non-linearity, stacking layers is meaningless.
  3. Gradient flow: The function affects how gradients propagate during backpropagation, impacting training speed and performance.

Evolution of Activation Functions

Let’s walk through a timeline of activation functions and why newer ones replaced earlier ones.

1. Sigmoid Function

  • Formula: $\sigma(x)=\frac{1}{1+e^{-x}}(x)$
  • Range: (0, 1)
  • Problems:
    • Vanishing gradients for large/small inputs
    • Outputs not centered at 0
  • Good for binary classification

2. Tanh (Hyperbolic Tangent)

  • Formula: $\tanh(x)=\frac{e^x-e^{-1}}{e^x+e^{-x}}(x)$
  • Range: (-1, 1)
  • Zero-centered output
  • Still suffers from vanishing gradients

3. ReLU (Rectified Linear Unit)

  • Formula: $ReLU(x)=max(0,x)$
  • Simple and efficient
  • Sparse activation (some neurons deactivate)
  • Dying ReLU problem (neurons output zero for all inputs)

4. Leaky ReLU

  • Formula: $LeakyReLU(x)=max(\alpha x,x),\text{where alpha is small(e.g., 0.01)}$
  • Solves dying ReLU by allowing small negative slope
  • Still not perfect, requires tuning

5. ELU (Exponential Linear Unit)

  • Formula: $ELU(x)=\left\{\begin{matrix} x & if x> 0 \\ \alpha(e^x-1) & if x\leq 0 \\ \end{matrix}\right.$​
  • Smooth at 0
  • Can produce negative outputs, pushing mean activations closer to zero
  • Slightly more expensive to compute

6. Swish (by Google)

  • Formula: $Swish(x)=x\cdot \sigma(x)$
  • Smooth, non-monotonic
  • Often outperforms ReLU in deeper models
  • Slightly more computational overhead

7. Mish (newer, similar to Swish)

  • Formula: $Mish(x)=x\cdot \tanh(ln(1+e^x))$
  • Promotes smoother gradients
  • Shows better generalization in some tasks
  • Complex and computationally heavier

Comparison Diagram of Activation Functions

Here’s a plot comparing their shapes:


Reference

  • Sheng Shen, Zhewei Yao, Amir Gholami, Michael W. Mahoney, Kurt Keutzer. (2020). PowerNorm: Rethinking Batch Normalization in Transformers. arXiv:2003.07845 [cs].
  • Prajit Ramachandran, Barret Zoph, Quoc V. Le. (2017). Swish: A Self-Gated Activation Function. arXiv:1710.05941 [cs.NE].
  • Diganta Misra. (2019). Mish: A Self Regularized Non-Monotonic Neural Activation Function. arXiv:1908.08681 [cs.LG].
Tags

Comments

Post a Comment

Popular

How to Fine-Tune LLaMA 3.2-1B-Instruct for Korean Instruction Tasks with LoRA and Hugging Face

LLaMA 3.2-1B-Instruct is a lightweight instruction-tuned language model released by Meta. It is designed to handle a wide range of instruction-based tasks with relatively low computational resources. Although the model was trained with multilingual capabilities, its performance on languages not included in its training set—such as Korean—is limited. This tutorial demonstrates how to fine-tune this open-source model on a Korean dataset using Hugging Face Transformers and PEFT (specifically LoRA), enabling it to better respond to Korean instructions. 1. Prerequisites Before running the example code below, ensure you have the following libraries installed: pip install torch transformers datasets peft accelerate mlflow huggingface_hub To use the LLaMA model or KoAlpaca datasets, you'll need a Hugging Face token. Additionally, you may need to handle potential CUDA Out-Of-Memory (OOM) errors. The following code takes care of both: from huggingface_hub import login login("y...
Post a Comment
Read more

RF-DETR: Overcoming the Limitations of DETR in Object Detection

RF-DETR (Region-Focused DETR), proposed in April 2025, is an advanced object detection architecture designed to overcome fundamental drawbacks of the original DETR (DEtection TRansformer) . In this technical article, we explore RF-DETR's contributions, architecture, and how it compares with both DETR and the improved model D-FINE . We also provide experimental benchmarks and discuss its real-world applicability. RF-DETR Architecture diagram for object detection Limitations of DETR DETR revolutionized object detection by leveraging the Transformer architecture, enabling end-to-end learning without anchor boxes or NMS (Non-Maximum Suppression). However, DETR has notable limitations: Slow convergence, requiring heavy data augmentation and long training schedules Degraded performance on low-resolution objects and complex scenes Lack of locality due to global self-attention mechanisms Key Innovations in RF-DETR RF-DETR intr...
Post a Comment
Read more

Using Gemini API in LangChain: Step-by-Step Tutorial

What is LangChain and Why Use It? LangChain  is an open-source framework that simplifies the use of  Large Language Models (LLMs)  like OpenAI, Gemini (Google), and others by adding structure, tools, and memory to help build real-world applications such as chatbots, assistants, agents, or AI-enhanced software. Why Use LangChain for LLM Projects? Chainable Components : Easily build pipelines combining prompts, LLMs, tools, and memory. Multi-Model Support : Work with Gemini, OpenAI, Anthropic, Hugging Face, etc. Built-in Templates : Manage prompts more effectively. Supports Multi-Turn Chat : Manage complex interactions with memory and roles. Tool and API Integration : Let the model interact with external APIs or functions. Let's Walk Through the Code: Gemini + LangChain I will break the code into  4 main parts , each showcasing different features of LangChain and Gemini API. Part 1: Basic Gemini API Call Using LangChain import os from dotenv import load_dotenv load_dot...
Post a Comment
Read more