Skip to main content

Understanding Information Theory and Information Bottleneck

What is Information Theory?

Information theory is a mathematical framework developed by Claude Shannon in the 1940s to understand how information is measured, transmitted, and compressed.

At its core, it deals with questions like:

  • How much information is in a message?
  • How can we represent that information efficiently?
  • How can we reduce noise when transmitting information?

Key Concept: Entropy

Entropy is a measure of uncertainty or unpredictability.

Think of it like this:

  • A fair coin (50% heads, 50% tails) has high entropy because it’s unpredictable.
  • A coin that always lands on heads has zero entropy because it’s completely predictable.

In deep learning, entropy tells us how much uncertainty there is in the model’s prediction.

What is the Information Bottleneck?

Imagine you're trying to compress an image to send over the internet. You want to remove unnecessary parts (like background noise) but keep the important content $\text{(like a person’s face)}$.

This is the idea behind the Information Bottleneck.

In deep learning:

  • A neural network tries to learn a mapping from input (X) to output (Y).
  • The Information Bottleneck principle says: try to compress the input X into a hidden representation T, such that T contains as little information about X as possible while still keeping enough information to predict Y well.

This forces the model to focus only on relevant information and ignore noise.

Example: Classifying Handwritten Digits (MNIST)

Imagine a neural network classifying digits from the MNIST dataset.

  • Input(X): Raw pixel values from an image.
  • Output(Y): Digit label (0–9).
  • We want the hidden layers (T) to keep only what’s needed to guess the digit (like shape), and discard irrelevant things (like handwriting style or stroke thickness).

Information Bottleneck in Neural Networks

Here’s a visual breakdown:


Overtraining or keeping too much detail in T can lead to:

  • Overfitting (memorizing noise)
  • Poor generalization

2025.04.22 - [AI] - Entropy & Cross Entropy Loss in Deep Learning

References

  1. Tishby et al. (2000) – The Information Bottleneck Method
    https://arxiv.org/abs/physics/0004057
  2. Tishby and Zaslavsky (2015) – Deep Learning and the Information Bottleneck Principle
    https://arxiv.org/abs/1503.02406
  3. Alemi et al. (2016) – Deep Variational Information Bottleneck
    https://arxiv.org/abs/1612.00410

Tags

Comments

Post a Comment

Popular

Building an MCP Agent with UV, Python & mcp-use

Model Context Protocol (MCP) is an open protocol designed to enable AI agents to interact with external tools and data in a standardized way. MCP is composed of three components: server , client , and host . MCP host The MCP host acts as the interface between the user and the agent   (such as Claude Desktop or IDE) and plays the role of connecting to external tools or data through MCP clients and servers. Previously, Anthropic’s Claude Desktop was introduced as a host, but it required a separate desktop app, license, and API key management, leading to dependency on the Claude ecosystem.   mcp-use is an open-source Python/Node package that connects LangChain LLMs (e.g., GPT-4, Claude, Groq) to MCP servers in just six lines of code, eliminating dependencies and supporting multi-server and multi-model setups. MCP Client The MCP client manages the MCP protocol within the host and is responsible for connecting to MCP servers that provide the necessary functions for the ...
Post a Comment
Read more

How to Save and Retrieve a Vector Database using LangChain, FAISS, and Gemini Embeddings

How to Save and Retrieve a Vector Database using LangChain, FAISS, and Gemini Embeddings Efficient storage and retrieval of vector databases is foundational for building intelligent retrieval-augmented generation (RAG) systems using large language models (LLMs). In this guide, we’ll walk through a professional-grade Python implementation that utilizes LangChain with FAISS and Google Gemini Embeddings to store document embeddings and retrieve similar information. This setup is highly suitable for advanced machine learning (ML) and deep learning (DL) engineers who work with semantic search and retrieval pipelines. Why Vector Databases Matter in LLM Applications Traditional keyword-based search systems fall short when it comes to understanding semantic meaning. Vector databases store high-dimensional embeddings of text data, allowing for approximate nearest-neighbor (ANN) searches based on semantic similarity. These capabilities are critical in applications like: Question Ans...
Post a Comment
Read more

Using Gemini API in LangChain: Step-by-Step Tutorial

What is LangChain and Why Use It? LangChain  is an open-source framework that simplifies the use of  Large Language Models (LLMs)  like OpenAI, Gemini (Google), and others by adding structure, tools, and memory to help build real-world applications such as chatbots, assistants, agents, or AI-enhanced software. Why Use LangChain for LLM Projects? Chainable Components : Easily build pipelines combining prompts, LLMs, tools, and memory. Multi-Model Support : Work with Gemini, OpenAI, Anthropic, Hugging Face, etc. Built-in Templates : Manage prompts more effectively. Supports Multi-Turn Chat : Manage complex interactions with memory and roles. Tool and API Integration : Let the model interact with external APIs or functions. Let's Walk Through the Code: Gemini + LangChain I will break the code into  4 main parts , each showcasing different features of LangChain and Gemini API. Part 1: Basic Gemini API Call Using LangChain import os from dotenv import load_dotenv load_dot...
Post a Comment
Read more