SVD and Truncated SVD Explained: Theory, Python Examples, and Applications in Machine Learning & Deep Learning

Singular Value Decomposition (SVD) is a matrix factorization method widely used in mathematics, engineering, and economics. Since it's a crucial concept applied in accelerating matrix computations and data compression, it's worth studying at least once.

SVD (Singular Value Decomposition) Theory

Singular Value Decomposition (SVD) is a matrix factorization technique applicable to any real or complex matrix. Any matrix A (m×n) can be decomposed as follows:

$A = U * \Sigma * V^T$

• U: Orthogonal matrix composed of left singular vectors $(m \times m)$
• Σ: Diagonal matrix $(m \times n)$ with singular values on the diagonal
• $V^T$: Transposed matrix of right singular vectors $(n \times n)$

The singular values represent the energy or information content of matrix A, enabling tasks like dimensionality reduction or noise filtering.

Truncated SVD

Truncated SVD approximates the original matrix using only the top k singular values and corresponding singular vectors:

$A \approx U_k * \Sigma_k * V_k^T$

This technique is useful for reducing dimensionality while retaining most of the information, especially in sparse matrices, text data, and image processing.

---

Implementing SVD and Truncated SVD in Python

import numpy as np

def compute_svd(A):
    AT_A = np.dot(A.T, A)
    eigvals, V = np.linalg.eigh(AT_A)
    idx = np.argsort(eigvals)[::-1]
    eigvals = eigvals[idx]
    V = V[:, idx]
    singular_values = np.sqrt(eigvals)
    Σ = np.diag(singular_values)
    U = np.dot(A, V) / singular_values
    return U, Σ, V.T

A = np.array([[3, 2], [2, 3]])
U, Σ, VT = compute_svd(A)
print("U:\n", U)
print("Σ:\n", Σ)
print("V^T:\n", VT)

Output:

U:
 [[ 0.70710678 -0.70710678]
 [ 0.70710678  0.70710678]]
Σ:
 [[5. 0.]
 [0. 1.]]
V^T:
 [[ 0.70710678  0.70710678]
 [-0.70710678  0.70710678]]

Truncated SVD with Scikit-learn

from sklearn.decomposition import TruncatedSVD
import numpy as np

X = np.array([[3, 2, 2], [2, 3, -2]])
svd = TruncatedSVD(n_components=2)
X_reduced = svd.fit_transform(X)
print("Reduced X:\n", X_reduced)
print("Components:\n", svd.components_)

Output:

Reduced X:
 [[ 3.53553391  2.12132034]
 [ 3.53553391 -2.12132034]]
Components:
 [[ 7.07106781e-01  7.07106781e-01  4.33680869e-18]
 [ 2.35702260e-01 -2.35702260e-01  9.42809042e-01]]

Truncated SVD is particularly suitable for sparse data (e.g., TF-IDF matrices, user-item matrices), and unlike PCA, it does not require mean-centering the data.

---

Applications of Truncated SVD in Deep Learning / Machine Learning

1. Dimensionality Reduction

Used to compress high-dimensional data (e.g., text, images) into lower dimensions, reducing computation and improving generalization.

2. Natural Language Processing (NLP)

• LSA (Latent Semantic Analysis): Applies Truncated SVD to a document-word matrix to extract semantic relationships between words.
• Word Embedding Preprocessing: Used to reduce the dimensionality of vectors like FastText and GloVe for memory optimization.

3. Recommendation Systems

Truncated SVD is applied to sparse user-item matrices to learn latent factor models, used in systems like Netflix and YouTube.

4. Model Compression in Deep Learning

LoRA (Low-Rank Adaptation) is a technique for efficient fine-tuning of large language models (e.g., GPT, BERT). It approximates weight matrices using a low-rank form:

W ≈ W0 + A * B  # A and B are low-rank matrices

A and B are learnable, allowing quick adaptation without retraining the entire model. LoRA is a representative method inspired by the principles of SVD.

5. Vision Applications

Used to approximate convolutional weights in neural networks with low-rank matrices to reduce memory and computation costs.

---

References

• Golub, G. H., & Van Loan, C. F. (2013). Matrix Computations.
• NumPy Documentation
• Scikit-learn TruncatedSVD
• Wikipedia: Singular Value Decomposition
• LoRA Paper (Hu et al., 2021)
• "Matrix Factorization Techniques for Recommender Systems", Koren et al. (2009)
• "Efficient Estimation of Word Representations in Vector Space", Mikolov et al. (2013)