Deep Learning Feature Normalization Methods Explained

Feature normalization methods are critical when training deep learning models because they help improve model performance, convergence speed, and training stability.

Why Feature Normalization is Used in Deep Learning

1. Accelerates Convergence

• Neural networks are typically trained using gradient-based optimizers like SGD, Adam, etc.
• If input features have different scales (e.g., one feature ranges from 0 to 1, while another ranges from 0 to 1000), the loss surface becomes distorted or ill-conditioned.
• This causes gradients to oscillate, slowing down learning or even making it unstable.
• Normalized inputs ensure the model sees data on a similar scale, resulting in smoother loss surfaces and faster convergence.

2. Improves Numerical Stability

• Deep models can suffer from exploding or vanishing gradients if activations or weights grow too large or small.
• Normalization (especially internal ones like Batch Normalization) helps maintain stable distributions throughout layers.
• This prevents overflow or underflow, especially important in deep or recurrent networks.

3. Better Generalization

• When features are on different scales, the model might become biased toward features with larger values.
• Normalizing ensures all features contribute equally during learning.
• This leads to more balanced weights, helping the model generalize better to unseen data.

4. Reduces Sensitivity to Weight Initialization

• If feature scales vary widely, the model's performance can become highly sensitive to initial weight values.
• Normalized features allow the use of standard initialization techniques (e.g., Xavier, He), which improves stability across runs.

Common Normalization Techniques

1. Batch Normalization (BN)

BN normalizes the input along the batch dimension for each feature channel, making the distribution of each feature more stable.

Applicable Shape:

Assume input is a 3D feature tensor:
(B, C, L)

• B: Batch size
• C: Number of channels
• L: Sequence length / spatial size (1D case for simplicity)

BN normalizes over B and L for each channel.

Formula:

2. Layer Normalization (LN)

LN normalizes across all features per sample, not across the batch.

Applicable Shape:

Input: (B, C, L)
LN normalizes across (C, L) per sample (i.e., for each B).

Formula:

For each sample i:

3. L1 Normalization (Manhattan Norm)

Scales input by the L1 norm (sum of absolute values).

Typically used to normalize vectors:

Use Case:

• Sparse models, attention distributions (e.g., softmax after L1), or interpretability.

4. L2 Normalization (Euclidean Norm)}

Scales input vector to unit norm based on Euclidean (L2) norm:

Use Case:

• Feature scaling for distance-based tasks: e.g., face recognition, text embedding similarity.

5. Comparison Table:

MethodNormalization AxisTraining DependencyProsCons

BatchNorm	Across batch & spatial (per channel)	Needs batch	- Accelerates training - Reduces covariate shift	- Poor for small batches - Depends on training mode
LayerNorm	Across features (per sample)	No batch needed	- Stable for NLP/RNNs - Good for transformers	- Slightly slower - Less effective for CNNs
L1 Norm	Per vector	No batch needed	- Sparsity-friendly - Simple & interpretable	- Not suitable for all tasks - Less stable in DL
L2 Norm	Per vector	No batch needed	- Useful for similarity tasks - Keeps scale info	- Not trainable - May not improve training speed

References

1. Captured from Sheng Shen, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020. PowerNorm: Rethinking Batch Normalization in Transformers. arXiv:2003.07845 [cs] (June 2020)