AI Practitioner

Relational Deep Learning (RDL)  proposes a unified graph-based way to model multi-table databases for end-to-end learning using GNNs. This retains relational semantics, avoids joins, and supports temporal reasoning. It’s a paradigm shift that bridges the gap between ML and databases. 1. Motivation: From Tables to Graphs Traditional Setup Relational databases store structured data across multiple normalized tables , each capturing different types of entities (e.g., users, orders, products). These tables are linked by foreign-key (FK) and primary-key (PK) constraints. To train machine learning models, these databases are typically flattened into a single table using joins , and domain experts manually select and engineer features. Problems: Joins are expensive and brittle (schema changes break pipelines). Manual feature engineering is time-consuming and lacks relational awareness. Loss of information about cross-entity relationships . 2. Core Idea: Learn Direc...

In the realm of Natural Language Processing (NLP), tokenization plays a pivotal role in preparing text data for machine learning models. Traditional tokenization methods often rely on language-specific rules and pre-tokenized inputs, which can be limiting when dealing with diverse languages and scripts. Enter SentencePiece—a language-independent tokenizer and detokenizer designed to address these challenges and streamline the preprocessing pipeline for neural text processing systems. What is SentencePiece? SentencePiece is an open-source tokenizer and detokenizer developed by Google, tailored for neural-based text processing tasks such as Neural Machine Translation (NMT). Unlike conventional tokenizers that depend on whitespace and language-specific rules, SentencePiece treats the input text as a raw byte sequence, enabling it to process languages without explicit word boundaries, such as Japanese, Chinese, and Korean. This approach allows SentencePiece to train subword models di...

Implementation of RoPE Rotary Position Embedding (RoPE) is a positional encoding method introduced in the 2021 RoFormer paper ( https://arxiv.org/pdf/2104.09864 ). This technique overcomes the limitations of absolute positional encoding and enhances a Transformer model's ability to capture sequence order and relative positions effectively. 1. Limitations of Traditional Positional Encoding Since Transformers cannot inherently model token order, positional encodings are added to token embeddings. Early models used sinusoidal encodings, and later learnable embeddings were introduced. However, these approaches have several drawbacks: They encode absolute rather than relative positions, reducing contextual precision Struggle with generalizing to sequences of varying lengths Increase model parameters and often degrade in long-range dependencies ...