Natural Language Processing

Natural Language Processing (NLP) focuses on translating text sequences into mathematical representations that algorithms can learn from.

Statistical Representation: TF-IDF

TF-IDF (Term Frequency-Inverse Document Frequency) measures how important a word is to a document in a corpus:

$\text{TF-IDF}(t, d, D) = \text{TF}(t, d) \times \log\left(\frac{|D|}{1 + |\{d \in D : t \in d\}|}\right)$

While effective for classification, TF-IDF ignores word order and semantic similarity.

Sequence Modeling: RNNs and Attention

To handle sequential context, Recurrent Neural Networks (RNNs) maintain a hidden state vector $h_t$ that updates at each step:

$h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$

The Bottleneck of RNNs

RNNs struggle with long-term dependencies because gradients vanish over long sequence lengths. While LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) architectures use gating mechanisms to help, they still process tokens sequentially, which limits parallel training.

Self-Attention and the Transformer

The Transformer architecture replaces recurrent loops with Self-Attention. For queries $Q$, keys $K$, and values $V$, attention weights are computed in parallel:

$\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$

where $d_k$ is the dimension of the key vectors. This allows the model to capture relationships between words regardless of their distance in the sequence. Positional encodings are added to the input embeddings to represent sequence order. Transformers process the entire sequence in parallel via multi-head attention, capturing complex relationships between distant words. This represents a significant advancement over sequential RNN processing.

Example: TF-IDF Vectorization

The following example demonstrates tokenizing and vectorizing text using TF-IDF:

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
  "Machine learning learns from data.",
  "Data science uses machine learning."
]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(corpus)

print("Vocabulary:")
print(vectorizer.get_feature_names_out())
print("TF-IDF shape:")
print(tfidf_matrix.shape)

Vocabulary:
['data' 'learning' 'learns' 'machine' 'science' 'uses']
TF-IDF shape:
(2, 6)

Exercise

Validate your understanding of text vectorization properties:

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning (3rd ed.). O’Reilly Media.