One-Hot Encoding

One-Hot Encoding is a method of representing data where each item (e.g., a word) is represented by a vector consisting entirely of zeros, except for a single 1 at the index corresponding to that item.

Limitation in LLMs

While useful in some contexts, One-Hot Encoding fails to capture semantic relationships between words.

• Example: A One-Hot vector for “dog” and a One-Hot vector for “puppy” are orthogonal (completely different) and show no mathematical closeness, despite the words being related in meaning.

This limitation led to the development of Vector Embeddings, or dense vectors, which can encode these relationships.

In Classification Tasks

One-Hot Encoding is widely used to represent class targets in classification problems.

• If there are $N$ classes, the target is a vector of length $N$ with a 1 at the correct class index and 0 everywhere else.
• Example: For classes Red, Green, Blue:
  • Red: [1, 0, 0]
  • Green: [0, 1, 0]
  • Blue: [0, 0, 1]