2. Embedding Model

An AI/ML model that converts text, images, or other data into numerical vectors in a high-dimensional space. It bridges the gap between human language and machine computation.

---

Table of Contents

• What is an Embedding Model?
• The Core Concept
• Types of Embeddings
• How Embeddings Capture Meaning
• Popular Embedding Models
• Choosing the Right Model
• Code Example
• Key Takeaway

---

What is an Embedding Model?

An embedding model is a neural network trained to map inputs — words, sentences, documents, images — into dense numerical vectors. These vectors live in a shared latent space where geometric proximity reflects semantic similarity.

  Input Text            Embedding Vector (1536 dimensions)
  ──────────            ──────────────────────────────────
  "I love pizza"   →   [0.21, 0.81, -0.11, 0.44, ..., 0.35]
  "I like pizza"   →   [0.19, 0.79, -0.09, 0.42, ..., 0.33]  ← very close!
  "Weather today"  →   [-0.45, 0.12, 0.66, -0.21, ..., 0.08] ← far away

---

The Core Concept

Embedding models are trained on massive datasets to learn that semantically similar inputs should produce numerically similar vectors. This is typically achieved via:

• Contrastive learning — pushing similar pairs closer, dissimilar pairs apart
• Masked language modelling — predicting hidden tokens to build contextual understanding
• Bi-encoder architecture — encoding two inputs independently, then comparing

The result is a model that "understands" meaning, enabling machines to reason about similarity without handcrafted rules.

---

Types of Embeddings

Word Embeddings

Early models that produce one vector per word. Limited — they don't capture context (the word "bank" has one vector regardless of whether it means a river bank or a financial bank).

Model	Released	Dimensions	Notes
Word2Vec	2013	100–300	Shallow neural net, fast
GloVe	2014	50–300	Co-occurrence matrix based
FastText	2016	100–300	Subword-aware, handles rare words

Sentence / Contextual Embeddings

Modern models produce one vector per sentence/passage, capturing full context.

Model	Dimensions	Notes
Sentence-BERT (SBERT)	384–1024	Fast, open-source, great for similarity
Universal Sentence Encoder	512	Google, multilingual
E5 / BGE	768–1024	Strong benchmarks, open weights

Document Embeddings

For encoding entire documents or long passages.

Model	Provider	Notes
text-embedding-3-small	OpenAI	1536 dims, great balance of quality/cost
text-embedding-3-large	OpenAI	3072 dims, highest quality
embed-english-v3.0	Cohere	Enterprise-grade, multilingual
amazon-titan-embed-text	AWS	Optimized for Bedrock ecosystem

Multimodal Embeddings

Embed text and images into the same vector space, enabling cross-modal search.

Model	Notes
CLIP (OpenAI)	Text ↔ Image in shared space
ImageBind (Meta)	Text, image, audio, video, IMU, depth

---

How Embeddings Capture Meaning

Semantic Arithmetic

A famous property of embeddings: vector arithmetic preserves meaning.

vector("King") - vector("Man") + vector("Woman") ≈ vector("Queen")
vector("Paris") - vector("France") + vector("Germany") ≈ vector("Berlin")

Similarity Measurement

The most common metric for comparing embeddings is cosine similarity:

similarity = (A · B) / (|A| × |B|)

Range: -1.0 (opposite) → 0.0 (unrelated) → 1.0 (identical)

Score	Interpretation
> 0.9	Near-identical meaning
0.7–0.9	Highly similar
0.5–0.7	Related
< 0.5	Mostly unrelated

---

Popular Embedding Models

                    Quality
                       ▲
              Large ●  │  ● text-embedding-3-large
                    │  │
         E5-Large ● │  │
                    │  │
      SBERT-Base ● ─┼──┼──────────────► Speed
                    │  │
         GloVe ●   │  │

Benchmark Performance (MTEB — Massive Text Embedding Benchmark)

Model	Avg Score	Dims	Cost
text-embedding-3-large	64.6	3072	$$
text-embedding-3-small	62.3	1536	$
Cohere embed-v3	64.5	1024	$$
E5-large-v2 (open)	62.2	1024	Free
all-MiniLM-L6-v2 (open)	56.2	384	Free

---

Choosing the Right Model

Criteria	Recommendation
Best quality, don't mind cost	text-embedding-3-large (OpenAI)
Balanced quality/cost	text-embedding-3-small (OpenAI)
Open-source, high quality	E5-large-v2 or BGE-large
Fast, local, lightweight	all-MiniLM-L6-v2 (sentence-transformers)
Multilingual	Cohere embed-multilingual-v3
Images + Text	CLIP or ImageBind

⚠️ Important: When switching embedding models, you must re-embed all stored data in your vector database — different models produce incompatible vector spaces.

---

Code Example

Using OpenAI Embeddings (Python)

from openai import OpenAI
import numpy as np

client = OpenAI(api_key="YOUR_API_KEY")

def get_embedding(text: str, model="text-embedding-3-small") -> list[float]:
    response = client.embeddings.create(input=text, model=model)
    return response.data[0].embedding

def cosine_similarity(a: list, b: list) -> float:
    a, b = np.array(a), np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Generate embeddings
emb1 = get_embedding("I love pizza")
emb2 = get_embedding("I really enjoy eating pizza")
emb3 = get_embedding("The stock market crashed today")

print(cosine_similarity(emb1, emb2))  # → ~0.93 (very similar)
print(cosine_similarity(emb1, emb3))  # → ~0.12 (unrelated)

Using sentence-transformers (local, free)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

sentences = [
    "I love pizza",
    "I enjoy eating pizza",
    "The weather is nice today"
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # → (3, 384)

# Compute similarity matrix
similarities = model.similarity(embeddings, embeddings)
print(similarities)

---

Key Takeaway

Embedding Models transform text (and other data) into numerical vectors that machines can understand, enabling semantic similarity search, smarter retrieval, and the foundation for RAG, recommendation systems, and modern AI memory.

---

← Vector Database | Back to Overview | RAG →