Introduction

Traditional keyword search finds documents containing the words supplied by a user. This works well when the query and the relevant document use matching words, identifiers, or phrases. However, two texts may express similar meanings while using different vocabulary.

An embedding model converts text, images, audio, products, users, or other objects into numerical vectors. Semantically related objects are generally represented by vectors that are close to one another according to a selected similarity or distance measure.

A vector database or vector-search system stores these vectors together with identifiers, metadata, and source information. It uses specialized indexes to retrieve the vectors most similar to a query vector.

FAISS provides a high-performance library for building similarity-search indexes. OpenSearch combines vector retrieval with document search, filtering, and search-engine capabilities. Pinecone provides a managed vector-search platform with indexes, namespaces, records, and metadata filtering.

Learning Objectives

Prerequisites

python -m pip install \
  numpy \
  faiss-cpu \
  sentence-transformers \
  opensearch-py \
  pinecone

What Is a Vector?

A vector is an ordered list of numerical values. In Machine Learning, a vector can represent the learned characteristics of a sentence, document, image, product, customer, or another object.

What Is an Embedding?

An embedding is a dense numerical representation created by a trained model. Embeddings attempt to preserve useful relationships among objects in a vector space.

What Is a Vector Database?

A vector database stores embeddings and provides indexes and query operations optimized for similarity-based retrieval.

Keyword Search vs Vector Search

Nearest-Neighbor Search

Nearest-neighbor search finds stored vectors closest to a query vector according to a selected distance or similarity function.

Here, \(\mathbf{q}\) is the query vector, \(\mathbf{x}_i\) is an indexed vector, and \(d\) is the selected distance function.

Similarity and Distance Measures

Vector Normalization

L2 normalization scales a non-zero vector to unit length.

For unit-normalized vectors, cosine similarity equals the dot product. This permits cosine-oriented retrieval using an inner-product index when normalization is applied consistently.

import numpy as np


def l2_normalize(
    vectors
):
    vectors = np.asarray(
        vectors,
        dtype=np.float32
    )

    norms = np.linalg.norm(
        vectors,
        axis=1,
        keepdims=True
    )

    if np.any(
        norms == 0
    ):
        raise ValueError(
            "Zero-length vectors cannot "
            "be normalized safely."
        )

    return vectors / norms

Exact vs Approximate Search

Common Vector Index Strategies

Metadata and Filtering

Metadata describes an indexed object without forming part of the dense embedding. Metadata filters restrict the set of records eligible for similarity search.

{
    "id": "document-42-chunk-3",
    "vector": [
        0.12,
        -0.47,
        0.81
    ],
    "metadata": {
        "document_id": "document-42",
        "category": "technical-manual",
        "language": "en",
        "version": "3",
        "access_group": "engineering"
    }
}

Namespaces and Collections

Vector systems may organize records through indexes, collections, namespaces, shards, or other logical boundaries. These structures can support workload organization, lifecycle management, and retrieval isolation.

Chunking for Vector Retrieval

Long documents are commonly divided into smaller chunks before embeddings are created. Chunking controls how much meaning and context each vector represents.

Simple Overlapping Chunker

def create_overlapping_chunks(
    text,
    chunk_size,
    overlap
):
    if chunk_size <= 0:

        raise ValueError(
            "chunk_size must be positive."
        )

    if (
        overlap < 0
        or overlap >= chunk_size
    ):
        raise ValueError(
            "overlap must be greater than "
            "or equal to zero and smaller "
            "than chunk_size."
        )

    words = str(
        text
    ).split()

    step_size = (
        chunk_size
        -
        overlap
    )

    chunks = []

    for start_index in range(
        0,
        len(
            words
        ),
        step_size
    ):
        chunk_words = words[
            start_index:
            start_index + chunk_size
        ]

        if not chunk_words:
            break

        chunks.append(
            " ".join(
                chunk_words
            )
        )

        if (
            start_index
            +
            chunk_size
            >=
            len(
                words
            )
        ):
            break

    return chunks

Vector Ingestion Pipeline

Validate Embeddings Before Indexing

import numpy as np


def validate_embeddings(
    embeddings,
    expected_dimension
):
    embeddings = np.asarray(
        embeddings,
        dtype=np.float32
    )

    if embeddings.ndim != 2:

        raise ValueError(
            "Embeddings must be a "
            "two-dimensional array."
        )

    if embeddings.shape[
        1
    ] != expected_dimension:

        raise ValueError(
            "Embedding dimension does "
            "not match the index."
        )

    if not np.isfinite(
        embeddings
    ).all():

        raise ValueError(
            "Embeddings contain non-finite values."
        )

    if embeddings.shape[
        0
    ] == 0:

        raise ValueError(
            "At least one embedding is required."
        )

    return embeddings

Generate Text Embeddings

from sentence_transformers import SentenceTransformer


embedding_model_id = (
    "sentence-transformers/"
    "all-MiniLM-L6-v2"
)


embedding_model = SentenceTransformer(
    embedding_model_id
)


documents = [
    "MLOps automates and governs the Machine Learning lifecycle.",
    "A feature store manages reusable features for training and inference.",
    "Vector search retrieves objects using embedding similarity.",
    "Model monitoring tracks production quality and operational behavior."
]


document_embeddings = (
    embedding_model.encode(
        documents,
        convert_to_numpy=True,
        normalize_embeddings=True,
        show_progress_bar=False
    )
    .astype(
        "float32"
    )
)


print(
    document_embeddings.shape
)

What Is FAISS?

FAISS is a similarity-search and clustering library for dense vectors. It provides multiple index structures and CPU and GPU implementations for supported workflows.

Build an Exact FAISS Index

import faiss
import numpy as np


document_embeddings = np.asarray(
    document_embeddings,
    dtype=np.float32
)


dimension = document_embeddings.shape[
    1
]


faiss_index = faiss.IndexFlatIP(
    dimension
)


faiss_index.add(
    document_embeddings
)


print(
    {
        "dimension": dimension,
        "indexed_vectors": (
            faiss_index.ntotal
        )
    }
)

Search the FAISS Index

query_text = (
    "How can semantic similarity "
    "be used to retrieve information?"
)


query_embedding = (
    embedding_model.encode(
        [
            query_text
        ],
        convert_to_numpy=True,
        normalize_embeddings=True,
        show_progress_bar=False
    )
    .astype(
        "float32"
    )
)


top_k = 3


scores, indices = faiss_index.search(
    query_embedding,
    top_k
)


results = []


for score, index_position in zip(
    scores[
        0
    ],
    indices[
        0
    ]
):
    if index_position == -1:
        continue

    results.append(
        {
            "document": documents[
                index_position
            ],
            "score": float(
                score
            )
        }
    )


print(
    results
)

Use Explicit IDs with FAISS

import faiss
import numpy as np


base_index = faiss.IndexFlatIP(
    dimension
)


id_index = faiss.IndexIDMap2(
    base_index
)


vector_ids = np.array(
    [
        101,
        102,
        103,
        104
    ],
    dtype=np.int64
)


id_index.add_with_ids(
    document_embeddings,
    vector_ids
)


scores, retrieved_ids = (
    id_index.search(
        query_embedding,
        top_k
    )
)


print(
    retrieved_ids
)

Save and Load a FAISS Index

import faiss
from pathlib import Path


index_path = Path(
    "artifacts/document_index.faiss"
)


index_path.parent.mkdir(
    parents=True,
    exist_ok=True
)


faiss.write_index(
    id_index,
    str(
        index_path
    )
)


loaded_index = faiss.read_index(
    str(
        index_path
    )
)


print(
    loaded_index.ntotal
)

Build a FAISS HNSW Index

import faiss


hnsw_neighbors = 32


hnsw_index = (
    faiss.IndexHNSWFlat(
        dimension,
        hnsw_neighbors,
        faiss.METRIC_INNER_PRODUCT
    )
)


hnsw_index.hnsw.efConstruction = 100

hnsw_index.hnsw.efSearch = 64


hnsw_index.add(
    document_embeddings
)


scores, indices = hnsw_index.search(
    query_embedding,
    top_k
)

Build a FAISS IVF Index

import faiss


number_of_partitions = 16


quantizer = faiss.IndexFlatIP(
    dimension
)


ivf_index = faiss.IndexIVFFlat(
    quantizer,
    dimension,
    number_of_partitions,
    faiss.METRIC_INNER_PRODUCT
)


if document_embeddings.shape[
    0
] < number_of_partitions:

    raise ValueError(
        "More representative training "
        "vectors are required for this "
        "IVF configuration."
    )


ivf_index.train(
    document_embeddings
)


ivf_index.add(
    document_embeddings
)


ivf_index.nprobe = 4


scores, indices = ivf_index.search(
    query_embedding,
    top_k
)

Evaluate Approximate Retrieval Recall

Here, \(R_k\) is the approximate result set and \(G_k\) is the exact reference result set.

def calculate_recall_at_k(
    retrieved_ids,
    ground_truth_ids
):
    retrieved_set = set(
        retrieved_ids
    )

    ground_truth_set = set(
        ground_truth_ids
    )

    if not ground_truth_set:

        raise ValueError(
            "Ground-truth IDs are required."
        )

    return (
        len(
            retrieved_set
            &
            ground_truth_set
        )
        /
        len(
            ground_truth_set
        )
    )

FAISS Index Selection

What Is OpenSearch Vector Search?

OpenSearch can store vector fields alongside document content and metadata. This makes it suitable for applications combining vector similarity, lexical search, filters, and document-oriented search capabilities.

Connect to OpenSearch

import os

from opensearchpy import OpenSearch


opensearch_host = os.environ[
    "OPENSEARCH_HOST"
]


opensearch_username = os.environ[
    "OPENSEARCH_USERNAME"
]


opensearch_password = os.environ[
    "OPENSEARCH_PASSWORD"
]


opensearch_client = OpenSearch(
    hosts=[
        {
            "host": opensearch_host,
            "port": 443
        }
    ],
    http_auth=(
        opensearch_username,
        opensearch_password
    ),
    use_ssl=True,
    verify_certs=True
)

Create an OpenSearch Vector Index

index_name = (
    "course-articles-v1"
)


index_definition = {
    "settings": {
        "index": {
            "knn": True
        }
    },
    "mappings": {
        "properties": {
            "chunk_id": {
                "type": "keyword"
            },
            "document_id": {
                "type": "keyword"
            },
            "title": {
                "type": "text"
            },
            "chunk_text": {
                "type": "text"
            },
            "category": {
                "type": "keyword"
            },
            "language": {
                "type": "keyword"
            },
            "access_group": {
                "type": "keyword"
            },
            "embedding_model": {
                "type": "keyword"
            },
            "content_vector": {
                "type": "knn_vector",
                "dimension": dimension,
                "method": {
                    "name": "hnsw",
                    "space_type": "cosinesimil",
                    "engine": "lucene",
                    "parameters": {
                        "ef_construction": 128,
                        "m": 24
                    }
                }
            }
        }
    }
}


if not opensearch_client.indices.exists(
    index=index_name
):

    opensearch_client.indices.create(
        index=index_name,
        body=index_definition
    )

Index an OpenSearch Document

document_record = {
    "chunk_id": "article-20-8-chunk-1",
    "document_id": "article-20-8",
    "title": (
        "Vector Databases"
    ),
    "chunk_text": documents[
        2
    ],
    "category": "generative-ai",
    "language": "en",
    "access_group": "learners",
    "embedding_model": (
        embedding_model_id
    ),
    "content_vector": (
        document_embeddings[
            2
        ].tolist()
    )
}


opensearch_client.index(
    index=index_name,
    id=document_record[
        "chunk_id"
    ],
    body=document_record,
    refresh=True
)

Search OpenSearch by Vector

query_body = {
    "size": 3,
    "_source": [
        "chunk_id",
        "document_id",
        "title",
        "chunk_text",
        "category"
    ],
    "query": {
        "knn": {
            "content_vector": {
                "vector": (
                    query_embedding[
                        0
                    ].tolist()
                ),
                "k": 3
            }
        }
    }
}


search_response = (
    opensearch_client.search(
        index=index_name,
        body=query_body
    )
)


for hit in search_response[
    "hits"
][
    "hits"
]:
    print(
        {
            "score": hit[
                "_score"
            ],
            "source": hit[
                "_source"
            ]
        }
    )

OpenSearch Vector Search with a Filter

filtered_query = {
    "size": 3,
    "_source": [
        "chunk_id",
        "title",
        "chunk_text",
        "category",
        "language"
    ],
    "query": {
        "knn": {
            "content_vector": {
                "vector": (
                    query_embedding[
                        0
                    ].tolist()
                ),
                "k": 3,
                "filter": {
                    "bool": {
                        "must": [
                            {
                                "term": {
                                    "language": "en"
                                }
                            },
                            {
                                "term": {
                                    "access_group": (
                                        "learners"
                                    )
                                }
                            }
                        ]
                    }
                }
            }
        }
    }
}


filtered_response = (
    opensearch_client.search(
        index=index_name,
        body=filtered_query
    )
)

Hybrid Retrieval

Hybrid retrieval combines lexical and vector signals. Lexical retrieval can preserve exact terms, while vector retrieval can identify semantically related content.

What Is Pinecone?

Pinecone is a managed vector-search platform. Applications can create an index, write records, organize them through namespaces, attach metadata, and retrieve similar records using vector or supported text-oriented search workflows.

Connect to Pinecone

import os

from pinecone import Pinecone


pinecone_client = Pinecone(
    api_key=os.environ[
        "PINECONE_API_KEY"
    ]
)

Create a Pinecone Vector Index

from pinecone import ServerlessSpec


pinecone_index_name = (
    "course-articles-v1"
)


existing_indexes = {
    index_description[
        "name"
    ]
    for index_description in (
        pinecone_client.list_indexes()
    )
}


if (
    pinecone_index_name
    not in existing_indexes
):
    pinecone_client.create_index(
        name=pinecone_index_name,
        dimension=dimension,
        metric="cosine",
        spec=ServerlessSpec(
            cloud="aws",
            region=(
                "replace-with-approved-region"
            )
        )
    )

Target the Pinecone Index

index_description = (
    pinecone_client.describe_index(
        pinecone_index_name
    )
)


pinecone_index = (
    pinecone_client.Index(
        host=index_description[
            "host"
        ]
    )
)

Upsert Vectors into Pinecone

pinecone_records = []


for position, (
    document_text,
    embedding
) in enumerate(
    zip(
        documents,
        document_embeddings
    ),
    start=1
):
    pinecone_records.append(
        {
            "id": (
                f"course-chunk-{position}"
            ),
            "values": embedding.tolist(),
            "metadata": {
                "document_id": (
                    "vector-database-lesson"
                ),
                "chunk_number": position,
                "chunk_text": document_text,
                "category": "generative-ai",
                "language": "en",
                "access_group": "learners",
                "embedding_model": (
                    embedding_model_id
                )
            }
        }
    )


pinecone_index.upsert(
    vectors=pinecone_records,
    namespace="machine-learning-course"
)

Query Pinecone

pinecone_response = (
    pinecone_index.query(
        namespace=(
            "machine-learning-course"
        ),
        vector=(
            query_embedding[
                0
            ].tolist()
        ),
        top_k=3,
        include_values=False,
        include_metadata=True
    )
)


for match in pinecone_response[
    "matches"
]:
    print(
        {
            "id": match[
                "id"
            ],
            "score": match[
                "score"
            ],
            "metadata": match[
                "metadata"
            ]
        }
    )

Pinecone Metadata Filtering

filtered_pinecone_response = (
    pinecone_index.query(
        namespace=(
            "machine-learning-course"
        ),
        vector=(
            query_embedding[
                0
            ].tolist()
        ),
        top_k=3,
        include_metadata=True,
        filter={
            "$and": [
                {
                    "language": {
                        "$eq": "en"
                    }
                },
                {
                    "access_group": {
                        "$eq": "learners"
                    }
                },
                {
                    "category": {
                        "$in": [
                            "generative-ai",
                            "mlops"
                        ]
                    }
                }
            ]
        }
    )
)

FAISS vs OpenSearch vs Pinecone

Vector Databases in RAG

Reranking

Reranking applies a second-stage model or scoring rule to the candidates returned by the initial retriever.

Evaluate Retrieval Quality

Calculate Retrieval Metrics

def precision_at_k(
    retrieved_ids,
    relevant_ids,
    k
):
    if k <= 0:

        raise ValueError(
            "k must be positive."
        )

    selected_ids = retrieved_ids[
        :k
    ]

    relevant_set = set(
        relevant_ids
    )

    relevant_count = sum(
        retrieved_id
        in relevant_set
        for retrieved_id in selected_ids
    )

    return (
        relevant_count
        /
        k
    )


def reciprocal_rank(
    retrieved_ids,
    relevant_ids
):
    relevant_set = set(
        relevant_ids
    )

    for rank, retrieved_id in enumerate(
        retrieved_ids,
        start=1
    ):
        if retrieved_id in relevant_set:

            return 1.0 / rank

    return 0.0

Build a Retrieval Evaluation Set

{
    "query_id": "query-001",
    "query": "How does model monitoring detect production degradation?",
    "relevant_chunk_ids": [
        "monitoring-definition",
        "delayed-quality",
        "drift-detection"
    ],
    "required_filters": {
        "language": "en",
        "access_group": "learners"
    }
}

Embedding and Index Versioning

Re-Embedding and Index Migration

Changing the embedding model can alter vector dimensions and the geometry of the embedding space. Existing embeddings cannot generally be mixed safely with embeddings from an incompatible model.

Updates and Deletions

Security and Access Control

Prompt Injection and Retrieved Content

Retrieved text is untrusted data. A document may contain malicious, misleading, obsolete, or instruction-like content intended to alter the behavior of the downstream model.

Vector Database Monitoring

Retrieval Failure Handling

Cost and Performance Optimization

Responsible Vector Retrieval

Common Vector Database Mistakes

Troubleshooting

Recommended Vector Database Workflow

Hands-On Practice

Knowledge Check

Vector Database Readiness Checklist

Lesson Summary