Introduction

A search engine cannot answer questions about content that it has not discovered, processed, and indexed.

Content sources
      |
      v
Discovery and crawling
      |
      v
Content extraction
      |
      v
Normalization and validation
      |
      v
Deduplication and chunking
      |
      v
Embedding and indexing
      |
      v
Search and retrieval

Crawling and ingestion are related but different responsibilities.

• Crawling discovers and fetches content from connected sources.
• Ingestion converts fetched content into validated, searchable records.

Prerequisites

Crawling vs Scraping vs Ingestion

End-to-End Architecture

Seeds and connectors
      |
      v
URL discovery
      |
      v
Crawl frontier
      |
      v
Fetcher
      |
      v
Raw-content store
      |
      v
Parser and extractor
      |
      v
Normalizer and validator
      |
      v
Deduplicator
      |
      v
Chunker
      |
      +-- Keyword indexing
      |
      +-- Embedding generation
              |
              v
        Vector indexing
              |
              v
        Search and RAG

Content Discovery

Discovery begins with one or more trusted seeds.

Common discovery sources include:

• Configured starting URLs
• XML sitemaps
• Links extracted from fetched pages
• RSS or Atom feeds
• CMS and repository connectors
• Object-storage notifications
• Database change feeds
• User-uploaded files

Seed URL
    |
    v
Fetch page
    |
    v
Extract allowed links
    |
    v
Normalize discovered URLs
    |
    v
Add unseen URLs
to crawl frontier

Crawl Frontier

The crawl frontier is the managed collection of URLs that are waiting to be fetched.

A useful frontier records:

• Normalized URL
• Source and domain
• Discovery time
• Priority
• Next eligible fetch time
• Attempt count
• Previous fetch status
• Content-change hints
• Applicable access policy

URL Normalization

The same logical resource can appear through several syntactically different URLs.

Possible forms:

https://example.test/course/42

https://example.test/course/42/

https://example.test/course/42?tracking=campaign

https://EXAMPLE.test/course/42

A normalization policy can consider:

• Protocol and hostname normalization
• Default ports
• Fragment removal
• Trailing-slash policy
• Query-parameter filtering
• Canonical URL declarations
• Redirect targets

Query parameters must not be removed blindly because some parameters identify genuinely different resources.

Robots Rules and Sitemaps

A crawler should evaluate the permissions and discovery guidance applicable to the source.

• robots.txt communicates crawler-access directives.
• sitemap.xml can provide a list of discoverable URLs.

Authorization to access content must come from the applicable legal, contractual, organizational, and technical permissions. A robots file is not a substitute for authentication or data-usage authorization.

Politeness and Rate Limiting

An uncontrolled crawler can overload the source system.

Before fetch:

- Check domain concurrency
- Check crawl delay
- Check recent failures
- Check server guidance
- Check retry budget
- Check authorization

Useful controls include:

• Per-domain request limits
• Per-host concurrency limits
• Connection and response timeouts
• Capped exponential backoff with jitter
• Respect for valid server retry guidance
• Circuit breakers for failing sources
• Bounded crawl queues

Fetching

The fetcher retrieves a resource and records enough evidence to process and audit it.

Useful fetch metadata includes:

• Requested URL
• Final URL after redirects
• HTTP status
• Content type
• Content length
• Response timestamp
• ETag or modification metadata when available
• Content hash
• Source identity

Static vs Dynamic Content

Parsing and Extraction

Parsing converts downloaded content into a structured representation.

Extracted elements can include:

• Page title and headings
• Main body text
• Tables and lists
• Links and canonical metadata
• Document properties
• Image captions and approved image-derived text
• Code blocks
• Language and publication metadata

The internal IAI4E_KB_Ingestion_Agent_Ingestion_Tutorial_1_1.pdf describes a multi-format ingestion service that processes PDF, Word, PowerPoint, HTML, Markdown, CSV, and Excel content through parsing, chunking, embedding, and indexing stages. 【1-73f1a5】

Content Cleaning

Extracted text can contain repeated navigation, cookie banners, headers, footers, menus, legal notices, advertisements, and hidden elements.

Raw page
      |
      v
Remove repeated boilerplate
      |
      v
Preserve meaningful structure
      |
      v
Normalize text and metadata
      |
      v
Produce canonical document

Cleaning must not remove meaningful warnings, qualifications, citations, or section boundaries.

Deduplication

Duplicate content wastes storage and can cause repeated or biased search results.

Chunking

Long documents are commonly divided into smaller retrieval units called chunks.

Document
      |
      +-- Chunk 1: Introduction
      +-- Chunk 2: Architecture
      +-- Chunk 3: Configuration
      +-- Chunk 4: Failure handling
      +-- Chunk 5: Troubleshooting

A chunk should preserve:

• Document ID
• Chunk ID
• Heading path
• Source URL or file reference
• Content position
• Permissions
• Language
• Version and ingestion timestamp

Chunk-size Trade-off

Embeddings and Indexing

Processed chunks can be indexed for keyword retrieval, vector retrieval, or hybrid retrieval.

Clean chunk
      |
      +-- Text and metadata
      |      |
      |      v
      |   Keyword index
      |
      +-- Embedding model
             |
             v
          Vector
             |
             v
        Vector index

The internal IAI4E_KB_Ingestion_Agent_Ingestion_Tutorial_1_1.pdf describes extraction, chunking, embedding, and indexing into Azure AI Search for downstream semantic vector search. 【1-73f1a5】

Full vs Incremental Ingestion

Incremental Recrawling

Incremental crawling avoids downloading and processing every document during every run.

Previously observed metadata
      |
      v
Check source for change
      |
      +-- Unchanged:
      |      skip expensive processing
      |
      +-- Changed:
             fetch, parse,
             validate, and reindex

Change signals can include:

• Source-provided modification metadata
• ETag or conditional-fetch metadata
• Content hash
• Repository change token
• Database Change Data Capture
• Object-storage event
• Explicit version number

Deleted and Revoked Content

An ingestion system must account for content that was removed, renamed, expired, or made inaccessible.

Previously indexed document
      |
      v
Source document deleted
or permission revoked
      |
      v
Deletion synchronization
      |
      v
Remove, deactivate,
or restrict indexed content

Failing to synchronize deletions can expose stale or unauthorized content in search results.

Security Trimming

Search results must not reveal content that the requesting user is not permitted to access.

Preserve:

• Source identity
• Tenant identity
• Document permissions
• Group or role restrictions
• Classification labels
• Retention and deletion policy
• Permission-change timestamps

Backpressure

Fetchers can retrieve content faster than parsers, embedding services, or search indexes can process it.

Crawl rate
      |
      v
Raw-document queue grows
      |
      v
Parser backlog grows
      |
      v
Embedding service throttles
      |
      v
Index freshness decreases

Useful controls include:

• Bounded crawl and ingestion queues
• Per-stage concurrency limits
• Batching
• Source rate limiting
• Embedding-service quotas
• Index-write backoff
• Priority for high-value or recently changed content
• Autoscaling within downstream capacity

Retries and DLQs

Ingestion stage fails
      |
      v
Classify failure
      |
      +-- Temporary:
      |      retry with backoff
      |
      +-- Permanent:
      |      route to DLQ
      |
      +-- Unknown:
             bounded retry
             then DLQ

Examples include:

• Retry a temporary source timeout.
• Retry temporary embedding throttling.
• Dead-letter a corrupt document.
• Dead-letter an unsupported schema after validation.
• Quarantine content that violates the ingestion policy.

Conceptual Pipeline Configuration

crawlingAndIngestion:
  discovery:
    seeds:
      - approved-source-root
    sitemapDiscovery: enabled
    linkDiscovery: enabled

  crawling:
    robotsPolicy: respected
    perHostConcurrency: approved-limit
    requestTimeout: approved-boundary
    retries:
      bounded: true
      backoff: exponential-with-jitter

  extraction:
    supportedTypes:
      - html
      - pdf
      - docx
      - pptx
      - markdown
      - csv
      - xlsx

  normalization:
    urlCanonicalization: enabled
    contentHashing: enabled
    exactDeduplication: enabled

  chunking:
    strategy: structure-aware
    maximumSize: approved-limit
    overlap: workload-specific

  indexing:
    keywordIndex: enabled
    vectorIndex: enabled
    preservePermissions: true

  failureHandling:
    deadLetterDestination: ingestion-failures

  observability:
    crawlRate: enabled
    fetchFailures: enabled
    ingestionLag: enabled
    indexingFailures: enabled
    freshness: enabled

Python Pipeline Skeleton

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class CrawledDocument:
    source_url: str
    content_type: str
    content: bytes
    content_hash: str


@dataclass(frozen=True)
class SearchChunk:
    document_id: str
    chunk_id: str
    text: str
    metadata: dict


def ingest_document(
    document: CrawledDocument
) -> Iterable[SearchChunk\]:
    parsed_document = parse_by_content_type(
        document.content_type,
        document.content,
    )

    normalized_document = normalize_document(
        parsed_document
    )

    validate_document(
        normalized_document
    )

    for chunk in create_structure_aware_chunks(
        normalized_document
    ):
        yield SearchChunk(
            document_id=normalized_document.document_id,
            chunk_id=chunk.chunk_id,
            text=chunk.text,
            metadata={
                "source_url": document.source_url,
                "heading_path": chunk.heading_path,
                "content_hash": document.content_hash,
            },
        )

This educational example omits access control, robots evaluation, network safety, retries, DLQs, malware scanning, parser isolation, embedding, persistence, deletion synchronization, and index-write guarantees.

Ingestion Tracking Table

CREATE TABLE ingestion_documents
(
    document_id VARCHAR(150) PRIMARY KEY,
    source_uri TEXT NOT NULL,
    source_type VARCHAR(50) NOT NULL,
    content_hash VARCHAR(150) NOT NULL,
    ingestion_status VARCHAR(30) NOT NULL,
    schema_version INTEGER NOT NULL,
    discovered_at TIMESTAMP NOT NULL,
    fetched_at TIMESTAMP NULL,
    indexed_at TIMESTAMP NULL,
    deleted_at TIMESTAMP NULL,
    failure_code VARCHAR(100) NULL
);

Freshness

Search freshness is the delay between a source change and the corresponding searchable index update.

A simplified measure is:

\[ FreshnessLag = IndexedAt - SourceChangedAt \]

When a reliable source-change timestamp is unavailable, freshness must be estimated using available discovery, fetch, and index timestamps.

Learning-platform Example

Sources:

Course articles
Tutorial pages
PDF notes
Code examples
Quiz explanations


Pipeline:

Discover approved content
      |
      v
Fetch changed documents
      |
      v
Extract title, headings, and body
      |
      v
Remove navigation and repeated footer
      |
      v
Create section-aware chunks
      |
      v
Generate keyword and vector records
      |
      v
Update searchable index

Observability

Useful metrics include:

• URLs discovered
• URLs pending in the frontier
• Fetch success and failure rate
• Responses by status and content type
• Bytes downloaded
• Robots or policy exclusions
• Parse failures by file type
• Documents and chunks created
• Exact and near duplicates
• Embedding requests and failures
• Index writes and failures
• Ingestion backlog and oldest-item age
• Source-to-index freshness
• Deleted-document synchronization lag
• DLQ depth and oldest failure age

Alert Conditions

• Frontier depth continues growing
• One source returns repeated failures
• Parse failures increase for one format
• Embedding or indexing becomes throttled
• Freshness lag exceeds the search objective
• Deletion synchronization stops progressing
• Duplicate-content rate increases unexpectedly
• One source generates unlimited URL variations
• DLQ depth or failure age increases
• Indexed permissions differ from source permissions

Common Mistakes

Recommended Test Cases

Best Practices

Frequently Asked Questions