Introduction

Traditional language models process text. Traditional computer vision models process images. Many real-world tasks require both. A support engineer may need to analyze a screenshot together with a defect report. A learner may ask a question about a diagram. A business user may need information extracted from a chart, scanned form, receipt, or presentation slide.

A Vision-Language Model, commonly abbreviated as VLM, combines visual information with natural language. Depending on its architecture and training, a VLM may accept an image and text prompt and generate a textual answer, caption, classification, structured record, or grounded description.

Vision-language capability does not mean that a model sees exactly as a human does. Images are resized, normalized, divided into patches or regions, and converted into learned representations. These visual representations are aligned or integrated with the language model so that the system can generate text conditioned on visual content.

Learning Objectives

Prerequisites

python -m pip install \
  torch \
  transformers \
  accelerate \
  pillow \
  requests \
  scikit-learn

What Is a Modality?

A modality is a form in which information is expressed or observed. Multimodal systems process two or more modalities.

What Is Multimodal AI?

Multimodal AI processes information from multiple modalities and learns relationships across them.

What Is a Vision-Language Model?

A Vision-Language Model is a multimodal model designed to process visual and textual information together. Some VLMs focus on matching images and text. Others generate language conditioned on one or more images.

Simplified VLM Architecture

Image Representation

A digital image can be represented as a tensor containing height, width, and channel dimensions.

The image processor may resize the input, normalize pixel values, convert color channels, and create the tensor format expected by the selected model.

Image Patches and Visual Tokens

A Vision Transformer commonly divides an image into fixed-size patches. Each patch is flattened and projected into an embedding.

Here, \(P\) is the patch height and width when square patches are used.

\(\mathbf{x}_i\) is a flattened image patch, \(E\) is a learned projection, and \(\mathbf{p}_i\) is positional information.

Cross-Modal Alignment

Cross-modal alignment places related visual and textual concepts into representations that can be compared or combined.

The approximation indicates that a matching image \(I\) and text description \(T\) should have compatible or nearby representations under the trained alignment objective.

Contrastive Image-Text Learning

Contrastive learning teaches a model to assign higher similarity to matching image-text pairs and lower similarity to non-matching pairs.

\(B\) is the batch size, \(s_{ii}\) is the similarity of a matching image-text pair, \(s_{ij}\) represents comparisons with other text entries, and \(\tau\) is a temperature parameter.

Dual Encoder vs Generative VLM

Cross-Attention

Cross-attention allows one representation stream to attend to another. In a VLM, textual queries may attend to visual keys and values.

Common Vision-Language Tasks

Image Captioning

Image captioning produces a textual description of visual content. The desired level of detail depends on the use case.

Visual Question Answering

Visual Question Answering, or VQA, provides an image and a natural-language question and asks the model to answer using visual evidence.

\(I\) is the image, \(Q\) is the question, and \(\hat{y}\) is the generated answer.

Document Understanding

Document images may contain text, tables, logos, signatures, diagrams, handwriting, and spatial layouts. VLMs can analyze these elements together, but specialized OCR or document parsers may remain valuable.

OCR vs Vision-Language Models

Chart and Diagram Understanding

Visual Grounding

Visual grounding connects a text phrase with a location or region in an image.

Load and Validate an Image

from pathlib import Path

from PIL import Image
from PIL import UnidentifiedImageError


def load_validated_image(
    image_path,
    maximum_pixels=20_000_000
):
    image_path = Path(
        image_path
    )

    if not image_path.is_file():

        raise FileNotFoundError(
            f"Image not found: {image_path}"
        )

    try:
        with Image.open(
            image_path
        ) as opened_image:

            opened_image.verify()

        image = Image.open(
            image_path
        ).convert(
            "RGB"
        )

    except UnidentifiedImageError as error:

        raise ValueError(
            "Unsupported or invalid image."
        ) from error

    width, height = image.size

    if (
        width
        *
        height
        >
        maximum_pixels
    ):
        raise ValueError(
            "Image exceeds the approved "
            "pixel limit."
        )

    return image

Resize Images Safely

from PIL import Image


def resize_with_aspect_ratio(
    image,
    maximum_width,
    maximum_height
):
    if (
        maximum_width <= 0
        or maximum_height <= 0
    ):
        raise ValueError(
            "Maximum dimensions must "
            "be positive."
        )

    resized_image = image.copy()

    resized_image.thumbnail(
        (
            maximum_width,
            maximum_height
        ),
        Image.Resampling.LANCZOS
    )

    return resized_image

Load a Vision-Language Model

import torch
from transformers import AutoProcessor
from transformers import AutoModelForVision2Seq


model_id = (
    "replace-with-approved-vision-language-model"
)


processor = AutoProcessor.from_pretrained(
    model_id
)


model = (
    AutoModelForVision2Seq
    .from_pretrained(
        model_id,
        torch_dtype=(
            torch.bfloat16
            if torch.cuda.is_available()
            and torch.cuda.is_bf16_supported()
            else torch.float16
        ),
        device_map="auto"
    )
)


model.eval()

Create a Multimodal Prompt

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": image
            },
            {
                "type": "text",
                "text": (
                    "Describe only the visible "
                    "content. If a detail cannot "
                    "be verified from the image, "
                    "state that it is unclear."
                )
            }
        ]
    }
]


prompt = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

Run Vision-Language Inference

model_inputs = processor(
    text=[
        prompt
    ],
    images=[
        image
    ],
    return_tensors="pt"
)


model_inputs = {
    key: value.to(
        model.device
    )
    if hasattr(
        value,
        "to"
    )
    else value
    for key, value
    in model_inputs.items()
}


with torch.inference_mode():

    generated_ids = model.generate(
        **model_inputs,
        max_new_tokens=256,
        do_sample=False
    )


generated_text = processor.batch_decode(
    generated_ids,
    skip_special_tokens=True
)[
    0
]


print(
    generated_text
)

Request Structured Extraction

Analyze the supplied document image.

Return only valid JSON using this schema:

{
  "document_type": "",
  "visible_fields": [
    {
      "field_name": "",
      "value": "",
      "evidence": ""
    }
  ],
  "unclear_fields": [],
  "warnings": []
}

Rules:
1. Use only information visible in the image.
2. Do not guess missing or unreadable values.
3. Put uncertain fields in unclear_fields.
4. Do not invent document identifiers.
5. Preserve the visible wording.

Validate Structured VLM Output

import json

from pydantic import BaseModel
from pydantic import Field
from pydantic import ValidationError


class ExtractedField(
    BaseModel
):
    field_name: str = Field(
        min_length=1
    )

    value: str

    evidence: str


class DocumentExtraction(
    BaseModel
):
    document_type: str

    visible_fields: list[
        ExtractedField
    ]

    unclear_fields: list[
        str
    ]

    warnings: list[
        str
    ]


def validate_vlm_json(
    generated_output
):
    try:
        parsed_output = json.loads(
            generated_output
        )

        validated_output = (
            DocumentExtraction
            .model_validate(
                parsed_output
            )
        )

        return {
            "valid": True,
            "data": (
                validated_output.model_dump()
            ),
            "error": None
        }

    except (
        json.JSONDecodeError,
        ValidationError
    ) as error:

        return {
            "valid": False,
            "data": None,
            "error": str(
                error
            )
        }

Multiple-Image Reasoning

Some VLMs can receive multiple images in a single request. This can support comparison, change detection, visual sequence analysis, and cross-document reasoning.

Memory and Compute Requirements

VLM inference uses memory for model weights, visual representations, text tokens, key-value cache, generation, and temporary operations.

VLM Evaluation Dimensions

Build a VLM Evaluation Dataset

{
  "case_id": "chart-001",
  "image_id": "approved-image-reference",
  "question": "Which category has the highest visible value?",
  "reference_answer": "Category B",
  "required_evidence": [
    "Category labels",
    "Visible bar heights"
  ],
  "task": "chart-question-answering",
  "difficulty": "medium",
  "language": "en",
  "risk_level": "standard",
  "expected_behavior": {
    "must_use_visual_evidence": true,
    "must_not_invent_exact_values": true
  }
}

Evaluate Grounding with Intersection over Union

Intersection over Union, or IoU, measures overlap between a predicted region and a reference region.

def calculate_iou(
    predicted_box,
    reference_box
):
    pred_x1, pred_y1, pred_x2, pred_y2 = (
        predicted_box
    )

    ref_x1, ref_y1, ref_x2, ref_y2 = (
        reference_box
    )

    intersection_x1 = max(
        pred_x1,
        ref_x1
    )

    intersection_y1 = max(
        pred_y1,
        ref_y1
    )

    intersection_x2 = min(
        pred_x2,
        ref_x2
    )

    intersection_y2 = min(
        pred_y2,
        ref_y2
    )

    intersection_width = max(
        0,
        intersection_x2
        -
        intersection_x1
    )

    intersection_height = max(
        0,
        intersection_y2
        -
        intersection_y1
    )

    intersection_area = (
        intersection_width
        *
        intersection_height
    )

    predicted_area = max(
        0,
        pred_x2
        -
        pred_x1
    ) * max(
        0,
        pred_y2
        -
        pred_y1
    )

    reference_area = max(
        0,
        ref_x2
        -
        ref_x1
    ) * max(
        0,
        ref_y2
        -
        ref_y1
    )

    union_area = (
        predicted_area
        +
        reference_area
        -
        intersection_area
    )

    if union_area == 0:

        raise ValueError(
            "Bounding boxes must have "
            "a positive union area."
        )

    return (
        intersection_area
        /
        union_area
    )

Evaluate Extracted Fields

def calculate_field_metrics(
    predicted_fields,
    reference_fields
):
    predicted_keys = set(
        predicted_fields
    )

    reference_keys = set(
        reference_fields
    )

    correctly_extracted = {
        key
        for key in (
            predicted_keys
            &
            reference_keys
        )
        if (
            predicted_fields[
                key
            ]
            ==
            reference_fields[
                key
            ]
        )
    }

    precision = (
        len(
            correctly_extracted
        )
        /
        len(
            predicted_keys
        )
        if predicted_keys
        else 0.0
    )

    recall = (
        len(
            correctly_extracted
        )
        /
        len(
            reference_keys
        )
        if reference_keys
        else 0.0
    )

    f1_score = (
        2
        *
        precision
        *
        recall
        /
        (
            precision
            +
            recall
        )
        if (
            precision
            +
            recall
        )
        else 0.0
    )

    return {
        "precision": precision,
        "recall": recall,
        "f1": f1_score,
        "correct_fields": sorted(
            correctly_extracted
        )
    }

Visual Hallucination

Visual hallucination occurs when a model claims that an object, attribute, relationship, text, count, or event is visible even though the image does not support the claim.

Adversarial Images and Prompt Injection

Images can contain text designed to influence the model, such as an instruction embedded inside a screenshot, document, QR code, diagram, or visual overlay. That text is untrusted content.

Privacy and Sensitive Visual Data

Accessibility

Vision-language systems can assist with image descriptions and document navigation, but accessibility output must be evaluated for correctness, relevance, clarity, and respect.

Production Monitoring

Responsible VLM Use

Common VLM Mistakes

Troubleshooting

Recommended VLM Workflow

Hands-On Practice

Knowledge Check

VLM Readiness Checklist

Lesson Summary