Table of Contents

Transfer Learning and Pretrained Models

Rumman Ansari September 15, 2026 42 views Subject Details

Deep Learning Fundamentals

Transfer Learning and Pretrained Models

Learn how to reuse knowledge from pretrained neural networks, replace task-specific output layers, freeze and unfreeze parameters, extract reusable features, fine-tune models safely, and evaluate whether transferred knowledge improves a new Machine Learning task.

Introduction

Training a large neural network from random initialization can require substantial data, computing resources, experimentation, and time. Many practical projects do not have enough labeled data to learn reliable representations from the beginning.

Transfer learning addresses this challenge by starting with a model that has already learned useful patterns from another dataset or task. The model's learned parameters are then reused, adapted, or partially retrained for a new target task.

In computer vision, early and intermediate layers of a pretrained network may recognize reusable visual patterns such as edges, textures, shapes, and object components. A new classification head can then learn how those representations relate to the classes in the target dataset.

Simple definition: Transfer learning reuses knowledge learned by one model as the starting point for a new but sufficiently related task.

Learning Objectives

By the End of This Lesson

Explain transfer learning and pretrained models
Describe why learned representations can be reusable
Distinguish source and target tasks
Compare training from scratch with transfer learning
Compare feature extraction with fine-tuning
Load pretrained TorchVision weights correctly
Use preprocessing associated with selected weights
Replace a pretrained model's classification head
Freeze and unfreeze model parameters
Verify which parameters remain trainable
Create an optimizer using only trainable parameters
Apply discriminative learning rates
Use gradual unfreezing
Recognize negative transfer and domain mismatch
Prevent leakage in transfer-learning workflows
Handle Batch Normalization and Dropout correctly
Save the best fine-tuned checkpoint
Evaluate target-task performance independently
Document model, weights, preprocessing, and licenses

Prerequisites

Recommended Knowledge

Python functions, loops, classes, and modules
PyTorch tensors and tensor shapes
torch.nn.Module
Forward propagation and backpropagation
Autograd and gradient calculation
Loss functions and optimizers
Dataset and DataLoader
Training and validation loops
Overfitting and underfitting
Classification metrics
Basic image preprocessing

Required Packages

PyTorch
TorchVision
Pillow
NumPy
Matplotlib

python -m pip install torch torchvision pillow numpy matplotlib

Environment requirement: Use compatible PyTorch and TorchVision versions. Verify the selected device, package versions, model-weight version, and preprocessing configuration before training.

Why Use Transfer Learning?

Potential Benefit	Explanation
Reusable Representations	The source model has already learned patterns that may be useful for the target task
Reduced Data Requirement	The target model does not need to learn every representation from random initialization
Faster Convergence	Training begins from meaningful parameter values rather than entirely random weights
Lower Compute Requirement	Only selected layers may need to be trained initially
Strong Baseline	A pretrained backbone can provide a practical reference before custom architecture work
Improved Generalization	Representations learned from diverse source data may reduce overfitting on a small target dataset

Important: These are potential benefits, not guarantees. Transfer learning must be evaluated against a suitable baseline on representative target-task data.

Source and Target Tasks

Source Task

Provides the original training experience
May contain a large and diverse dataset
Produces the pretrained parameters
May use different output classes from the target task

Target Task

Represents the new problem to solve
Uses the project's own labels and evaluation objective
May contain fewer labeled examples
Requires independent validation on its own data distribution

Transfer-Learning Flow

Source Dataset → Pretrained Parameters → Target-Specific Head → Target-Task Adaptation → Independent Evaluation

Train from Scratch vs Transfer Learning

Area	Training from Scratch	Transfer Learning
Initial Parameters	Generally randomly initialized	Loaded from source-task training
Representation Learning	Must be learned from the target training data	Begins with reusable learned representations
Target Data Need	Often higher	Can be lower when source and target are suitably related
Compute Requirement	Often higher	Can be lower when much of the backbone is frozen
Adaptation Risk	Overfitting or failure to learn useful features	Negative transfer or catastrophic forgetting
Dependency	Architecture and target data	Architecture, weights, source data, preprocessing, and target data

Good Experimental Practice Compare transfer learning with at least one reasonable target-task baseline under the same validation design.

Representation Hierarchy

A deep network transforms input data through several levels of representation. In a vision model, early layers often respond to relatively basic visual structures, while later layers combine those structures into more task-specific representations.

Simplified Vision Hierarchy

Pixels → Edges and Colors → Textures and Shapes → Object Components → Task Output

Transfer learning is useful when at least some of these learned representations remain informative for the target task.

Key principle: The closer the source data and task are to the target data and task, the more reusable the learned representation may be. This relationship must still be tested rather than assumed.

Two Main Transfer-Learning Strategies

Fixed Feature Extraction

Load pretrained parameters
Freeze the pretrained backbone
Replace the original output head
Train only the new target-specific head
Useful as a fast and controlled baseline

Fine-Tuning

Load pretrained parameters
Replace the task-specific output head
Train some or all pretrained layers
Commonly use a smaller learning rate for pretrained parameters
Allows representations to adapt to the target domain

Fixed Feature Extraction

In fixed feature extraction, the pretrained backbone acts as a reusable feature generator. Its parameters remain frozen while the newly added task head learns the target labels.

Feature-Extraction Model

\[ \hat{y} = h_{\theta_h} \left( g_{\theta_b}(x) \right) \]

Here, \(g_{\theta_b}\) is the frozen backbone and \(h_{\theta_h}\) is the trainable target-specific head.

Possible Advantages

Fewer trainable parameters
Lower memory requirement for gradients
Faster initial experimentation
Reduced risk of immediately destroying pretrained features
Useful starting point when target data is limited

Possible Limitations

The frozen representation may not suit the target domain
The new head may not compensate for major domain mismatch
Target-specific visual patterns may remain unlearned
Performance may plateau below a carefully fine-tuned model

Fine-Tuning

Fine-tuning updates selected pretrained parameters using target-task data. The objective is to preserve useful source knowledge while adapting the representation to the target domain.

Fine-Tuning Objective

\[ \theta^{*} = \arg\min_{\theta} \mathcal{L}_{\text{target}} \left( f_{\theta}(X), y \right) \]

Fine-Tuning May Help When

The target domain differs from the source domain
The target dataset contains enough representative examples
The fixed backbone underfits important target patterns
Independent validation supports further adaptation
Compute and memory resources permit additional training

Common Risk Unfreeze the complete network immediately and use the same large learning rate for every parameter.

Safer Starting Strategy Train the new head first, evaluate it, then gradually unfreeze selected backbone layers using appropriately smaller learning rates.

Choosing a Strategy

Target Situation	Reasonable Starting Strategy
Small target dataset and similar source domain	Freeze the backbone and train the new head
Moderate target dataset and related domain	Train the head, then fine-tune later backbone layers
Larger target dataset and meaningful domain difference	Consider broader fine-tuning with careful regularization
Severe source-target mismatch	Compare alternative pretraining, broader fine-tuning, and training-from-scratch baselines
Very limited compute	Use a frozen backbone or smaller pretrained architecture
High explainability or latency requirement	Evaluate whether the transferred architecture is deployable before extensive tuning

Decision rule: These are starting hypotheses rather than universal laws. Select the final strategy using target-task validation evidence.

Load a Pretrained TorchVision Model

TorchVision provides model architectures and associated pretrained weight definitions. Use an explicit weight version when reproducibility is important.

import torch

from torchvision.models import resnet18
from torchvision.models import ResNet18_Weights


selected_weights = (
    ResNet18_Weights.IMAGENET1K_V1
)


model = resnet18(
    weights=selected_weights
)


print(model)

Reproducibility Risk

model = resnet18(
    weights=ResNet18_Weights.DEFAULT
)

A default alias can resolve to a different weight version after a future library update.

Version-Controlled Approach Pin the PyTorch and TorchVision dependencies and record the explicit weight identifier used by the experiment.

Use the Weight-Specific Preprocessing

A pretrained model expects input data to be prepared consistently with the preprocessing used for its selected weights.

preprocess = selected_weights.transforms()


print(preprocess)

This transformation can then be applied during inference:

from PIL import Image


image = Image.open(
    "sample-image.jpg"
).convert(
    "RGB"
)


input_tensor = preprocess(
    image
)


input_batch = input_tensor.unsqueeze(
    dim=0
)


print(
    input_batch.shape
)

Common Mistake Use arbitrary image dimensions or normalization values that do not match the selected pretrained weights.

Correct Practice Retrieve and preserve the transformations associated with the exact weight version.

Training and Validation Transformations

Training transformations may include reasonable augmentation. Validation transformations should remain deterministic so that evaluation is stable and comparable.

from torchvision import transforms


train_transform = transforms.Compose([
    transforms.RandomResizedCrop(
        size=224
    ),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[
            0.485,
            0.456,
            0.406
        ],
        std=[
            0.229,
            0.224,
            0.225
        ]
    )
])


validation_transform = transforms.Compose([
    transforms.Resize(
        size=256
    ),
    transforms.CenterCrop(
        size=224
    ),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[
            0.485,
            0.456,
            0.406
        ],
        std=[
            0.229,
            0.224,
            0.225
        ]
    )
])

Important: The values above illustrate a commonly used preprocessing pattern. Confirm the requirements of the exact selected weight definition rather than copying transformations without verification.

Prepare an ImageFolder Dataset

A common directory structure for torchvision.datasets.ImageFolder is:

dataset/
    train/
        class_a/
            image_001.jpg
            image_002.jpg
        class_b/
            image_003.jpg
            image_004.jpg
    validation/
        class_a/
            image_101.jpg
        class_b/
            image_102.jpg
    test/
        class_a/
            image_201.jpg
        class_b/
            image_202.jpg

from pathlib import Path

from torch.utils.data import DataLoader
from torchvision.datasets import ImageFolder


dataset_root = Path(
    "dataset"
)


train_dataset = ImageFolder(
    root=dataset_root / "train",
    transform=train_transform
)


validation_dataset = ImageFolder(
    root=dataset_root / "validation",
    transform=validation_transform
)


train_loader = DataLoader(
    dataset=train_dataset,
    batch_size=32,
    shuffle=True,
    num_workers=2
)


validation_loader = DataLoader(
    dataset=validation_dataset,
    batch_size=64,
    shuffle=False,
    num_workers=2
)


print(
    train_dataset.class_to_idx
)

Label mapping: Preserve class_to_idx with the model artifacts. An incorrect or reordered mapping can silently produce incorrect class names during inference.

Split Images Without Leakage

Transfer learning does not remove the need for careful dataset splitting. Near-duplicate or related images across partitions can produce an unrealistically strong evaluation.

Leakage Risks

Different crops of one original image across partitions
Consecutive video frames split between training and validation
Several images of the same person across partitions
Several images of the same object or product across partitions
Images from the same patient, device, location, or session across partitions
Watermarks or folder names that reveal the class
Augmented copies created before the dataset split
Target-specific metadata encoded in filenames

Better Practice Group related observations before splitting and generate augmentation only inside the training pipeline.

Replace the Classification Head

A pretrained classifier's original output layer matches its source classes. Replace that layer so the model outputs one logit per target class.

from torch import nn


number_of_target_classes = len(
    train_dataset.classes
)


input_features = model.fc.in_features


model.fc = nn.Linear(
    in_features=input_features,
    out_features=number_of_target_classes
)


print(model.fc)

New Classification Head

\[ z = Wh+b \]

In this expression, \(h\) is the representation produced by the pretrained backbone, and \(z\) contains target-class logits.

Architecture note: Different model families store their task-specific heads under different attributes. Inspect the selected architecture before replacement.

Freeze the Backbone

for parameter in model.parameters():

    parameter.requires_grad = False


model.fc = nn.Linear(
    in_features=input_features,
    out_features=number_of_target_classes
)

The new head is created after the existing parameters are frozen. Its newly created parameters are trainable by default.

Verify Trainable Parameters

for name, parameter in model.named_parameters():

    print(
        name,
        parameter.requires_grad
    )

Expected Feature-Extraction State Backbone parameters are frozen and the replacement head parameters remain trainable.

Count Trainable Parameters

total_parameters = sum(
    parameter.numel()
    for parameter in model.parameters()
)


trainable_parameters = sum(
    parameter.numel()
    for parameter in model.parameters()
    if parameter.requires_grad
)


print(
    "Total parameters:",
    total_parameters
)

print(
    "Trainable parameters:",
    trainable_parameters
)

print(
    "Frozen parameters:",
    total_parameters
    - trainable_parameters
)

Verification rule: Count and log trainable parameters before every transfer-learning stage. This catches accidental freezing or unintended full-model training.

Create the Feature-Extraction Optimizer

from torch import optim


optimizer = optim.AdamW(
    params=[
        parameter
        for parameter in model.parameters()
        if parameter.requires_grad
    ],
    lr=0.001,
    weight_decay=0.0001
)

Common Mistake Create the optimizer before replacing or unfreezing layers, then assume the new trainable parameters were added automatically.

Better Approach Create or update the optimizer after finalizing the trainable parameter set for the current training stage.

Select the Device

import torch


if torch.cuda.is_available():

    device = torch.device(
        "cuda"
    )

elif hasattr(
    torch.backends,
    "mps"
) and torch.backends.mps.is_available():

    device = torch.device(
        "mps"
    )

else:

    device = torch.device(
        "cpu"
    )


model = model.to(
    device
)


print(
    "Selected device:",
    device
)

Choose the Loss Function

from torch import nn


loss_function = nn.CrossEntropyLoss()

Input	Expected Form
Model Output	Raw logits with shape `[batch_size, classes]`
Target	Integer class indices with shape `[batch_size]`
Target Data Type	Usually `torch.int64`

Common Mistake Apply softmax to the logits before passing them to CrossEntropyLoss.

Correct Training Pattern Pass raw logits to the loss function. Convert logits into probabilities only when they are needed for interpretation or downstream decisions.

Reusable Training Function

def train_one_epoch(
    model,
    data_loader,
    loss_function,
    optimizer,
    device
):
    model.train()

    total_loss = 0.0
    total_correct = 0
    total_examples = 0

    for images, targets in data_loader:

        images = images.to(
            device
        )

        targets = targets.to(
            device
        )

        optimizer.zero_grad(
            set_to_none=True
        )

        logits = model(
            images
        )

        loss = loss_function(
            logits,
            targets
        )

        loss.backward()

        optimizer.step()

        batch_size = images.size(
            0
        )

        total_loss += (
            loss.item()
            * batch_size
        )

        predictions = logits.argmax(
            dim=1
        )

        total_correct += (
            predictions
            == targets
        ).sum().item()

        total_examples += batch_size

    mean_loss = (
        total_loss
        / total_examples
    )

    accuracy = (
        total_correct
        / total_examples
    )

    return mean_loss, accuracy

Reusable Validation Function

def evaluate_model(
    model,
    data_loader,
    loss_function,
    device
):
    model.eval()

    total_loss = 0.0
    total_correct = 0
    total_examples = 0

    with torch.inference_mode():

        for images, targets in data_loader:

            images = images.to(
                device
            )

            targets = targets.to(
                device
            )

            logits = model(
                images
            )

            loss = loss_function(
                logits,
                targets
            )

            batch_size = images.size(
                0
            )

            total_loss += (
                loss.item()
                * batch_size
            )

            predictions = logits.argmax(
                dim=1
            )

            total_correct += (
                predictions
                == targets
            ).sum().item()

            total_examples += batch_size

    mean_loss = (
        total_loss
        / total_examples
    )

    accuracy = (
        total_correct
        / total_examples
    )

    return mean_loss, accuracy

Evaluation requirement: Accuracy is used here to demonstrate the loop. Add precision, recall, F1 score, a confusion matrix, class-specific metrics, and other measures according to the target problem.

Train the New Classification Head

number_of_epochs = 5


best_validation_loss = float(
    "inf"
)


for epoch in range(
    number_of_epochs
):

    train_loss, train_accuracy = train_one_epoch(
        model=model,
        data_loader=train_loader,
        loss_function=loss_function,
        optimizer=optimizer,
        device=device
    )

    validation_loss, validation_accuracy = evaluate_model(
        model=model,
        data_loader=validation_loader,
        loss_function=loss_function,
        device=device
    )

    print(
        f"Epoch {epoch + 1:02d} | "
        f"Train Loss: {train_loss:.4f} | "
        f"Train Accuracy: {train_accuracy:.4f} | "
        f"Validation Loss: {validation_loss:.4f} | "
        f"Validation Accuracy: {validation_accuracy:.4f}"
    )

    if validation_loss < best_validation_loss:

        best_validation_loss = validation_loss

        torch.save(
            {
                "model_state_dict": model.state_dict(),
                "class_to_idx": train_dataset.class_to_idx,
                "weight_name": "IMAGENET1K_V1",
                "number_of_classes": number_of_target_classes
            },
            "best_feature_extractor.pt"
        )

Unfreeze Selected Layers

After training the new head, later backbone layers can be unfrozen so they can adapt to the target task.

for parameter in model.layer4.parameters():

    parameter.requires_grad = True


for name, parameter in model.named_parameters():

    if parameter.requires_grad:

        print(
            "Trainable:",
            name
        )

Gradual unfreezing: Start with the layers closest to the output, evaluate the result, and unfreeze additional blocks only when target validation evidence supports more adaptation.

Use Discriminative Learning Rates

The newly initialized head may need larger updates than pretrained backbone layers. Parameter groups allow different learning rates.

optimizer = optim.AdamW(
    [
        {
            "params": model.layer4.parameters(),
            "lr": 0.00001
        },
        {
            "params": model.fc.parameters(),
            "lr": 0.0001
        }
    ],
    weight_decay=0.0001
)

Discriminative Update

\[ \theta_{b}^{(t+1)} = \theta_{b}^{(t)} - \eta_b \nabla_{\theta_b} \mathcal{L} \] \[ \theta_{h}^{(t+1)} = \theta_{h}^{(t)} - \eta_h \nabla_{\theta_h} \mathcal{L} \]

Here, \(\eta_b\) is the backbone learning rate and \(\eta_h\) is the head learning rate.

Typical objective: Protect useful pretrained representations with smaller updates while allowing the new head to learn the target classes more rapidly.

Progressive Fine-Tuning

Train the New Head

Freeze the backbone, train the target-specific head, and establish a validated feature-extraction baseline.

Unfreeze the Final Block

Enable training for the latest backbone block and use a smaller learning rate than the head.

Evaluate Generalization

Compare validation loss, class-specific metrics, and errors with the frozen-backbone result.

Unfreeze More Only if Needed

Adapt earlier blocks only when performance and domain evidence justify the additional flexibility.

Select the Best Checkpoint

Keep the stage and epoch with the best predefined validation result rather than automatically selecting the most extensively fine-tuned model.

Freezing Nuance: Parameters vs Module Behavior

Setting requires_grad=False prevents a parameter from receiving gradient-based updates. It does not automatically change the training behavior of Dropout or Batch Normalization modules.

Control	What It Changes
`requires_grad=False`	Prevents gradient calculation for the parameter
`model.train()`	Enables training behavior in modules such as Dropout and Batch Normalization
`model.eval()`	Enables evaluation behavior in those modules
`torch.inference_mode()`	Disables gradient recording for inference operations

Incorrect Assumption Freezing all backbone parameters automatically freezes every running statistic and training-time module behavior.

Correct Approach Decide explicitly how frozen modules containing Batch Normalization or Dropout should behave, and validate that decision for the chosen architecture and target dataset.

Domain Shift

Domain shift occurs when the source and target data distributions differ in ways that affect model behavior.

Difference	Example	Possible Effect
Capture Device	Professional camera vs industrial sensor	Different color, noise, or resolution patterns
Visual Style	Natural photographs vs medical scans	Source features may transfer weakly
Environment	Studio images vs field conditions	Background and lighting shortcuts may change
Population	Source population vs deployment population	Performance may vary across groups
Class Definition	Broad source categories vs fine-grained target categories	Later features may not separate the target classes
Time	Historical images vs new product generations	Previously useful patterns may become outdated

Validation requirement: A pretrained model must be evaluated on data that represents the target deployment environment. Source-dataset performance is not evidence of target-task reliability.

Negative Transfer

Negative transfer occurs when reused source knowledge harms target performance compared with a suitable alternative.

Warning Signs

A transferred model performs worse than a simpler baseline
The frozen backbone cannot separate target classes
Fine-tuning quickly overfits the target training set
Target predictions rely on source-specific artifacts
Performance varies severely across target subgroups
An architecture trained on a different modality or input structure is used without justification

How to Investigate Compare frozen features, partial fine-tuning, broader fine-tuning, alternative pretrained weights, smaller architectures, and a training-from-scratch baseline using the same validation design.

Catastrophic Forgetting

Catastrophic forgetting occurs when target-task updates overwrite useful pretrained representations too aggressively.

Risk-Reduction Strategies

Use a smaller learning rate for pretrained layers
Train the replacement head before unfreezing the backbone
Unfreeze layers gradually
Use regularization and appropriate weight decay
Monitor validation performance throughout adaptation
Stop when further unfreezing reduces generalization
Preserve the original pretrained checkpoint

Learning-Rate Scheduling

scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
    optimizer=optimizer,
    mode="min",
    factor=0.5,
    patience=2
)


for epoch in range(
    number_of_epochs
):

    train_loss, train_accuracy = train_one_epoch(
        model,
        train_loader,
        loss_function,
        optimizer,
        device
    )

    validation_loss, validation_accuracy = evaluate_model(
        model,
        validation_loader,
        loss_function,
        device
    )

    scheduler.step(
        validation_loss
    )

Scheduler note: Different schedulers require different stepping patterns. Confirm whether the scheduler should be advanced per batch, per epoch, or with a monitored validation value.

Early Stopping and Best Checkpoints

best_validation_loss = float(
    "inf"
)

patience = 5
epochs_without_improvement = 0


for epoch in range(
    number_of_epochs
):

    train_loss, train_accuracy = train_one_epoch(
        model,
        train_loader,
        loss_function,
        optimizer,
        device
    )

    validation_loss, validation_accuracy = evaluate_model(
        model,
        validation_loader,
        loss_function,
        device
    )

    if validation_loss < best_validation_loss:

        best_validation_loss = validation_loss

        epochs_without_improvement = 0

        torch.save(
            {
                "epoch": epoch,
                "model_state_dict": model.state_dict(),
                "optimizer_state_dict": optimizer.state_dict(),
                "class_to_idx": train_dataset.class_to_idx,
                "weight_name": "IMAGENET1K_V1",
                "number_of_classes": number_of_target_classes
            },
            "best_transfer_model.pt"
        )

    else:

        epochs_without_improvement += 1

    if epochs_without_improvement >= patience:

        print(
            "Early stopping activated."
        )

        break

Load the Best Model

checkpoint = torch.load(
    "best_transfer_model.pt",
    map_location=device
)


selected_weights = (
    ResNet18_Weights.IMAGENET1K_V1
)


loaded_model = resnet18(
    weights=None
)


loaded_input_features = (
    loaded_model.fc.in_features
)


loaded_model.fc = nn.Linear(
    in_features=loaded_input_features,
    out_features=checkpoint["number_of_classes"]
)


loaded_model.load_state_dict(
    checkpoint["model_state_dict"]
)


loaded_model = loaded_model.to(
    device
)


loaded_model.eval()

Artifact requirement: Loading the learned parameters is only one part of restoration. Preserve the architecture, class mapping, preprocessing, weight identity, package versions, and inference rules.

Run Inference on One Image

from PIL import Image


image = Image.open(
    "new-image.jpg"
).convert(
    "RGB"
)


input_tensor = preprocess(
    image
).unsqueeze(
    dim=0
).to(
    device
)


with torch.inference_mode():

    logits = loaded_model(
        input_tensor
    )

    probabilities = torch.softmax(
        logits,
        dim=1
    )

    predicted_index = probabilities.argmax(
        dim=1
    ).item()


index_to_class = {
    index: class_name
    for class_name, index
    in checkpoint["class_to_idx"].items()
}


predicted_class = index_to_class[
    predicted_index
]


predicted_probability = probabilities[
    0,
    predicted_index
].item()


print(
    "Predicted class:",
    predicted_class
)

print(
    "Predicted probability:",
    predicted_probability
)

Probability caution: Softmax produces normalized scores, but those scores are not automatically well calibrated. Evaluate probability calibration before using them as reliable confidence estimates.

Evaluate Beyond Accuracy

Evaluation Area	Questions to Ask
Class-Specific Performance	Which classes have weak precision or recall?
Confusion Matrix	Which classes are regularly confused?
Imbalance	Does overall accuracy hide minority-class failure?
Calibration	Do predicted probabilities match observed frequencies?
Subgroups	Does performance differ by device, location, population, or capture condition?
Robustness	How does performance change with lighting, noise, blur, crop, or compression?
Latency	Can the model meet serving requirements?
Memory	Can the model run within deployment limits?
Error Consequence	Which mistakes create the greatest practical harm?

Analyze Target-Task Errors

Review These Cases

High-confidence incorrect predictions
Low-confidence correct predictions
Minority-class examples
Images from new devices or locations
Low-quality or corrupted files
Ambiguous labels
Potential duplicates
Examples with unusual backgrounds
Cases near class boundaries
Errors with serious operational consequences

Diagnostic goal: Determine whether failure is caused by labels, preprocessing, insufficient target coverage, domain mismatch, architecture, optimization, or deployment conditions.

Bias and Responsible Reuse

A pretrained model can inherit limitations from its source dataset, labels, collection process, and objective. Fine-tuning does not automatically remove these limitations.

Responsible Review

Investigate the source dataset and intended use where information is available
Review licensing and model-use restrictions
Evaluate target performance across relevant groups
Inspect false positives and false negatives separately
Check whether images contain sensitive information
Review whether labels reflect historical bias
Document unsuitable or unsupported uses
Provide human oversight for consequential decisions
Monitor behavior after deployment

Incorrect Assumption A widely used pretrained model is automatically fair and suitable for every target population.

Correct Approach Treat the pretrained model as a candidate component that requires target-specific evaluation, risk review, and governance.

Licensing and Provenance

Item	What to Record
Architecture	Model family and exact variant
Weight Version	Exact pretrained weight identifier
Source	Repository, package, or approved model registry
License	Terms applying to code, weights, and associated data
Intended Use	Documented supported uses and known restrictions
Preprocessing	Resize, crop, interpolation, normalization, and input format
Dependencies	PyTorch and TorchVision versions
Target Adaptation	Replaced layers, frozen layers, fine-tuned layers, and learning rates

Usage responsibility: Verify that the architecture, pretrained weights, datasets, and dependencies can be used for the intended project and distribution context.

Offline and Restricted Environments

Loading pretrained weights may download an artifact when the required weight file is not already available locally. Restricted production or development environments may not permit runtime downloads.

Controlled Artifact Practice

Acquire weights through an approved process
Store them in a controlled artifact repository
Record the exact version and checksum
Scan artifacts according to organizational requirements
Test loading without internet access
Prevent unapproved model replacement
Restrict write access to production model artifacts

Efficiency Considerations

Technique	Potential Benefit	Review Requirement
Frozen Backbone	Reduces gradient computation and trainable parameters	Verify that Batch Normalization behavior is appropriate
Smaller Architecture	Reduces latency and memory	Compare target-task performance
Mixed Precision	May improve supported-device training efficiency	Check numerical stability and hardware compatibility
Cached Features	Avoids repeating frozen-backbone computation	Version the feature extractor and transformations
Larger Batch	May improve hardware utilization	Review memory, optimization, and generalization
Gradient Accumulation	Simulates a larger effective batch with limited memory	Scale loss and update boundaries correctly

Cache Frozen Features

If a backbone remains completely frozen and deterministic, its output features can be computed once and used to train a separate target classifier.

from torch import nn


feature_model = resnet18(
    weights=selected_weights
)


feature_model.fc = nn.Identity()


feature_model = feature_model.to(
    device
)


feature_model.eval()


cached_features = []
cached_targets = []


with torch.inference_mode():

    for images, targets in train_loader:

        images = images.to(
            device
        )

        features = feature_model(
            images
        )

        cached_features.append(
            features.cpu()
        )

        cached_targets.append(
            targets.clone()
        )


cached_features = torch.cat(
    cached_features,
    dim=0
)


cached_targets = torch.cat(
    cached_targets,
    dim=0
)


print(
    cached_features.shape
)

Important limitation: Caching removes the opportunity to generate new random augmentation results through the backbone during each epoch. Decide whether that trade-off is suitable for the experiment.

Common Transfer-Learning Mistakes

Mistake Use preprocessing that does not match the selected weights.

Better Approach Use or verify the transformations associated with the explicit weight version.

Mistake Forget to replace the original classification head.

Better Approach Replace the output layer so it matches the target classes.

Mistake Assume frozen parameters also place every module into evaluation behavior.

Better Approach Handle parameter gradients and module modes separately.

Mistake Use aggressive learning rates on all pretrained layers.

Better Approach Begin conservatively and validate discriminative learning rates.

Mistake Create an optimizer before adjusting trainable parameters.

Better Approach Recreate or update optimizer groups after freezing or unfreezing.

Mistake Duplicate related images across training and validation data.

Better Approach Group related source items before splitting.

Mistake Assume pretrained performance transfers automatically.

Better Approach Evaluate on representative target-task data and relevant subgroups.

Mistake Save weights without the class mapping or preprocessing.

Better Approach Save all artifacts required to reproduce target-task inference.

Recommended Transfer-Learning Workflow

Define the Target Task

Document inputs, classes, prediction point, expected users, metrics, and important error consequences.

Audit the Target Data

Review label quality, class balance, duplicates, sensitive content, related observations, and deployment coverage.

Select a Candidate Pretrained Model

Consider architecture, source-task relevance, input requirements, compute cost, latency, license, and provenance.

Reproduce the Input Preprocessing

Use the transformations associated with the exact selected weight version.

Replace the Task Head

Configure the output dimension and loss function for the new target.

Train a Frozen-Backbone Baseline

Train the new head and evaluate class-specific target-task performance.

Fine-Tune Gradually

Unfreeze later blocks, use smaller backbone learning rates, and compare with the frozen baseline.

Analyze Errors and Risks

Review domain shift, shortcuts, subgroup behavior, calibration, negative transfer, and important failures.

Select the Best Validated Model

Choose the model using predefined metrics, deployment constraints, and independent validation evidence.

Package and Monitor

Preserve architecture, weights, preprocessing, labels, dependencies, licenses, evaluation evidence, and monitoring requirements.

Hands-On Practice

Inspect Pretrained Weight Metadata

Load an explicit TorchVision weight definition and inspect its associated transformations and available metadata.

Create a Target Dataset

Organize an authorized dataset into training, validation, and test partitions with related images grouped appropriately.

Replace the Model Head

Modify a pretrained model so that its output dimension matches the target classes.

Train a Fixed Feature Extractor

Freeze the backbone, verify trainable parameters, and train only the replacement head.

Fine-Tune the Final Block

Unfreeze the final backbone block and apply a smaller learning rate than the task head.

Compare Strategies

Compare training from scratch, frozen feature extraction, and partial fine-tuning under the same data split and metrics.

Analyze Domain Errors

Review mistakes by class, capture condition, device, source, image quality, and relevant subgroup.

Restore the Best Checkpoint

Load the saved architecture state, class mapping, preprocessing, and model parameters in a clean inference workflow.

Knowledge Check

What is transfer learning?

Explain how knowledge from a source task is reused for a target task.

What is a pretrained model?

Explain the difference between a model architecture and a model containing learned parameters.

What is fixed feature extraction?

Explain the role of the frozen backbone and trainable task head.

What is fine-tuning?

Explain how selected pretrained parameters are adapted using target-task data.

Why replace the original output layer?

Explain how the source and target class definitions differ.

Why use a smaller learning rate for pretrained layers?

Explain how conservative updates can preserve reusable representations.

What is negative transfer?

Explain how source-task knowledge can reduce target-task performance.

What is catastrophic forgetting?

Explain how aggressive target updates can overwrite useful pretrained knowledge.

Why must preprocessing match the selected weights?

Explain how resize, crop, channel order, range, and normalization affect model inputs.

Why is target-task evaluation still required?

Explain why source performance does not establish target reliability.

Lesson Summary

Concept	Main Point
Transfer Learning	Reuses learned knowledge for a related target task
Pretrained Model	Combines a model architecture with parameters learned previously
Source Task	Original task used to learn the reusable parameters
Target Task	New problem receiving the transferred representation
Feature Extraction	Freezes the backbone and trains a target-specific head
Fine-Tuning	Updates selected pretrained parameters on target data
Gradual Unfreezing	Enables adaptation progressively from later layers
Discriminative Learning Rate	Uses different update sizes for backbone and head
Weight Transform	Defines preprocessing compatible with the selected weights
Domain Shift	Represents differences between source and target distributions
Negative Transfer	Occurs when transferred knowledge harms target performance
Catastrophic Forgetting	Occurs when target updates overwrite useful source knowledge
Class Mapping	Connects numerical output indices with target labels
Checkpoint	Stores the validated adapted model and restoration metadata
Responsible Reuse	Requires provenance, licensing, target evaluation, and risk review

Key Takeaway

Transfer learning can reduce the data and compute required to build an effective target model by reusing representations learned previously. A reliable workflow must use the correct preprocessing for the selected weights, replace the source-task head, verify frozen and trainable parameters, establish a fixed-feature baseline, fine-tune conservatively, evaluate domain shift and negative transfer, prevent dataset leakage, preserve the class mapping and model provenance, and select the final model using representative target-task validation rather than source reputation alone.

Get Machine Learning updates by email

Short, practical lessons and course updates from Rumman Ansari. No spam.