Introduction

File upload looks simple from a user's perspective. A user selects a file, clicks an upload button, and waits for completion.

Behind that interface, a production upload system must handle authentication, authorization, metadata, content validation, size limits, unreliable networks, large files, retries, duplicate requests, checksums, storage failures, abandoned uploads, and database synchronization.

A small file can often be uploaded in one request. A large file is commonly divided into smaller parts that can be transferred independently and assembled into one final object.

This process is called multipart transfer or multipart upload.

In your System Design curriculum, Uploads and Multipart Transfer is Topic 6.5 under Storage, Files, Objects & Search Basics. It follows metadata and precedes inverted indexes and full-text search.

Prerequisites

What Is a File Upload?

A file upload transfers content from a client to an application or storage service.

User selects file
      |
      v
Client validates basic requirements
      |
      v
Client requests upload authorization
      |
      v
File bytes are transferred
      |
      v
Server or storage verifies content
      |
      v
Business metadata is finalized
      |
      v
Content becomes available

The client can be a web browser, mobile application, desktop application, command-line tool, batch process, or another service.

Upload Workflow Stages

Single-request Upload

A single-request upload sends the complete file in one request.

Complete file
      |
      v
One upload request
      |
      v
Application or storage receives all bytes
      |
      v
Object is created

Suitable Characteristics

• The file is relatively small
• The network connection is stable
• The file can be retried completely
• The request remains within proxy and server limits
• The server can stream the request safely

Limitations

• A late network failure can require retransmitting the complete file
• Large requests can exceed gateway or server limits
• Long request durations can trigger timeouts
• Loading the complete body into memory can exhaust application resources
• Resume support is difficult without a separate resumable protocol

Conceptual Single-object Upload

PUT /objects/tenants/17/assets/981/video.mp4 HTTP/1.1
Host: storage.example.com
Content-Type: video/mp4
Content-Length: 52428800
X-Checksum-Algorithm: SHA-256
X-Checksum-Value: expected-checksum-value

[BINARY CONTENT]

Header names, supported checksums, authentication, object-key rules, and maximum request size depend on the selected storage API.

What Is Multipart Upload?

Multipart upload splits one logical file or object into several contiguous parts. The parts are transferred independently and later assembled into one final object.

Large object

+-----------+-----------+-----------+-----------+
|  Part 1   |  Part 2   |  Part 3   |  Part 4   |
+-----------+-----------+-----------+-----------+
      |           |           |           |
      v           v           v           v
   Upload      Upload      Upload      Upload
      |           |           |           |
      +-----------+-----------+-----------+
                          |
                          v
                 Complete multipart upload
                          |
                          v
                    Final object

Parts can often be uploaded independently and in parallel. If one part fails, the client can retry that part rather than restarting the entire object upload.

Multipart-upload Lifecycle

1. Initiate the multipart upload.
2. Receive an upload-session identifier.
3. Split the file into numbered parts.
4. Upload each part.
5. Record each successful part's identifier and integrity metadata.
6. Retry failed parts.
7. Submit the ordered uploaded-part list.
8. Complete the multipart upload.
9. Verify the final object.
10. Finalize the application metadata.

Initiate
    |
    v
Receive upload ID
    |
    v
Upload numbered parts
    |
    v
Record successful part results
    |
    v
Complete upload
    |
    v
Storage assembles final object
    |
    v
Verify and publish

Initiate the Upload

The initiation request creates an upload session associated with the target object.

The request can establish:

• Target object key
• Expected content type
• Expected content length
• Checksum algorithm
• Encryption settings
• Storage class
• Retention configuration
• Custom metadata
• Tenant and asset identifiers

Application Initiation Request

POST /api/v1/assets/multipart-uploads HTTP/1.1
Content-Type: application/json
Authorization: Bearer access-token
Idempotency-Key: upload-request-981

{
  "courseId": 42,
  "lessonId": 7,
  "fileName": "system-design-introduction.mp4",
  "contentType": "video/mp4",
  "contentLength": 524288000,
  "checksumAlgorithm": "SHA-256",
  "completeObjectChecksum": "expected-checksum-value"
}

Application Initiation Response

{
  "assetId": 981,
  "uploadSessionId": "upload-session-id",
  "objectKey": "tenants/17/courses/42/lessons/7/assets/981/video.mp4",
  "partSize": 8388608,
  "maximumParallelParts": 4,
  "expiresAt": "upload-session-expiry",
  "status": "initiated"
}

The exact part size, parallelism, session lifetime, and authorization model should be selected by trusted server-side policy.

Upload-session Identifier

The storage service returns an upload identifier that associates later part, listing, completion, and abort operations with one multipart upload.

Upload session:

upload_id = unique provider upload identifier


Required for:

- Upload part
- List uploaded parts
- Complete upload
- Abort upload

Treat the upload identifier as sensitive workflow state. A caller should not be able to use another tenant's upload identifier.

Splitting a File into Parts

Let \(S\) be the complete file size in bytes and \(P\) be the selected part size.

The required number of parts can be calculated as:

\[ NumberOfParts = \left\lceil \frac{S}{P} \right\rceil \]

Example

File size:

100 MiB


Part size:

8 MiB


Number of parts:

ceil(100 / 8) = 13


Parts 1 through 12:

8 MiB each


Final part:

4 MiB

Provider-specific minimum, maximum, and part-count rules must be applied when selecting the part size.

Choosing Part Size

Part-size selection should consider:

• Total file size
• Provider part-count limits
• Minimum part size
• Network quality
• Expected concurrency
• Client memory
• Request overhead
• Retry cost
• Checksum-processing cost

Upload a Part

Every part request identifies the upload session and the part's position in the final object.

PUT /storage/object-key?uploadId=upload-session-id&partNumber=3 HTTP/1.1
Content-Type: application/octet-stream
Content-Length: 8388608
X-Part-Checksum: expected-part-checksum

[BINARY PART CONTENT]

The storage response can include a part identifier, ETag, or checksum that must be retained for completion.

{
  "partNumber": 3,
  "etag": "provider-part-identifier",
  "checksum": "verified-part-checksum",
  "size": 8388608
}

Part Metadata

The client or application should retain the metadata returned for every successfully uploaded part.

Parallel Part Uploads

Independent parts can often be uploaded concurrently to improve throughput.

Upload queue
    |
    +-- Worker 1 uploads Part 1
    |
    +-- Worker 2 uploads Part 2
    |
    +-- Worker 3 uploads Part 3
    |
    +-- Worker 4 uploads Part 4
    |
    v
Each completed worker takes next part

Parallelism should be bounded. Excessive concurrency can overload the client's CPU, memory, network, storage service, or application-control API.

Parallelism Trade-offs

Retry Failed Parts

A failed part can be retried independently.

Upload Part 7
      |
      v
Request fails
      |
      v
Classify failure
      |
      +-- Permanent:
      |      stop and report failure
      |
      +-- Transient:
             wait with backoff and jitter
             resend Part 7
             retain successful other parts

The client should not resend every completed part after one transient part failure.

Retries should use:

• Bounded attempts
• Exponential backoff
• Randomized jitter
• Request deadlines
• Checksum verification
• Safe replacement semantics for the same part number

Part Idempotency

A part retry can occur after the storage service accepted the bytes but the response was lost.

Client uploads Part 5
      |
      v
Storage stores Part 5
      |
      v
Network loses response
      |
      v
Client does not know result
      |
      v
Client retries Part 5

The provider's part-replacement semantics determine how the repeated upload behaves. The client should use the same upload identifier and intended part number and then retain the result from the successful retry.

Pause and Resume

Multipart upload can support pausing and resuming because completed parts remain associated with the upload session until it is completed, aborted, or removed by policy.

Upload begins

Parts completed:
1, 2, 3, 4


Client closes


Later:

Load upload-session state
List or verify completed parts
Resume with Parts 5 onward
Complete final object

A resumable design needs durable client-side or server-side upload-session metadata.

Upload-session Table

CREATE TABLE asset_upload_sessions
(
    upload_session_id VARCHAR(200) PRIMARY KEY,
    asset_id BIGINT NOT NULL,
    tenant_id BIGINT NOT NULL,
    object_key VARCHAR(500) NOT NULL,
    provider_upload_id VARCHAR(500) NOT NULL,

    expected_content_length BIGINT NOT NULL,
    expected_checksum_algorithm VARCHAR(30) NOT NULL,
    expected_checksum_value VARCHAR(200) NOT NULL,

    part_size BIGINT NOT NULL,
    upload_status VARCHAR(30) NOT NULL,

    created_at TIMESTAMP NOT NULL,
    expires_at TIMESTAMP NOT NULL,
    completed_at TIMESTAMP NULL,
    aborted_at TIMESTAMP NULL,

    CONSTRAINT fk_asset_upload_session_asset
        FOREIGN KEY (asset_id)
        REFERENCES course_assets (asset_id),

    CONSTRAINT uq_asset_active_upload
        UNIQUE
        (
            asset_id,
            upload_session_id
        ),

    CONSTRAINT ck_upload_expected_length
        CHECK (expected_content_length > 0),

    CONSTRAINT ck_upload_part_size
        CHECK (part_size > 0),

    CONSTRAINT ck_upload_status
        CHECK
        (
            upload_status IN
            (
                'initiated',
                'uploading',
                'completing',
                'completed',
                'failed',
                'expired',
                'aborted'
            )
        )
);

Uploaded-part Table

CREATE TABLE asset_upload_parts
(
    upload_session_id VARCHAR(200) NOT NULL,
    part_number INT NOT NULL,
    part_size BIGINT NOT NULL,
    provider_etag VARCHAR(500) NOT NULL,
    checksum_algorithm VARCHAR(30) NULL,
    checksum_value VARCHAR(200) NULL,
    uploaded_at TIMESTAMP NOT NULL,

    PRIMARY KEY
    (
        upload_session_id,
        part_number
    ),

    CONSTRAINT fk_asset_upload_parts_session
        FOREIGN KEY (upload_session_id)
        REFERENCES asset_upload_sessions
        (
            upload_session_id
        )
        ON DELETE CASCADE,

    CONSTRAINT ck_upload_part_number
        CHECK (part_number > 0),

    CONSTRAINT ck_upload_part_length
        CHECK (part_size > 0)
);

List Uploaded Parts

A resume workflow can request the parts already stored for one upload session.

GET /api/v1/assets/981/multipart-uploads/upload-session-id/parts HTTP/1.1
Authorization: Bearer access-token

{
  "uploadSessionId": "upload-session-id",
  "status": "uploading",
  "parts": [
    {
      "partNumber": 1,
      "size": 8388608,
      "etag": "part-1-identifier"
    },
    {
      "partNumber": 2,
      "size": 8388608,
      "etag": "part-2-identifier"
    }
  ]
}

The application should verify that the caller owns or is authorized for the upload rather than accepting an upload identifier as sufficient permission.

Complete the Multipart Upload

After every required part is uploaded successfully, the client submits the ordered part information to complete the upload.

POST /api/v1/assets/981/multipart-uploads/upload-session-id/complete HTTP/1.1
Content-Type: application/json
Authorization: Bearer access-token
Idempotency-Key: complete-upload-981

{
  "parts": [
    {
      "partNumber": 1,
      "etag": "part-1-identifier"
    },
    {
      "partNumber": 2,
      "etag": "part-2-identifier"
    },
    {
      "partNumber": 3,
      "etag": "part-3-identifier"
    }
  ],
  "completeObjectChecksum": "expected-checksum-value"
}

The storage service assembles the final object according to the part ordering required by its API.

Completion Validation

Before final completion, verify:

• The upload session exists
• The caller is authorized
• The upload has not expired or been aborted
• Part numbers are valid
• Required part identifiers are present
• The final expected size is valid
• The full-object checksum is supplied when required
• The completion request has not already been finalized differently

Post-completion Verification

A successful completion response should be followed by application-level verification before publication.

Storage reports completion
      |
      v
Retrieve final object properties
      |
      v
Compare object key
      |
      v
Compare content length
      |
      v
Compare full-object checksum
      |
      v
Verify object version
      |
      +-- All checks pass:
      |      finalize metadata
      |
      +-- Any check fails:
             mark failed
             quarantine or remove object
             investigate

Abort a Multipart Upload

An aborted upload releases the incomplete upload session and its stored parts according to the provider contract.

DELETE /api/v1/assets/981/multipart-uploads/upload-session-id HTTP/1.1
Authorization: Bearer access-token

An abort can occur when:

• The user cancels the upload
• The upload session expires
• Validation fails
• Authorization is revoked
• The target asset is deleted
• The client cannot continue after repeated failures
• A conflicting finalized upload already exists

Abort operations should be idempotent where practical. Repeating an abort should not recreate the session or produce a different business effect.

Incomplete-upload Cleanup

Uploaded parts can consume storage and request cost even though no final object was created.

Initiated upload
      |
      v
Several parts uploaded
      |
      v
Client disappears
      |
      v
Upload never completed or aborted
      |
      v
Parts continue consuming storage

A cleanup process should:

1. Find sessions beyond their approved expiry time.
2. Confirm that they are not actively completing.
3. Abort the provider multipart upload.
4. Mark the application session expired or aborted.
5. Remove temporary resources.
6. Record cleanup results.

Storage lifecycle rules can provide a second line of defense for abandoned multipart parts.

Idempotent Upload Initiation

A client can repeat the initiation request when a response is lost.

Client initiates upload
      |
      v
Application creates metadata
and provider upload session
      |
      v
Response is lost
      |
      v
Client retries initiation

Without idempotency, the retry can create multiple pending assets and multiple provider upload sessions.

A scoped idempotency record can associate one logical upload request with one asset and upload session.

Idempotent Completion

The complete request can also be repeated after the storage service created the final object but the application did not receive the response.

Complete request sent
      |
      v
Final object created
      |
      v
Response lost
      |
      v
Client retries completion
      |
      v
Application retrieves known outcome
instead of creating another asset

Store the completion outcome and final object version so repeated requests can retrieve the same logical result.

Secure Upload Validation

Upload security requires several independent controls.

• Authenticate the caller
• Authorize the target tenant and resource
• Limit file size
• Limit upload frequency and concurrency
• Validate the allowed content category
• Inspect the actual content type
• Generate the storage key server-side
• Validate the original filename separately
• Verify content length and checksum
• Scan or analyze content where required
• Keep new content private until approved
• Protect processing and administrative endpoints

Filename Security

The original filename is untrusted metadata and should not determine the physical or object-storage location.

Storage path:

/uploads/{client-supplied-file-name}

Object key:

tenants/{trustedTenantId}/assets/{generatedAssetId}/original


Display filename:

Validated separately and stored as metadata

Content-type Validation

A filename extension and browser-provided content type can be incorrect or intentionally misleading.

Client claims:

image/jpeg


Actual content:

Executable, archive,
or unsupported file format

Validate content using an approved server-side process. For images or documents, format-aware parsing or rewriting can provide stronger validation than trusting headers alone.

Compressed-upload Risks

Archive uploads require controls beyond the compressed file's outer size.

Review:

• Allowed archive formats
• Maximum number of entries
• Maximum nested depth
• Expected uncompressed size
• Compression ratio
• Path traversal attempts
• Duplicate archive paths
• Symbolic links or special files
• Available extraction storage

Do not extract an untrusted archive directly into a public or executable directory.

Malware and Content Processing

Uploaded object
      |
      v
Private quarantine location
      |
      +-- Integrity verification
      +-- File-type validation
      +-- Malware or security analysis
      +-- Metadata extraction
      +-- Content transformation
      |
      +-- Approved:
      |      publish or move to active state
      |
      +-- Rejected:
             retain restricted evidence
             or delete according to policy

The required scanning and analysis depend on the content type, threat model, environment, and organizational policies.

Direct-to-storage Upload

In a direct-to-storage design, the application authorizes the upload, but the file bytes flow directly from the client to object storage.

Client
   |
   | 1. Request upload authorization
   v
Application API
   |
   | 2. Create metadata and temporary authorization
   v
Client
   |
   | 3. Upload parts directly
   v
Object storage
   |
   | 4. Client requests finalization
   v
Application verifies the stored object

Benefits

• Application servers do not relay every upload byte
• Upload bandwidth can scale separately from API traffic
• The client can use provider multipart capabilities directly
• Application connection and memory pressure can be reduced

Responsibilities

• Issue narrowly scoped temporary authorization
• Restrict object key and operations
• Restrict content length and type where supported
• Expire authorization quickly
• Verify the final object independently
• Do not trust a client completion claim

Proxy Upload through the Application

In a proxy upload, the file passes through the application before reaching storage.

Client
   |
   v
Application upload endpoint
   |
   +-- Authentication
   +-- Authorization
   +-- Streaming validation
   +-- Rate limiting
   |
   v
Storage service

Benefits

• Centralized request control
• Application can inspect the stream directly
• Storage credentials remain hidden from the client
• Useful when direct storage access is unavailable

Considerations

• Consumes application bandwidth
• Can consume worker, connection, CPU, and memory resources
• Requires streaming and backpressure
• Can be limited by gateways and request deadlines
• Can increase infrastructure cost

Direct vs Proxied Upload

Stream Uploads Instead of Buffering

Loading an entire large upload into application memory can exhaust the process or force excessive temporary buffering.

Read complete 4 GiB file into memory
      |
      v
Validate
      |
      v
Upload to storage

Read bounded chunk
      |
      +-- Update checksum
      +-- Count bytes
      +-- Apply streaming controls
      |
      v
Write chunk to storage
      |
      v
Repeat until complete

Streaming keeps memory usage bounded by processing data in chunks.

Backpressure

Backpressure prevents a fast producer from overwhelming a slower consumer.

Client sends bytes quickly
        |
        v
Application receives stream
        |
        v
Storage accepts bytes more slowly
        |
        v
Application limits additional reads
until storage catches up

Without backpressure, unprocessed chunks can accumulate in memory or queues.

PHP Stream-copy Example

<?php

declare(strict_types=1);

function copyUploadStream(
    $inputStream,
    $outputStream,
    int $maximumBytes
): array {
    if (!is_resource($inputStream) ||
        !is_resource($outputStream)) {

        throw new InvalidArgumentException(
            'Valid streams are required.'
        );
    }

    $hashContext =
        hash_init(
            'sha256'
        );

    $totalBytes =
        0;

    while (!feof($inputStream)) {
        $chunk =
            fread(
                $inputStream,
                1024 * 1024
            );

        if ($chunk === false) {
            throw new RuntimeException(
                'The upload stream could not be read.'
            );
        }

        if ($chunk === '') {
            continue;
        }

        $chunkLength =
            strlen(
                $chunk
            );

        $totalBytes +=
            $chunkLength;

        if ($totalBytes > $maximumBytes) {
            throw new RuntimeException(
                'The upload exceeds the allowed size.'
            );
        }

        hash_update(
            $hashContext,
            $chunk
        );

        $written =
            fwrite(
                $outputStream,
                $chunk
            );

        if ($written !== $chunkLength) {
            throw new RuntimeException(
                'The upload could not be written completely.'
            );
        }
    }

    return [
        'contentLength' =>
            $totalBytes,
        'sha256' =>
            hash_final(
                $hashContext
            )
    ];
}

Production storage SDKs can provide their own streaming and multipart abstractions. Prefer supported libraries rather than building a low-level provider implementation unnecessarily.

Browser Part-upload Concept

async function uploadPart({
  file,
  partNumber,
  partSize,
  uploadUrl
}) {
  const start =
    (partNumber - 1) * partSize;

  const end =
    Math.min(
      start + partSize,
      file.size
    );

  const partBlob =
    file.slice(
      start,
      end
    );

  const response =
    await fetch(
      uploadUrl,
      {
        method: "PUT",
        body: partBlob
      }
    );

  if (!response.ok) {
    throw new Error(
      `Part ${partNumber} failed`
    );
  }

  return {
    partNumber,
    etag:
      response.headers.get("ETag")
  };
}

A production browser uploader also needs bounded concurrency, authorization refresh, retries, cancellation, progress, checksum handling, and prevention of accidental duplicate completion.

Upload Progress

Multipart progress can be calculated from completed and currently transferred bytes.

\[ UploadProgressPercent = \frac{TransferredBytes}{TotalBytes} \times 100 \]

Progress should be based on bytes transferred rather than only the number of completed parts because the final part can have a different size.

Useful progress fields include:

• Total bytes
• Transferred bytes
• Completed parts
• Total parts
• Current transfer rate
• Retrying-part count
• Paused, active, failed, or completing state

Cancellation

A user can cancel local part transfers while the application also requests an abort of the server-side upload session.

User selects Cancel
      |
      v
Stop scheduling new parts
      |
      v
Cancel in-flight client requests
      |
      v
Request server-side abort
      |
      v
Mark application session aborted
      |
      v
Clean temporary state

An in-flight part can still complete around the same time as the abort. Cleanup should follow the storage provider's documented behaviour.

Upload-session Expiration

Upload sessions should not remain valid indefinitely at the application level.

Expiration protects against:

• Abandoned storage parts
• Long-lived temporary authorization
• Unbounded pending metadata
• Stale client state
• Accumulating cleanup cost

When a client resumes after expiration, the application can create a new upload session according to its policy.

Part-level Checksums

A checksum for each part detects corruption at part granularity.

Part 1 bytes -> Checksum 1
Part 2 bytes -> Checksum 2
Part 3 bytes -> Checksum 3


Storage verifies each received part.


Mismatch:

Retry only the damaged part.

Part-level verification does not always replace a full-object checksum. Verify the complete assembled object using the provider-supported mechanism where required.

Full-object Checksum

The full-object checksum is calculated across the complete ordered file bytes.

Part 1 bytes
+
Part 2 bytes
+
Part 3 bytes
+
Part 4 bytes
      |
      v
Full-object checksum

A provider can support a direct complete-object checksum or a provider-defined checksum-of-checksums. These values must not be treated as interchangeable unless documented.

ETags and Checksums

An ETag identifies a stored entity or version according to the storage service. It must not automatically be interpreted as a cryptographic digest of the complete file.

ETag meaning can depend on:

• Single-part or multipart transfer
• Encryption
• Provider implementation
• Object composition
• Transformation or proxy behaviour

Concurrent Uploads to the Same Key

Two clients or retries can initiate separate multipart uploads for the same object key.

Upload Session A
      |
      +-- Parts uploaded
      |
      +-- Completion requested


Upload Session B
      |
      +-- Parts uploaded
      |
      +-- Completion requested


Risk:

Final current object depends on
provider versioning and overwrite semantics.

Protection can include:

• Generated unique object keys
• One active upload per asset
• Optimistic asset versions
• Conditional object creation
• Storage object versioning
• Idempotency keys

Replace vs Create-new Semantics

Uploading to an existing key can replace the current content according to the storage service contract.

Storage and Database Consistency

Object-storage operations and relational metadata updates do not normally participate in one ordinary local transaction.

Object completion succeeds
        |
        v
Database finalization fails
        |
        v
Object exists,
but metadata remains pending

The reverse can also occur:

Database says upload completed
        |
        v
Object completion failed
        |
        v
Metadata points to missing
or incomplete content

Safer Finalization Pattern

1. Keep metadata in pending state.

2. Complete the storage upload.

3. Retrieve and verify final object properties.

4. Update metadata to verified or processing.

5. Publish asynchronously after remaining checks.

6. Reconcile sessions stuck between stages.

Upload Events

After a verified upload is finalized, the application can create an event for downstream processing.

Metadata finalization transaction:

- Mark asset uploaded
- Store object version
- Store checksum
- Insert outbox event
- Commit


Downstream processing:

- Malware analysis
- Metadata extraction
- Thumbnail generation
- Video transcoding
- Search indexing

Use an outbox or another durable event-publication design when the metadata change and event record must be coordinated.

Processing Retries

Post-upload processing steps should be repeatable without corrupting the asset's state.

Processing message received
      |
      v
Load asset and object version
      |
      v
Check whether this processing version
already completed
      |
      +-- Completed:
      |      return stored outcome
      |
      +-- Not completed:
             process content
             store result idempotently
             update metadata

Resumable Upload State

A robust resume flow should not rely only on the browser's in-memory state.

Persist:

• Asset ID
• Application upload-session ID
• Provider upload ID
• Object key
• Expected file size
• Expected full checksum
• Part size
• Completed part numbers
• Part identifiers
• Part checksums
• Expiration time
• Lifecycle status

Do not persist raw temporary authorization beyond what is required and permitted.

Upload Authorization

Authorization must be checked at every consequential stage.

Upload Limits

Apply limits before and during transfer.

• Maximum file size
• Maximum active uploads per user
• Maximum active uploads per tenant
• Maximum parts per upload
• Minimum and maximum part size
• Maximum parallel parts
• Upload-session lifetime
• Daily or billing-period quota
• Allowed content types
• Maximum extracted archive size

The client should display limits for usability, but the server and storage authorization must enforce them.

Timeouts

Different upload operations need separate timeouts.

A client timeout does not prove that the server-side operation failed. After an uncertain completion result, retrieve the upload or object status before starting a duplicate upload.

Unreliable Networks

Mobile and remote clients can lose connectivity while uploading.

Upload parts 1 through 8
      |
      v
Network disconnects
      |
      v
Client preserves session state
      |
      v
Network returns
      |
      v
Client verifies completed parts
      |
      v
Resume remaining parts

Resume capability is especially valuable when restarting the complete file would be expensive.

Adaptive Transfer

A client can respond to network and device conditions by adjusting transfer behaviour.

Possible controls include:

• Reduce parallelism after repeated timeout or throttling
• Increase part size for stable high-bandwidth transfers
• Reduce part size for unreliable networks
• Pause when connectivity is unavailable
• Respect server rate-limit and retry guidance
• Stop when the user cancels or the session expires

Adaptive behaviour is an implementation strategy and should remain within the storage provider's part-count, size, and request limits.

Multipart Cost

Multipart upload can create additional storage operations and temporarily stored parts.

Cost dimensions can include:

• Initiation request
• Part-upload requests
• Part-list requests
• Completion or abort request
• Temporary storage for incomplete parts
• Network transfer
• Checksum processing
• Lifecycle cleanup

Very small parts increase request count. Abandoned parts can continue consuming storage until they are completed, aborted, or removed by policy.

Upload Observability

Useful upload metrics include:

• Upload sessions initiated
• Uploads completed
• Uploads failed
• Uploads aborted
• Uploads expired
• Files and bytes transferred
• Part-upload latency
• Part retry count
• Completion latency
• Checksum mismatch count
• Upload throughput
• Active upload count
• Abandoned part storage
• Uploads by content category
• Post-upload processing delay
• Metadata finalization failures

Alert Conditions

Alert when:

• Multipart completion failures increase
• Part retries increase unexpectedly
• Checksum mismatches occur repeatedly
• Abandoned multipart storage grows
• Upload-session cleanup stops
• Post-upload processing backlog grows
• Published metadata references missing objects
• Upload-initiation latency exceeds its objective
• One tenant exceeds approved upload concurrency or quota

Upload Troubleshooting Workflow

1. Identify the asset, upload session, and tenant scope safely.
2. Check the application upload-session status.
3. Check the provider upload identifier.
4. List completed parts where supported.
5. Compare expected and uploaded part numbers.
6. Compare part sizes, identifiers, and checksums.
7. Check authorization and temporary-access expiration.
8. Check network, proxy, gateway, and storage errors.
9. Check retry history and rate limiting.
10. Check final-object size, checksum, and version.
11. Check application metadata finalization.
12. Check scanning, extraction, and publishing state.
13. Abort or reconcile unrecoverable sessions.

Common Upload and Multipart Mistakes

Recommended Test Cases

Upload and Multipart Best Practices

Practice Exercise

Design a secure resumable video-upload workflow for your online learning platform.

Requirements

1. Authenticate the instructor.
2. Authorize the target tenant, course, and lesson.
3. Create a pending course-asset record.
4. Generate a stable object key.
5. Create a multipart upload session.
6. Return the approved part size and maximum parallelism.
7. Upload parts directly to object storage.
8. Calculate and verify a checksum for every part.
9. Persist successful part numbers and identifiers.
10. Support pause and resume.
11. Retry failed parts with bounded backoff and jitter.
12. Complete the upload idempotently.
13. Verify the final content length and complete-object checksum.
14. Run required security and media-processing checks.
15. Publish the asset only after all checks succeed.
16. Abort expired uploads and clean abandoned parts.
17. Reconcile pending metadata with storage.
18. Measure upload speed, retries, failures, and processing delay.

Upload-state Template

Frequently Asked Questions