Introduction

Data does not remain in one place or one state forever. It is created, collected, validated, stored, transformed, queried, shared, archived, and eventually deleted.

This complete journey is called the data lifecycle.

Understanding the lifecycle helps system designers answer practical questions:

• Where did the data come from?
• Who owns it?
• Can the application trust it?
• Where should it be stored?
• Who may access it?
• How long should it remain?
• Can it be recovered after a failure?
• When should it be archived or deleted?
• How can deletion be verified?

Different frameworks divide the lifecycle into different numbers of stages. This article uses a practical system-design flow:

Plan
  |
  v
Create or Collect
  |
  v
Validate and Classify
  |
  v
Store and Protect
  |
  v
Process and Transform
  |
  v
Use and Share
  |
  v
Monitor and Maintain
  |
  v
Archive or Retain
  |
  v
Delete or Destroy

Prerequisites

What Is a Data Lifecycle?

A data lifecycle is the sequence of stages through which data passes from its initial creation or collection to its final deletion or destruction.

Beginning of lifecycle:

A learner creates an account.
A course instructor uploads a video.
An application records a quiz attempt.
A service receives an event.


Middle of lifecycle:

Data is validated.
Data is stored.
Data is transformed.
Data is searched.
Data is shared with authorized services.
Data supports application decisions.


End of lifecycle:

Data becomes inactive.
Data moves to archive.
Required retention expires.
Data is deleted according to policy.

Not every dataset follows exactly the same path. Some temporary data can be deleted soon after processing. Financial or audit data can require a longer retention period. A derived search index can be rebuilt instead of backed up like its authoritative source.

Data Lifecycle vs Data Pipeline

A pipeline can operate during one or more lifecycle stages. The lifecycle is the broader management model.

Stage 1: Plan

Lifecycle management should begin before data is collected.

Planning defines:

• The business purpose
• The authoritative owner
• The minimum required fields
• The expected data volume
• The required quality
• The classification and sensitivity
• The permitted uses
• The storage model
• The retention period
• The recovery requirements
• The deletion process

Learning-platform Example

Data:

Learner quiz attempts


Purpose:

Display results,
calculate progress,
and support approved learning analytics.


Owner:

Learning-platform domain


Authoritative store:

Relational database


Retention:

Defined by approved business
and organizational requirements.


Derived copies:

Progress dashboard
and analytics dataset

Stage 2: Create or Collect

Data enters the system through user actions, application events, sensors, uploaded files, APIs, database transactions, imports, or external sources.

Collection Flow

Source produces data
      |
      v
Authenticate source
      |
      v
Authorize operation
      |
      v
Validate format and size
      |
      v
Attach provenance metadata
      |
      v
Accept or reject data

Collection should record enough provenance to explain where the data came from and when it entered the system.

Stage 3: Validate and Classify

Newly collected data should not automatically be treated as complete, accurate, safe, or trustworthy.

Validation can include:

• Required-field checks
• Data-type checks
• Length and range checks
• Allowed-value checks
• Uniqueness checks
• Referential-integrity checks
• File-size and content-type checks
• Checksum verification
• Schema validation
• Duplicate detection
• Business-rule validation

SQL Validation Example

CREATE TABLE learner_progress
(
    learner_id BIGINT NOT NULL,
    course_id BIGINT NOT NULL,
    completion_percentage DECIMAL(5, 2) NOT NULL,
    progress_status VARCHAR(30) NOT NULL,
    last_activity_at TIMESTAMP NOT NULL,

    PRIMARY KEY
    (
        learner_id,
        course_id
    ),

    CONSTRAINT ck_completion_percentage
        CHECK
        (
            completion_percentage
            BETWEEN 0 AND 100
        ),

    CONSTRAINT ck_progress_status
        CHECK
        (
            progress_status IN
            (
                'not-started',
                'in-progress',
                'completed'
            )
        )
);

Classification

Classification identifies how the data should be handled.

Possible classifications:

Public
Internal
Confidential
Restricted


Possible data categories:

Business data
Account data
Course content
Operational logs
Audit data
Temporary processing data

Use the classifications and handling rules approved by the relevant organization. The example labels above are illustrative.

Stage 4: Store and Protect

Validated data is stored using an appropriate storage model.

Protection can include:

• Least-privilege access
• Encryption in transit
• Encryption at rest
• Checksums
• Replication or erasure coding
• Snapshots and backups
• Versioning
• Audit records
• Recovery testing

Stage 5: Process and Transform

Stored data can be cleaned, normalized, enriched, aggregated, converted, or indexed for downstream use.

Raw data
    |
    v
Validate
    |
    v
Clean
    |
    v
Standardize
    |
    v
Transform
    |
    v
Enrich
    |
    v
Publish prepared data

Media Example

Original lesson video
      |
      v
Integrity verification
      |
      v
Security analysis
      |
      v
Metadata extraction
      |
      v
Video transcoding
      |
      v
Thumbnail generation
      |
      v
Approved renditions published

Search Example

Published article
      |
      v
Extract searchable fields
      |
      v
Tokenize and normalize text
      |
      v
Build search document
      |
      v
Update inverted index

Raw, Prepared, and Derived Data

Derived data should retain lineage to the authoritative inputs, processing version, and generation time when reproducibility matters.

Data Lineage

Data lineage records where data originated, how it moved, and which transformations were applied.

Source quiz attempts
      |
      v
Validation pipeline
      |
      v
Normalized attempt records
      |
      v
Course-progress calculation
      |
      v
Progress dashboard
      |
      v
Approved analytics report

Useful lineage metadata includes:

• Source system
• Source record or object identifier
• Pipeline name
• Pipeline version
• Transformation time
• Input and output schema versions
• Quality-check results
• Responsible owner

Stage 6: Use and Analyze

Authorized applications, users, reports, search systems, and analytical processes consume the prepared data.

Uses can include:

• Application transactions
• Operational dashboards
• Business reports
• Full-text search
• Notifications
• Data analysis
• Machine-learning features
• Audit and compliance review

Access Flow

User or service requests data
      |
      v
Authenticate identity
      |
      v
Authorize purpose and resource
      |
      v
Apply tenant and data filters
      |
      v
Return minimum required fields
      |
      v
Record audit evidence where required

Access should follow least privilege and purpose limitation. A service should receive only the data necessary for its approved responsibility.

Stage 7: Share and Distribute

Data can be shared between internal services, teams, applications, partners, reporting systems, or approved external recipients.

A sharing contract should define:

• Producer and consumer
• Purpose
• Schema
• Data owner
• Permitted fields
• Freshness
• Quality expectations
• Security requirements
• Retention
• Error handling
• Deletion propagation

Event-sharing Example

{
  "eventType": "EnrollmentCompleted",
  "eventVersion": 1,
  "eventId": "generated-event-id",
  "tenantId": "tenant-17",
  "learnerId": 1042,
  "courseId": 42,
  "occurredAt": "event-time"
}

Avoid placing unnecessary personal or confidential fields into broadly distributed events.

Stage 8: Monitor and Maintain

Active data needs continuous management. Data can become stale, duplicated, inconsistent, corrupted, inaccessible, or incorrectly classified.

Monitor:

• Data freshness
• Completeness
• Validity
• Duplicate rate
• Integrity failures
• Schema changes
• Pipeline failures
• Access anomalies
• Storage growth
• Backup age
• Retention exceptions
• Deletion backlog

Data-quality Dimensions

Stage 9: Archive and Retain

Data that is no longer needed for frequent operational access can move to an archive or lower-access storage tier.

Frequently used data
      |
      v
Infrequently used data
      |
      v
Archive storage
      |
      v
Retention period expires
      |
      v
Approved deletion

Archiving is not the same as deletion. Archived data remains stored and must still be protected, discoverable when authorized, included in retention management, and recoverable according to its requirements.

Retention Metadata

{
  "retentionClass": "completed-course-record",
  "retentionStart": "approved-start-time",
  "retainUntil": "approved-end-time",
  "legalHold": false,
  "lifecycleState": "archived"
}

Retention periods should come from approved legal, regulatory, organizational, contractual, and business requirements rather than from an arbitrary technical default.

Retention vs Backup

Backup copies must also follow approved retention and deletion requirements.

Stage 10: Delete or Destroy

At the end of the approved lifecycle, data should be deleted or destroyed according to policy.

Deletion becomes eligible
      |
      v
Check retention requirement
      |
      v
Check legal or investigation hold
      |
      v
Authorize deletion
      |
      v
Delete authoritative data
      |
      v
Delete or expire derived copies
      |
      v
Update search, cache, replicas,
exports, and processing systems
      |
      v
Record deletion outcome

Deletion can involve:

• Primary database records
• Object versions
• File-system copies
• Search documents
• Cache entries
• Analytics projections
• Temporary files
• Queued messages
• Exports
• Backups according to their retention process

Soft Delete vs Hard Delete

Soft-delete Schema Example

ALTER TABLE course_assets
ADD
    deletion_status VARCHAR(30) NOT NULL
        DEFAULT 'active',
    deletion_requested_at TIMESTAMP NULL,
    deleted_at TIMESTAMP NULL;

A soft-delete marker is a lifecycle state, not proof that the underlying content has been permanently destroyed.

Lifecycle State Machine

Created
   |
   v
Validated
   |
   v
Active
   |
   +-- Quality failure --> Quarantined
   |
   +-- Update --> Active new version
   |
   v
Inactive
   |
   v
Archived
   |
   +-- Legal hold --> Retained
   |
   v
Deletion pending
   |
   v
Deleted

Explicit state transitions make lifecycle behaviour easier to test, audit, retry, and reconcile.

Lifecycle Metadata Model

CREATE TABLE data_assets
(
    data_asset_id BIGINT PRIMARY KEY,
    tenant_id BIGINT NOT NULL,
    asset_name VARCHAR(255) NOT NULL,
    asset_type VARCHAR(50) NOT NULL,

    lifecycle_status VARCHAR(30) NOT NULL,
    classification VARCHAR(30) NOT NULL,
    owner_id BIGINT NOT NULL,

    source_system VARCHAR(100) NOT NULL,
    source_reference VARCHAR(500) NULL,
    schema_version INT NOT NULL,

    retention_class VARCHAR(100) NOT NULL,
    retain_until TIMESTAMP NULL,
    legal_hold BOOLEAN NOT NULL
        DEFAULT FALSE,

    created_at TIMESTAMP NOT NULL,
    last_verified_at TIMESTAMP NULL,
    archived_at TIMESTAMP NULL,
    deletion_requested_at TIMESTAMP NULL,
    deleted_at TIMESTAMP NULL,

    CONSTRAINT ck_data_asset_status
        CHECK
        (
            lifecycle_status IN
            (
                'created',
                'validated',
                'active',
                'quarantined',
                'inactive',
                'archived',
                'deletion-pending',
                'deleted'
            )
        )
);

Actual classification, retention, and status values should follow approved organizational policies and domain requirements.

Lifecycle Automation

Lifecycle rules can automate selected transitions.

lifecycle:
  inactive-content:
    condition:
      status: inactive
      age: approved-inactive-period
    action:
      move-to: archive

  expired-content:
    condition:
      retention-expired: true
      legal-hold: false
    action:
      queue-deletion: true

  abandoned-uploads:
    condition:
      status: pending
      session-expired: true
    action:
      abort-and-clean: true

This is conceptual configuration. Lifecycle automation must use approved rules, protected permissions, audit evidence, and safe retry behaviour.

Lifecycle Events

{
  "eventType": "DataAssetArchived",
  "eventVersion": 1,
  "eventId": "generated-event-id",
  "tenantId": "tenant-17",
  "dataAssetId": 981,
  "sourceVersion": 14,
  "occurredAt": "event-time"
}

Consumers can update search indexes, caches, analytics, and storage tiers in response to lifecycle events.

Event consumers should be idempotent and reject stale lifecycle versions.

Authoritative and Derived Copies

One logical dataset can exist in several forms.

Authoritative learner record
        |
        +-- Search projection
        +-- Cache entry
        +-- Analytics dataset
        +-- Export
        +-- Backup
        +-- Audit record

For every copy, document:

• Purpose
• Owner
• Source of truth
• Freshness
• Access rules
• Retention
• Deletion behaviour
• Rebuild or recovery process

Lifecycle Reconciliation

Partial failures can leave systems in different lifecycle states.

Database:

asset = deleted


Object storage:

object still exists


Search index:

document remains searchable


Cache:

download link remains cached


Result:

Lifecycle state is inconsistent.

A reconciliation job compares authoritative state with derived systems and repairs mismatches.

Security throughout the Lifecycle

Ownership and Responsibilities

Every important dataset should have clear ownership.

Lifecycle Observability

Useful metrics include:

• Records or objects created
• Validation failure rate
• Quarantined-data count
• Data-quality rule failures
• Pipeline processing delay
• Replication and backup health
• Data freshness
• Storage growth by classification
• Inactive-data volume
• Archive-transition count
• Retention exceptions
• Legal-hold count
• Deletion requests
• Deletion backlog
• Deletion failures
• Orphaned-data count
• Derived-copy synchronization delay

Alert Conditions

Alert when:

• Required validation stops running
• Data-quality failures increase unexpectedly
• Backups or integrity checks fail
• Authoritative and derived copies diverge
• Restricted data is accessed unexpectedly
• Retention-eligible data is not archived or deleted
• Lifecycle automation fails repeatedly
• Deletion remains incomplete across downstream systems
• A legal hold is bypassed or changed unexpectedly
• Storage growth exceeds the approved forecast

Lifecycle-review Workflow

1. Identify the dataset and business purpose.
2. Identify the owner and authoritative source.
3. Inventory all stored and derived copies.
4. Document collection and validation rules.
5. Document classification and access requirements.
6. Document storage, durability, and recovery requirements.
7. Document processing and lineage.
8. Document permitted consumers and sharing contracts.
9. Define data-quality and freshness objectives.
10. Define archive and retention requirements.
11. Define deletion and legal-hold behaviour.
12. Test recovery, archival, and deletion.
13. Monitor lifecycle events and reconciliation results.

Common Data-lifecycle Mistakes

Recommended Test Cases

Data-lifecycle Best Practices

Practice Exercise

Create a data-lifecycle design for your online learning platform.

Requirements

1. Inventory learner accounts, enrollments, quiz attempts, course videos, certificates, logs, and search documents.
2. Define the purpose of each dataset.
3. Identify the authoritative source.
4. Assign the data owner and technical owner.
5. Define validation and quality rules.
6. Define classification and access requirements.
7. Select the appropriate storage model.
8. Define backup and restore requirements.
9. Document transformations and derived copies.
10. Define data-sharing contracts.
11. Define active, inactive, archived, and deleted states.
12. Define approved retention requirements.
13. Define legal-hold handling.
14. Design deletion propagation.
15. Create a reconciliation job.
16. Test one complete lifecycle from creation to deletion.

Lifecycle-design Template

Frequently Asked Questions