Introduction

Sharding distributes data and workload across several database nodes. Ideally, each shard receives a manageable share of storage, reads, writes, CPU usage, memory usage, and network traffic.

Distribution is rarely perfect. One key, tenant, partition, shard, node, or geographic region can receive significantly more load than the others. This overloaded area is called a hotspot.

Shard A:

18% of requests


Shard B:

20% of requests


Shard C:

47% of requests


Shard D:

15% of requests


Result:

Shard C is the workload hotspot.

A hotspot limits the capacity of the complete system. Adding more shards does not help when incoming work continues targeting the same overloaded ownership unit.

Prerequisites

What Is a Hotspot?

A hotspot is a disproportionately busy or constrained part of a distributed system that limits overall performance or availability.

A hotspot can consume excessive:

• CPU time
• Memory
• Storage capacity
• Disk input and output operations
• Network bandwidth
• Database connections
• Locks
• Queue capacity
• Cache capacity
• Request-processing concurrency

Types of Hotspots

Data Skew vs Workload Skew

Balanced storage:

Shard A = 25%
Shard B = 25%
Shard C = 25%
Shard D = 25%


Unbalanced traffic:

Shard A = 10%
Shard B = 12%
Shard C = 68%
Shard D = 10%

Measuring Distribution

Let \(L_i\) represent the measured load on shard \(i\), and let \(n\) represent the total number of shards.

The average load is:

\[ AverageLoad = \frac{ \sum_{i=1}^{n} L_i }{ n } \]

A simple hotspot ratio is:

\[ HotspotRatio = \frac{ MaximumShardLoad }{ AverageShardLoad } \]

This ratio is only one signal. Compare CPU, storage, I/O, connections, latency, and traffic separately because different resources can have different hotspot owners.

Range-sharding Hotspots

Range sharding places contiguous key intervals together.

Shard A:

IDs 1 to 1,000,000


Shard B:

IDs 1,000,001 to 2,000,000


Shard C:

IDs 2,000,001 to 3,000,000

When the key increases sequentially, new records target the final active range.

New ID:

2,900,001
    -> Shard C


New ID:

2,900,002
    -> Shard C


New ID:

2,900,003
    -> Shard C

Shards A and B can remain mostly idle while Shard C receives all current writes.

Common Monotonic Keys

• Auto-incrementing identifiers
• Creation timestamps
• Ordered event sequence numbers
• Time-based keys
• Increasing invoice or transaction numbers

Time-series Hotspot

Historical partitions:

January -> inactive
February -> inactive
March -> inactive


Current partition:

April -> receives every new event

Time-based partitioning supports time-range scans and retention, but one current partition can receive all inserts.

Possible design directions include:

• Hash by entity or device before applying a time range
• Create several current write buckets
• Split the active range earlier
• Roll partitions at an appropriate size or load boundary
• Buffer and batch writes where correctness allows

Hash-sharding Hotspots

Hash sharding generally distributes many distinct keys more evenly than range sharding.

It does not guarantee even traffic when key popularity differs.

Key distribution:

1,000,000 keys spread evenly


Traffic:

course:42
    -> 500,000 requests per minute


Other keys:
    -> low traffic


Result:

The shard owning course:42
becomes hot.

Hashing the same key repeatedly always produces the same placement. The hash function cannot divide one indivisible key among several owners.

Consistent Hashing and Hotspots

Consistent hashing minimizes ownership movement during node changes. Virtual nodes can improve distribution of hash ranges among physical nodes.

Consistent hashing does not automatically solve:

• One extremely popular key
• One very large tenant
• Several hot virtual ranges placed on one node
• Unequal physical-server capacity
• Uneven operation cost among keys

Weighted virtual nodes can account for different server capacities, but weights must be validated against actual workload characteristics.

Directory-sharding Hotspots

Directory sharding maps tenants or keys explicitly to shards.

tenant-101 -> shard-a
tenant-102 -> shard-a
tenant-103 -> shard-b
tenant-104 -> shard-c

It allows a noisy or large tenant to be moved independently.

Before:

tenant-101 -> shared shard-a


After:

tenant-101 -> dedicated shard-d

Directory sharding offers flexible hotspot correction, but the move still requires controlled copying, synchronization, validation, and routing cutover.

Geo-sharding Hotspots

Geographic populations and traffic are rarely distributed equally.

Region A:

15% of users


Region B:

60% of users


Region C:

25% of users

Geo sharding can improve regional locality while creating a dominant regional shard group.

Possible controls include:

• Adding capacity inside the hot region
• Sub-sharding the region by tenant or user
• Separating reads from writes
• Using regional cache and delivery layers
• Moving only when approved residency rules permit it

Hot Tenant

Tenant-based sharding colocates tenant data and transactions, but tenants can have dramatically different sizes.

Tenant A:

10,000 records


Tenant B:

25,000 records


Tenant C:

500,000,000 records

Hashing each tenant ID once does not divide Tenant C. The complete tenant remains assigned to one shard.

Large-tenant Directions

• Move the tenant to a dedicated shard
• Sub-shard the tenant by user, account, object, or another key
• Separate interactive and analytical workloads
• Archive cold tenant data
• Apply tenant-specific rate and concurrency policies

Low-cardinality Shard Key

A shard key with few distinct values cannot distribute data among many shards effectively.

Shard key:

status


Possible values:

active
inactive
pending


Available shards:

20

Three key values cannot create balanced ownership across twenty independent shard destinations without another distribution dimension.

Write-amplification Hotspot

One logical write can trigger several physical operations.

One enrollment update
      |
      +-- Base table update
      +-- Secondary-index update
      +-- Audit record
      +-- Outbox event
      +-- Cache invalidation
      +-- Search update

A shard with moderate request count can still be hot because its operations are more expensive.

Contention Hotspots

A hotspot can occur around one shared record even when data and traffic are otherwise distributed.

Many workers update:

global_counter


Result:

- Lock contention
- Transaction retries
- Serialized execution
- Increased latency

Examples include:

• Global counters
• One inventory record
• One account balance
• One queue head
• One sequence generator
• One popular metadata record

Detecting Hotspots

Compare metrics at several ownership levels:

• Key
• Tenant
• Partition
• Shard
• Physical node
• Region
• API route
• Database query fingerprint

Architecture hotspot analysis can also correlate recurring incidents, outages, performance problems, bottlenecks, unstable integrations, and overloaded services with affected architecture components.

Conceptual Shard-load Query

SELECT
    shard_id,
    COUNT(*) AS operation_count,
    SUM(duration_ms) AS total_duration_ms,
    AVG(duration_ms) AS average_duration_ms,
    MAX(duration_ms) AS maximum_duration_ms
FROM database_operation_metrics
WHERE recorded_at >= :window_start
GROUP BY shard_id
ORDER BY total_duration_ms DESC;

Record counts alone are insufficient. Duration, CPU, reads, writes, bytes, locks, and failures can reveal different hotspots.

Hotspot Mitigation Categories

Caching Hot Reads

Popular read
      |
      v
Distributed cache
      |
      +-- Hit:
      |      return value
      |
      +-- Miss:
             read authoritative store
             populate cache
             return value

Caching can protect a datastore from repeated reads when the value is safe to cache for the caller, tenant, authorization scope, and required freshness.

Request Coalescing

Request coalescing lets concurrent misses for one key share one backend operation.

10,000 concurrent requests
for course:42
      |
      v
One active cache fill
      |
      v
Other requests wait within
a controlled boundary
      |
      v
Result is shared

Without coalescing, an expired popular cache entry can create a thundering herd against the backing datastore.

Read Replication

Read replicas or replicated cache entries can distribute reads for a hot key.

Hot read key
      |
      +-- Replica A
      +-- Replica B
      +-- Replica C

Account for replication lag, invalidation, authorization, routing, and failure-domain placement.

Read replication does not distribute writes requiring one authoritative order.

Key Salting

Key salting adds a controlled distribution component to a logical key.

Original key:

course:42


Salted keys:

course:42:0
course:42:1
course:42:2
course:42:3

Writes can be distributed among buckets. Reads that need the complete logical value can query and combine all relevant buckets.

Suitable Uses

• Distributed counters
• Append-only events
• Aggregations that can merge bucket results
• Work queues that do not require one strict global order

Risks

• Reads can require fan-out
• Global ordering becomes more difficult
• Uniqueness must be enforced separately
• Transactions spanning buckets require coordination
• Bucket count changes require migration planning

Composite Shard Keys

A composite key can preserve a business ownership prefix while distributing activity across buckets.

Composite key:

tenant_id
+
bucket_number


Example:

tenant-17:bucket-0
tenant-17:bucket-1
tenant-17:bucket-2
tenant-17:bucket-3

The bucket can be derived from a stable record attribute, random assignment, or a hash of a secondary identifier.

Hash Prefix before Time Range

Instead of:

timestamp -> one current partition


Use:

hash(device_id) -> bucket
timestamp -> range inside bucket

This spreads current writes across multiple buckets while retaining time-based organization inside each bucket.

Moving a Hot Tenant

Shared Shard A:

Tenant 11
Tenant 12
Tenant 17
Tenant 19


Tenant 17 becomes hot
      |
      v
Move Tenant 17
to dedicated Shard D

The move requires a controlled resharding process covering historical data, concurrent changes, validation, versioned routing, cutover, and rollback.

Splitting a Hot Range

Before:

Shard A owns 1 to 10,000,000


After:

Shard A owns 1 to 5,000,000

Shard B owns 5,000,001 to 10,000,000

Select the split point from actual data and workload distribution. Equal key ranges can still produce unequal traffic.

Rate and Concurrency Limits

A structural hotspot cannot always be corrected immediately. Overload controls protect the system while a longer-term fix is implemented.

Incoming operation
      |
      v
Identify trusted tenant,
key, route, and priority
      |
      v
Check rate and concurrency
      |
      +-- Capacity available:
      |      process
      |
      +-- Capacity unavailable:
             reject, defer,
             or queue within a bound

Useful boundaries include:

• Requests per tenant
• Concurrent operations per hot key
• Report generations per shard
• Writes per partition
• Queued jobs per ownership range

Workload Isolation

Interactive API pool:

Course access
Enrollment
Progress updates


Batch worker pool:

Reports
Exports
Analytics
Backfills

Separating expensive batch work can prevent reports, migrations, and analytics from consuming resources required by interactive requests.

Query and Index Optimization

A shard can appear hot because every operation performs unnecessary work.

Review:

• Query plans
• Missing or ineffective indexes
• Large scans
• Repeated joins
• Lock duration
• Transaction size
• Unbounded result sets
• Connection-pool behaviour
• Materialization and sorting

Resharding an inefficient query can distribute the inefficiency instead of removing it.

Autoscaling and Hotspots

Autoscaling adds general capacity. It can help when the workload can be distributed among new instances.

Autoscaling does not automatically help when:

• Every request targets one key
• One database shard owns all relevant data
• One global lock serializes operations
• One external dependency is rate limited
• The shard key does not permit finer distribution

Conceptual Hotspot Policy

hotspotControl:
  detection:
    dimensions:
      - shard
      - tenant
      - partition
      - key
      - route

    metrics:
      - request-rate
      - write-rate
      - latency
      - cpu
      - storage-io
      - connection-usage
      - lock-wait

  immediateProtection:
    rateLimiting: enabled
    concurrencyLimiting: enabled
    boundedQueues: enabled
    requestCoalescing: enabled

  readHotspot:
    cache: enabled
    replicaReads: workload-specific

  writeHotspot:
    keySplitting: workload-specific
    saltedBuckets: workload-specific
    batching: workload-specific

  isolation:
    hotTenantDedicatedShard: supported
    batchWorkloadSeparatePool: true

  structuralCorrection:
    splitRange: supported
    moveTenant: supported
    reshardingRequired: validated-process

  observability:
    hotKeyDetection: enabled
    shardSkewDetection: enabled
    migrationTracking: enabled

This configuration is conceptual. Thresholds and corrective actions must be based on measured workload, consistency requirements, and database-specific capabilities.

Learning-platform Examples

Security Considerations

Hotspot controls must preserve tenant and authorization boundaries.

• Derive tenant identity from trusted authentication context
• Do not expose raw hot-key identifiers unnecessarily
• Prevent callers from selecting privileged traffic priority
• Apply rate limits using trusted caller or tenant identity
• Maintain authorization on cached and replicated responses
• Audit tenant moves and shard-directory changes
• Protect hotspot metrics containing customer identifiers

Observability

Useful hotspot metrics include:

• Records and bytes by shard
• Reads and writes by shard
• Requests by tenant and key
• CPU, memory, I/O, and network by node
• Connection usage by shard
• Lock waits by key or table
• Latency by shard and route
• Cache hit rate by key class
• Top-key request concentration
• Queue depth by partition
• Rate-limit and concurrency-limit decisions
• Scatter-gather frequency
• Shard skew and storage skew
• Resharding and tenant-move progress

Structured Hotspot Event

{
  "hotspotType": "tenant",
  "ownershipUnit": "protected-tenant-reference",
  "shardId": "shard-c",
  "constrainedResource": "database-connections",
  "trafficShare": "measured-value",
  "action": "concurrency-limited",
  "policyVersion": "approved-policy-version"
}

Avoid including credentials, session identifiers, private request payloads, or unnecessary personal data in hotspot logs.

Alert Conditions

Alert when:

• One shard receives disproportionate traffic
• One tenant exceeds its expected capacity share
• One key dominates request volume
• Shard storage distribution becomes significantly uneven
• One node approaches CPU, memory, I/O, or connection limits
• Lock-wait duration increases around one record or index
• Cache misses for a popular key increase suddenly
• A time-based active partition approaches saturation
• Rate or concurrency limits remain continuously active
• Resharding fails to reduce the original hotspot
• A regional shard group approaches maximum capacity
• Useful throughput falls while incoming traffic remains stable

Troubleshooting Workflow

1. Identify the user-visible symptom.
2. Identify the constrained resource.
3. Compare workload by shard, tenant, partition, and key.
4. Separate storage skew from traffic skew.
5. Check the shard-key distribution and cardinality.
6. Check for monotonically increasing range keys.
7. Check for a single popular key or tenant.
8. Check query plans, indexes, locks, and transaction duration.
9. Check cache hit rate and cache-miss concentration.
10. Check replicas and read-routing distribution.
11. Check batch jobs, reports, migrations, and background workers.
12. Apply immediate rate, concurrency, or queue controls.
13. Select caching, splitting, isolation, or resharding based on the cause.
14. Verify that the corrective action reduces the constrained-resource load.

Common Hotspot Mistakes

Recommended Test Cases

Hotspot Best Practices

Practice Exercise

Design hotspot protection for your online learning platform.

Requirements

1. Measure traffic and storage by shard.
2. Record the most frequently accessed course keys.
3. Identify the largest and busiest tenants.
4. Create a traffic-skew dashboard.
5. Simulate one highly popular course.
6. Add caching and request coalescing.
7. Simulate one enterprise tenant generating heavy reports.
8. Move reports to a bounded worker queue.
9. Apply per-tenant concurrency limits.
10. Test a sequential progress-event key.
11. Replace it with a composite distributed key where appropriate.
12. Move one large tenant to a dedicated shard.
13. Validate the move and routing cutover.
14. Confirm that the original shard's resource pressure decreases.
15. Test traffic recovery after the hotspot ends.

Hotspot-design Template

Frequently Asked Questions