Introduction

A process can be running without being capable of serving a valid request. For example, an application can remain visible in the operating-system process list while its worker threads are blocked, initialization is incomplete, required configuration is missing, or database connections are exhausted.

Load balancers and orchestration platforms need a reliable way to determine whether an application instance should receive traffic or be restarted. Health checks provide this information through automated probes.

A health check sends a controlled request or operation to an application instance and interprets the result according to a defined contract.

Application Instance
        |
        +-- Startup check
        |      Has initialization completed?
        |
        +-- Liveness check
        |      Is the process capable of making progress?
        |
        +-- Readiness check
               Can this instance safely receive traffic?

Treating all these questions as one generic health check can create self-inflicted outages. A temporary database problem should not necessarily restart every application process, and an application that is still starting should not receive production traffic.

Prerequisites

What Is a Health Check?

A health check is an automated test used to evaluate the operational state of a service, process, server, container, or dependency.

Health-check system
        |
        v
Send probe to application instance
        |
        +-- Successful result:
        |      take configured healthy action
        |
        +-- Failed result:
               take configured unhealthy action

The action depends on the health-check type. The system can:

• Add an instance to traffic rotation
• Remove an instance from traffic rotation
• Restart a process or container
• Delay traffic during application startup
• Trigger replacement of an unhealthy instance
• Raise an operational alert
• Initiate failover

Health Check vs Monitoring

Health checks should not replace metrics and diagnostic monitoring. A healthy result does not prove that latency, error rate, throughput, or cost meets the required objective.

Main Health-check Types

Startup Check

A startup check determines whether an application has finished its initialization process.

Process starts
      |
      v
Load configuration
      |
      v
Initialize runtime
      |
      v
Load required assets
      |
      v
Create initial connections
      |
      v
Warm required components
      |
      v
Startup check succeeds

Until startup succeeds, the platform can delay liveness and readiness decisions that would otherwise restart or route traffic to the application prematurely.

Suitable Startup Conditions

• Required configuration loaded
• Application container initialized
• Required migrations or startup validation completed
• Runtime is capable of accepting probe requests
• Required local assets are available

Liveness Check

A liveness check answers whether the process remains capable of making progress.

Liveness succeeds:

Process event loop progresses
Worker thread can execute
Internal watchdog updates


Liveness fails:

Process deadlocked
Event loop permanently blocked
Critical internal runtime failure

A liveness failure commonly results in a process or container restart.

Good Liveness Characteristics

• Fast
• Local to the process
• Independent of external services
• Stable during temporary downstream outages
• Focused on failures that a restart can correct

Poor Liveness Dependencies

• External payment provider
• Remote search service
• Database query
• Shared cache availability
• Message-broker connection
• Internet connectivity

If a shared database fails and every application's liveness check fails, the orchestrator can restart every application instance. The restarts do not repair the database and can add further load during recovery.

Readiness Check

A readiness check answers whether the instance can safely receive new production traffic.

Load Balancer
      |
      v
Readiness probe
      |
      +-- Ready:
      |      include instance
      |      in backend pool
      |
      +-- Not ready:
             stop sending
             new requests

Readiness can fail temporarily without requiring a restart.

Possible Readiness Conditions

• Startup completed
• Required configuration loaded
• The instance is not shutting down
• Required local resources are available
• Connection pools have usable capacity
• The instance is not critically overloaded
• Required dependencies are available according to policy

Liveness vs Readiness

Startup vs Readiness

L4 Health Checks

A Layer 4 health check can verify whether a TCP connection can be established to a configured backend port.

Health checker
      |
      v
Open TCP connection
to backend port
      |
      +-- Connection accepted:
      |      transport endpoint available
      |
      +-- Connection refused or timed out:
             transport endpoint unavailable

A successful Layer 4 check proves that a process accepted the connection. It does not prove that the application can complete a useful request.

L7 Health Checks

A Layer 7 health check sends an application-level request and evaluates the response.

GET /health/ready HTTP/1.1
Host: api.example.com
User-Agent: platform-health-check

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store

{
  "status": "ready"
}

An HTTP-based probe can detect an application that accepts transport connections but cannot process HTTP requests correctly.

L4 vs L7 Health Checks

Recommended Endpoint Separation

/health/startup

Question:
Has application initialization completed?


/health/live

Question:
Is this process capable of making progress?


/health/ready

Question:
Can this instance safely receive traffic?


/health/details

Purpose:
Protected diagnostic information for operators.

Endpoint names are design choices. The important requirement is that each endpoint has one clear purpose and a predictable operational consequence.

Public vs Protected Health Endpoints

A basic health endpoint can be reachable by infrastructure without exposing sensitive implementation details.

{
  "status": "ready"
}

{
  "databaseHost": "internal-db-name",
  "databaseUser": "application-user",
  "cacheHost": "internal-cache-name",
  "connectionString": "sensitive-value",
  "exception": "complete internal stack trace"
}

Detailed health diagnostics should be protected by appropriate authentication, authorization, network restrictions, and logging controls.

Dependency Checks

Applications often depend on databases, caches, queues, object storage, and other services.

Application
    |
    +-- Database
    +-- Cache
    +-- Message Broker
    +-- Object Storage
    +-- External Service

Not every dependency should be included in every health check.

Questions to Ask

• Is the dependency required for every request?
• Can the application serve a degraded response without it?
• Will failing readiness reduce or worsen the incident?
• Can all instances fail the same dependency check together?
• Can the check create meaningful load on the dependency?
• Is cached dependency status sufficient?
• Can a circuit breaker provide a safer signal?

Required vs Optional Dependencies

Classification depends on the specific service and operation. One global readiness verdict can be too coarse for an application with independent capabilities.

Shallow vs Deep Checks

Shallow Check

Verify:

- Process responds
- Internal runtime is progressing
- Required local configuration is loaded
- Instance is not draining

Deep Check

Verify:

- Database query
- Cache operation
- Queue operation
- Object-storage operation
- External service request

Shallow checks are generally better for frequent routing decisions. Deep checks can be useful for protected diagnostics, synthetic monitoring, and carefully designed readiness conditions.

Health-check Amplification

Health checks themselves create traffic.

If \(I\) instances are checked every \(T\) seconds, the approximate check rate is:

\[ HealthCheckRate = \frac{I}{T} \]

If several load-balancer nodes, orchestrators, and monitoring systems probe independently, actual traffic can be higher.

100 application instances

Each readiness check executes:

- 1 database query
- 1 cache command
- 1 external API request


Result:

Health-check traffic reaches and
depends on every shared dependency.

Avoid turning a health endpoint into a recurring load-test request.

Probe Timing

Health-check timing controls how quickly a failure is detected and how easily temporary problems create false removals or restart loops.

Exact values should come from observed startup times, failure modes, transient latency, routing requirements, and recovery objectives.

Detection-time Intuition

A simplified failure-detection estimate is:

\[ DetectionTime \approx ProbeInterval \times FailureThreshold \]

Timeout duration, scheduling delay, network latency, and load-balancer implementation can increase the actual time.

More frequent checks detect failures faster but create more traffic and can react more aggressively to brief interruptions.

Preventing Health Flapping

Flapping occurs when an instance repeatedly moves between ready and not-ready states.

Probe succeeds
    -> add instance

Probe fails briefly
    -> remove instance

Next probe succeeds
    -> add instance

Load increases
    -> probe fails again

Possible controls include:

• Consecutive failure thresholds
• Consecutive success thresholds
• Cooldown periods
• Readiness hysteresis
• Cached dependency state
• Capacity-aware load balancing
• Removing the root cause of overload

Cascading Readiness Failure

Readiness failure can reduce capacity and increase load on the remaining instances.

One instance becomes overloaded
      |
      v
Readiness fails
      |
      v
Traffic moves to remaining instances
      |
      v
Remaining instances become overloaded
      |
      v
Their readiness checks fail
      |
      v
Service loses all healthy backends

Readiness should not be used as the only overload-control mechanism. Capacity planning, autoscaling, rate limiting, admission control, queueing, and load shedding can be safer controls.

Health Checks during Deployment

Start new instance
      |
      v
Startup check runs
      |
      v
Readiness succeeds
      |
      v
Instance enters traffic pool
      |
      v
Old instance marked not ready
      |
      v
New requests stop
      |
      v
In-flight requests drain
      |
      v
Old instance terminates

This sequence supports rolling deployment without sending traffic to an uninitialized instance or terminating an instance still processing work.

Graceful Shutdown and Readiness

When an instance begins shutdown, it should report not ready before the process exits.

Termination requested
      |
      v
Set readiness to false
      |
      v
Load balancer stops new routing
      |
      v
Drain in-flight work
      |
      v
Close background consumers
      |
      v
Close connections
      |
      v
Exit process

The time needed to update backend membership must be included in the shutdown grace period.

Automatic Repair

Infrastructure platforms can use per-instance health signals to replace instances that remain unhealthy.

Instance health repeatedly fails
      |
      v
Instance removed from traffic
      |
      v
Repair controller evaluates state
      |
      v
Unhealthy instance replaced
      |
      v
Replacement initializes
      |
      v
Readiness succeeds
      |
      v
Replacement joins backend pool

Automatic repair is useful when instances are disposable and initialization is reliable. It can create repeated replacement loops when the fault is in a shared configuration, image, dependency, or network.

Kubernetes Probe Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: course-api
spec:
  replicas: 3

  template:
    spec:
      containers:
        - name: course-api
          image: approved-course-api-image

          ports:
            - containerPort: 8080

          startupProbe:
            httpGet:
              path: /health/startup
              port: 8080

          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080

          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080

This example shows separation of probe purposes. Production timing, thresholds, security, termination, and resource settings must be selected from tested application behaviour.

PHP Health Controller

<?php

declare(strict_types=1);

final class HealthController
{
    public function __construct(
        private ApplicationState $applicationState,
        private ShutdownState $shutdownState
    ) {
    }

    public function live(): void
    {
        $this->writeResponse(
            200,
            [
                'status' => 'alive'
            ]
        );
    }

    public function ready(): void
    {
        if ($this->shutdownState->isDraining()) {
            $this->writeResponse(
                503,
                [
                    'status' => 'not-ready'
                ]
            );

            return;
        }

        if (!$this->applicationState->isInitialized()) {
            $this->writeResponse(
                503,
                [
                    'status' => 'not-ready'
                ]
            );

            return;
        }

        $this->writeResponse(
            200,
            [
                'status' => 'ready'
            ]
        );
    }

    public function startup(): void
    {
        if (!$this->applicationState->isInitialized()) {
            $this->writeResponse(
                503,
                [
                    'status' => 'starting'
                ]
            );

            return;
        }

        $this->writeResponse(
            200,
            [
                'status' => 'started'
            ]
        );
    }

    private function writeResponse(
        int $statusCode,
        array $payload
    ): void {
        http_response_code(
            $statusCode
        );

        header(
            'Content-Type: application/json'
        );

        header(
            'Cache-Control: no-store'
        );

        echo json_encode(
            $payload,
            JSON_THROW_ON_ERROR
        );
    }
}

This example keeps the public health response minimal. Production code should use your application's existing routing, response, error-handling, and dependency abstractions.

Protected Diagnostic Response

{
  "status": "degraded",
  "checks": {
    "runtime": {
      "status": "healthy"
    },
    "database": {
      "status": "healthy"
    },
    "cache": {
      "status": "unavailable",
      "required": false
    }
  },
  "serviceVersion": "approved-service-version"
}

This type of detailed response should not expose credentials, connection strings, internal addresses, stack traces, or confidential configuration.

Health-status Model

Synthetic Health Checks

A synthetic check exercises a controlled user-visible journey from outside the application process.

Synthetic monitor
      |
      v
Resolve DNS
      |
      v
Establish TLS
      |
      v
Call API gateway or web endpoint
      |
      v
Authenticate using a controlled test identity
      |
      v
Execute a safe representative operation
      |
      v
Validate response

Synthetic checks can detect problems that an internal process check misses, including DNS, certificates, routing, gateway policy, and selected backend failures.

Synthetic operations must be safe, isolated, identifiable, and cleaned up where they create data.

Security Considerations

Protect health-check systems against:

• Exposure of internal topology
• Exposure of software versions
• Exposure of credentials or configuration
• Unauthenticated access to detailed diagnostics
• Expensive public probe abuse
• Cache poisoning
• Forged health-check headers
• Direct backend bypass

Load balancers and orchestrators should reach health endpoints through approved network paths. If a special health-check header is used, do not treat the presence of that header alone as proof of a trusted caller.

Do Not Cache Health Responses

Health endpoints should return the current instance state rather than an old cached response.

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store

A stale healthy result can keep an unavailable backend in traffic rotation. A stale unhealthy result can keep a recovered backend unavailable.

Observability

Useful health-check metrics include:

• Startup duration
• Liveness probe success and failure count
• Readiness probe success and failure count
• Probe latency
• Healthy backend count
• Backend removal count
• Backend restoration count
• Process restart count
• Automatic repair count
• Health-state transition count
• Dependency-check latency
• Time spent not ready
• Connection-draining duration
• Synthetic-check success rate

Alert Conditions

Alert when:

• Healthy backend count falls below the required minimum
• All instances become not ready
• An instance repeatedly restarts
• Startup repeatedly exceeds its expected operational boundary
• Readiness rapidly changes between states
• Probe latency increases unexpectedly
• A shared dependency causes widespread readiness failure
• Automatic instance replacement repeats without recovery
• Synthetic checks fail while internal probes remain healthy
• An instance remains draining beyond the allowed period

Troubleshooting Workflow

1. Identify which probe is failing: startup, liveness, or readiness.
2. Identify what action the failed probe triggers.
3. Call the health endpoint through the expected network path.
4. Check response status, body, and latency.
5. Check application logs using the probe request time.
6. Check whether the application completed startup.
7. Check whether the instance is intentionally draining.
8. Check local CPU, memory, threads, file descriptors, and connections.
9. Check required dependency state.
10. Check probe interval, timeout, and thresholds.
11. Check load-balancer or orchestrator backend membership.
12. Check network rules between the checker and the application.
13. Check recent application or infrastructure deployments.
14. Determine whether restart, traffic removal, degradation, or alerting is the correct action.

Common Health-check Mistakes

Recommended Test Cases

Health-check Best Practices

Practice Exercise

Design health checks for your online learning platform's course API.

Requirements

1. Create separate startup, liveness, and readiness endpoints.
2. Define the exact question answered by each endpoint.
3. Keep liveness independent of the database.
4. Prevent traffic before configuration loading is complete.
5. Define database behaviour for course-read operations.
6. Define degraded behaviour when the recommendation service is unavailable.
7. Define readiness during connection-pool exhaustion.
8. Prevent public access to detailed diagnostics.
9. Add no-cache response headers.
10. Define probe interval, timeout, and thresholds from test results.
11. Mark the instance not ready during shutdown.
12. Drain existing requests before termination.
13. Test a slow startup.
14. Test a database outage.
15. Test an internal application deadlock.
16. Add an external synthetic course-read test.
17. Monitor startup duration, readiness changes, and restarts.

Health-check Design Template

Frequently Asked Questions