✏️ Explanatory Question

Your dashboard query suddenly takes 40 seconds in production — walk me through diagnosing it

👁 12 Views
📘 Detailed Answer
🟢 Easy
💡

Answer with Explanation

Section 12: Real Performance Debugging

87

A dashboard query suddenly takes 40 seconds in production — how do you diagnose and fix it?

Level: Very Hard — The quintessential senior interview question; they want your systematic methodology, not luck.

Scenario: A query that ran in 200ms last month now takes 40 seconds. Nothing in the code changed. The table grew from 500K to 20M rows. Users are timing out. You have 10 minutes to diagnose it on a live system. What's your step-by-step approach?

The Slow Query & Table

-- orders table: ~20 million rows
-- The slow dashboard query
SELECT o.customer_id, COUNT(*) AS orders, SUM(o.amount) AS revenue
FROM orders o
WHERE YEAR(o.created_at) = 2026
  AND o.status = 'completed'
GROUP BY o.customer_id
ORDER BY revenue DESC
LIMIT 100;

The Diagnostic Methodology (say this out loud in the interview)

  • Run EXPLAIN first — check type, key, and rows. Look for type=ALL (full scan) and key=NULL.
  • Check the Extra columnUsing filesort and Using temporary are red flags for the GROUP BY / ORDER BY.
  • Spot the index-killers — here, YEAR(o.created_at) wraps the column in a function, so no index can be used.
  • Rewrite to be sargable — convert the function to a range condition.
  • Add a covering composite index matching the WHERE + GROUP BY.
  • Re-run EXPLAIN to confirm the index is now used and rows examined dropped.

Reading the EXPLAIN — Before

typekeyrowsExtra
ALL NULL 20,000,000 Using where; Using temporary; Using filesort

Full table scan of 20M rows + temp table + filesort = 40 seconds.

The Fix — Step by Step

-- STEP 1: make the WHERE sargable (remove YEAR() function)
SELECT o.customer_id, COUNT(*) AS orders, SUM(o.amount) AS revenue
FROM orders o
WHERE o.created_at >= '2026-01-01'
  AND o.created_at <  '2027-01-01'
  AND o.status = 'completed'
GROUP BY o.customer_id
ORDER BY revenue DESC
LIMIT 100;

-- STEP 2: add a composite index covering the filter + group column
-- Order: equality (status) first, then range (created_at), then group/agg cols
CREATE INDEX idx_orders_dash
    ON orders (status, created_at, customer_id, amount);

-- STEP 3: refresh optimizer statistics
ANALYZE TABLE orders;

-- STEP 4: verify improvement
EXPLAIN SELECT o.customer_id, COUNT(*), SUM(o.amount)
FROM orders o
WHERE o.created_at >= '2026-01-01' AND o.created_at < '2027-01-01'
  AND o.status = 'completed'
GROUP BY o.customer_id ORDER BY SUM(o.amount) DESC LIMIT 100;

Reading the EXPLAIN — After

typekeyrowsExtra
range idx_orders_dash ~180,000 Using where; Using index

From 20M rows scanned to ~180K, "Using index" (covering), no filesort → ~200ms.

Two root causes to name: (1) the YEAR() function made the query non-sargable, and (2) there was no supporting index. As the table grew, the full scan cost grew linearly — 200ms at 500K became 40s at 20M.
Interviewer follow-up: "The query is still slow at peak load even after indexing — what else?" → Check for lock contention, an outdated ANALYZE causing a bad plan, buffer pool too small (disk reads), or move heavy aggregation to a read replica or a pre-computed summary table.