A model reviews 100 search results. 20 are truly relevant; it predicts 25 as relevant and 15 of those are right. That gives TP = 15, FP = 10, FN = 5, TN = 70.

Accuracy = (15 + 70) / 100 = 85%. The number looks strong, but notice that 70 of the 85 correct calls came from rejecting irrelevant results, which is the easy part. Only 15 came from actually finding relevant ones. Accuracy blends both kinds of correctness into a single figure, so it can hide weak performance on the class you care about.

Use accuracy as a sanity check when the classes are roughly balanced and both kinds of error cost about the same. Here they are not balanced (20 vs 80), so precision and recall tell the fuller story.

A payments team scores 10,000 transactions; 100 are fraudulent (1%). A do-nothing model that labels every transaction as legitimate produces TP = 0, FP = 0, FN = 100, TN = 9,900.

Accuracy = 9,900 / 10,000 = 99%, while catching zero fraud. A real model that flags 150 transactions and catches 80 of the 100 frauds has TP = 80, FP = 70, FN = 20, TN = 9,830 and accuracy = 9,910 / 10,000 = 99.1%. The two models are almost indistinguishable on accuracy even though one is useless and one is genuinely valuable.

This is the classic class-imbalance failure. Whenever the positive class is rare, compare against the majority-class baseline before you celebrate any accuracy number, and switch to precision, recall, or PR-AUC.

A support team classifies 2,000 product reviews as positive or negative; the set was deliberately balanced at 1,000 each. The model gets 920 positives and 880 negatives right.

Accuracy = (920 + 880) / 2,000 = 90%. Because the classes are balanced and mislabeling either direction costs roughly the same (a mis-routed ticket), accuracy is a reasonable headline metric here. Precision (920 / (920 + 120) = 88.5%) and recall (92%) sit close to it, which is what you expect when the data is balanced.

The lesson is that accuracy is not a bad metric; it is a metric that only tells the truth under specific conditions. Balanced classes and symmetric error costs are those conditions.

A retail app sends a notification whenever the model predicts a user will open it. Last week it sent 50,000 notifications; 12,000 were opened. Precision = 12,000 / 50,000 = 24%.

Every false positive here is a user interrupted for nothing, and interruptions drive opt-outs. Raising the score threshold from 0.30 to 0.55 cut sends to 22,000 with 9,500 opens: precision rose to 43% while total opens fell only 21%. Opt-out rate dropped by a third.

This is the typical precision trade: accept fewer positives so that the ones you act on are more likely to be right. It is the correct priority whenever acting on a false positive has a cost the user feels.

An email filter moved 4,000 messages to spam this month. Users rescued 320 of them as legitimate. Precision = 3,680 / 4,000 = 92%.

That 8% false-positive rate sounds small until you realise those 320 messages included invoices, interview invitations and a password reset. A missed spam email is an annoyance; a missed legitimate email can be a real loss, so this product should push precision toward 99%+ even if some spam leaks through to the inbox.

The team tightened the threshold and added a rule that never spam-folders senders the user has replied to. Precision reached 98.5% and recall fell from 96% to 91%: an acceptable trade for this product.

A search engine shows 10 results per query. Over 1,000 queries, human raters marked 6,800 of the 10,000 shown results as relevant. Precision of the displayed set = 68%.

Here precision is essentially Precision@10: the share of what the user sees that was worth seeing. The 3,200 irrelevant results are visible failures that make the product feel broken. Precision on the visible set is often the metric product managers feel most directly.

Precision says nothing about the relevant documents that were never surfaced, though. A second engine that shows only 3 results per query, all relevant, would score 100% precision while giving users far less. That is why precision is almost always reported alongside recall.

A screening model is run on 20,000 patients, 200 of whom have the condition. It flags 600 patients, 184 of whom are truly positive. Recall = 184 / 200 = 92%; precision is only 184 / 600 = 31%.

That precision looks poor, but 416 false positives simply mean 416 people get a follow-up test. The 16 false negatives mean 16 sick people go home believing they are fine. In screening, the asymmetry is extreme, so recall is the primary metric and precision is a cost to manage rather than a goal.

The team lowered the threshold further, reaching 97% recall at 22% precision, and judged the extra follow-ups worth it.

A bank’s model reviews 1,000,000 transactions with 2,000 confirmed frauds. It catches 1,400. Recall = 70%. Each missed fraud costs about $180 on average, so the 600 misses cost roughly $108,000 in a month.

Pushing recall to 85% would require flagging three times as many legitimate transactions for review. Whether that is worth it depends on the cost of each manual review and the friction to good customers, which is exactly the trade-off the recall number lets you reason about.

Recall answers the only question the fraud team is asked in the quarterly review: how much of the fraud did we stop?

A recommendation system first retrieves 500 candidate items from a catalog of 2 million, then a heavier ranker orders them. In offline evaluation, users’ held-out purchases numbered 10,000; 7,200 of those appeared somewhere in the 500-item candidate sets. Recall = 72%.

At this stage precision is almost irrelevant: nobody sees the 500 candidates directly. What matters is that the good items are in the pool at all, because the ranker cannot promote what retrieval never returned. The 28% of purchases missing from the pool are a hard ceiling on downstream quality.

Retrieval teams therefore optimise recall at a fixed candidate count, and hand precision responsibility to the ranker.

Model A has precision 0.90 and recall 0.10; Model B has precision 0.50 and recall 0.50. Their arithmetic means are identical: 0.50.

F1 for A = 2 · (0.9 × 0.1) / (1.0) = 0.18. F1 for B = 2 · (0.5 × 0.5) / 1.0 = 0.50. The harmonic mean drags the score toward the weaker of the two numbers, so a model cannot buy a high F1 by being excellent at one thing and terrible at the other.

This is exactly what you want when a single number has to summarise a classifier and neither error type clearly dominates.

A content-moderation model produces a score per post. The team sweeps thresholds on a validation set:

F1 peaks at 0.5, so that becomes the default operating point. But the team also computes F2 (recall-weighted) because a missed policy violation is worse than an over-flag sent to a human reviewer. F2 peaks at 0.3, and that is the threshold they ship for the auto-escalation queue.

F1 gives you a defensible default; Fβ lets you encode the asymmetry your product actually has.

An intent classifier for a support chatbot has 8 intents. Per-class F1 ranges from 0.91 (“reset password”) to 0.42 (“billing dispute”), with the rare intents scoring lowest.

Macro-F1 (the plain average of the 8 scores) is 0.68; micro-F1 (computed from the pooled confusion counts) is 0.84, because the frequent, easy intents dominate the pooled counts. Which one you report changes the story: macro exposes the weak rare classes, micro rewards getting the common ones right.

For a support bot where a mishandled billing dispute is expensive, the team tracks macro-F1 and a per-class table, not a single micro score.

Using the search example: TN = 70 irrelevant results correctly rejected, FP = 10 irrelevant results wrongly shown. Specificity = 70 / 80 = 87.5%.

Recall (75%) says how well the model handles the relevant items; specificity says how well it handles the irrelevant ones. Together they describe the model’s behaviour on both classes without depending on how common each class is, which is why the pair (recall, specificity) is used to draw the ROC curve.

A model can have superb recall by simply predicting positive for everything; its specificity would then be 0%. Reporting both closes that loophole.

A diagnostic test is trialled on 1,000 people known to be healthy. It returns a positive result for 30 of them. Specificity = 970 / 1,000 = 97%; the false-positive rate is 3%.

Now apply the test to a population where 1 in 500 has the condition. In 100,000 people there are 200 true cases and 99,800 healthy people; 3% of the healthy group is 2,994 false alarms. Even with 99% sensitivity, only 198 of the 3,192 positives are real: a positive result is right about 6% of the time.

This is why specificity matters so much for screening rare conditions: a small false-positive rate multiplied by a huge healthy population swamps the true positives.

A login-risk model challenges suspicious logins with a second factor. Of 500,000 legitimate logins last month, 15,000 were challenged. Specificity = 485,000 / 500,000 = 97%.

That 3% of good users being interrupted is the direct user-experience cost of the system, and it is the number the product team watches. If specificity dropped to 94%, twice as many good customers would face friction, regardless of whether recall on actual attacks changed at all.

Recall tells the security team how many attacks were stopped; specificity tells the product team how many customers were annoyed. Both belong on the dashboard.

Take 3 positives with scores 0.9, 0.7, 0.3 and 3 negatives with scores 0.6, 0.4, 0.2. There are 9 positive–negative pairs. The positive wins the comparison in 8 of them; the only loss is positive 0.3 vs negative 0.6 and 0.4 (two losses), so 7 wins out of 9. AUC = 0.78.

No threshold was chosen anywhere in that calculation. AUC evaluates the entire ordering at once, which makes it a good metric when the operating threshold has not been decided or will be tuned later by a different team.

It also means two models with the same AUC can behave very differently at the specific threshold you end up using.

Model A outputs 0.9, 0.8, 0.4, 0.3 for four items; Model B outputs 0.60, 0.55, 0.51, 0.50 for the same four. The first two items are positives.

Both models order the items identically, so both have AUC = 1.0. Yet Model B says every item is roughly a coin flip, while Model A is confident. If these scores feed a bid formula (expected value = p × value), Model B would bid nearly the same on everything and Model A would bid sensibly.

AUC is blind to that difference. When the probability itself is used downstream, pair AUC with log loss or a calibration check.

A click model is evaluated on 1,000,000 impressions with 5,000 clicks (0.5%). It scores ROC-AUC = 0.93, which looks excellent.

But at a threshold that recovers 60% of clicks, the model also flags 3% of non-clicks: 29,850 false positives against 3,000 true positives, so precision is about 9%. ROC-AUC never showed that, because the false-positive rate (3% of a huge negative class) looks tiny on the ROC axis.

With rare positives, ROC-AUC can stay high while the precision you actually experience is poor. PR-AUC exposes this; ROC-AUC does not.

A defect detector on a manufacturing line sees 2% defective parts. Its PR-AUC is 0.41. On its own that seems mediocre.

A random model’s expected precision is the prevalence, 0.02, so the baseline PR-AUC is roughly 0.02. The detector is about 20× better than random. The same model’s ROC-AUC of 0.91 would have suggested near-perfection, hiding the fact that at high recall, precision collapses.

Always state prevalence next to PR-AUC. A PR-AUC of 0.41 at 2% prevalence is strong; at 40% prevalence it would be barely useful.

Two models detect abusive comments (positive rate 1.5%). Model X: ROC-AUC 0.95, PR-AUC 0.52. Model Y: ROC-AUC 0.94, PR-AUC 0.61.

On ROC-AUC, X looks marginally better. On PR-AUC, Y is clearly better, meaning Y keeps precision higher across the recall range where moderation actually operates. Because moderators can only review a fixed number of items per day, precision at the operating recall is what matters, and Y wins.

When positives are rare and you care about the positive class, PR-AUC is the tiebreaker; ROC-AUC differences of a hundredth are noise.

A fraud model is retrained weekly. ROC-AUC has been stable at 0.97 for months. PR-AUC drifted from 0.58 to 0.44 over six weeks.

The investigation found that a new fraud pattern was scored slightly below many legitimate high-value transactions. Those few misorderings barely moved ROC-AUC (they are a handful of pairs among millions) but they sat exactly in the high-precision region of the PR curve, where the review team operates.

PR-AUC is the more sensitive early-warning signal for rare-class degradation, which is why fraud and abuse teams often alert on it rather than on ROC-AUC.

A pCTR model predicts 0.05 for ten impressions that each get clicked (loss −ln 0.05 = 3.00 each, total 30.0). Another model predicts 0.50 for the same ten (loss 0.69 each, total 6.9).

Now consider a single impression where the model predicted 0.999 and the user did not click: loss = −ln(0.001) = 6.9. That one confident miss costs as much as ten hedged predictions. Log loss is unbounded as p approaches 0 or 1, so it disciplines overconfidence.

This is why production probability models are often clipped to [0.001, 0.999]: a single extreme miss should not dominate a batch metric.

An ad system bids expected value = p(click) × value. Two models have identical ROC-AUC (0.82). Model A has log loss 0.31; Model B has log loss 0.27.

In a simulated auction, Model B’s better-calibrated probabilities produced 4% more clicks at the same spend, because its bids tracked the real click likelihood more closely. Model A over-bid on segments where it was overconfident and under-bid elsewhere.

Whenever the probability is consumed as a number rather than a rank, log loss is a better proxy for business value than AUC.

A model reports log loss 0.045 on a dataset with 1.2% positives. Is that good? A naive model that always predicts the base rate 0.012 would score −[0.012·ln 0.012 + 0.988·ln 0.988] ≈ 0.065.

Normalised log loss = 0.045 / 0.065 = 0.69, meaning the model removes about 31% of the uncertainty relative to guessing the prevalence. That ratio is comparable across datasets with different base rates; raw log loss is not.

Report log loss relative to the base-rate model so that a shift in prevalence between test sets is not mistaken for a change in model quality.

A subscription-churn model is evaluated in three score buckets:

The ordering is right, so AUC is excellent. But every prediction is roughly 20 points too high. A retention team offering discounts to everyone above 60% predicted churn would be spending on many customers who were never going to leave.

A reliability plot draws predicted vs actual per bucket; a calibrated model sits on the diagonal. This one sits well below it.

A gradient-boosted click model has AUC 0.84 but is overconfident at the top: predictions above 0.6 are realised at only 0.45. Its ECE is 0.08.

Fitting isotonic regression on a held-out set maps raw scores to calibrated probabilities. AUC stays at 0.84 (monotone mappings preserve order) while ECE drops to 0.01. Downstream, the expected-value bidder now spends 12% less for the same clicks.

Calibration is fixable without retraining. Platt scaling and isotonic regression are the standard tools, applied as a post-processing layer.

A delivery-time model is calibrated overall: predictions of “80% chance of on-time” arrive on time 79% of the time. Sliced by region, though, urban predictions of 80% are realised at 88% and rural ones at 64%.

Overall calibration is a weighted average that hides opposite errors cancelling out. Customers in rural areas receive systematically overoptimistic promises.

Compute calibration per segment (region, device, product category, new vs returning user) before trusting the aggregate. Good overall ECE is necessary but not sufficient.

Three events all occur (y = 1). Predictions: 0.9, 0.6, 0.01.

Mean Brier = 0.383; mean log loss = 1.74. Both punish the confident miss most, but log loss makes it 44× worse than the 0.9 prediction while Brier makes it 98× worse in absolute terms but caps at 1.0. Brier stays bounded and interpretable; log loss goes to infinity.

Choose Brier when you want a stable, easy-to-explain number; log loss when you specifically want to hammer overconfidence.

A weather-style rain model has Brier score 0.18 over 10,000 forecasts. Decomposing it: reliability (calibration error) = 0.02, resolution (how much the forecasts separate rainy from dry days) = 0.05, and uncertainty (base-rate variance) = 0.21. Brier = 0.21 − 0.05 + 0.02 = 0.18.

The decomposition shows most of the score comes from inherent uncertainty in the data, with modest calibration error and modest resolution. A model that improves resolution (sharper, more decisive forecasts) while keeping reliability low will reduce Brier.

This makes Brier useful for diagnosing whether to work on calibration or on discrimination next.

A conversion model scores Brier = 0.031. The base-rate model (always predict the 3.4% conversion rate) scores 0.034 − 0.034² ≈ 0.0328.

The Brier skill score = 1 − 0.031 / 0.0328 = 0.055, meaning the model is only 5.5% better than guessing the average. That is a much less flattering picture than its ROC-AUC of 0.79 suggested, and it tells the team the probabilities themselves are not yet useful for revenue forecasting even though the ranking is decent.

Like log loss, Brier is most meaningful relative to a trivial baseline.

A recipe app shows five recommendations above the fold. For one user, three of the five match her dietary preferences and past saves. P@5 = 0.60.

Averaged over 50,000 users, P@5 is 0.48. The product team knows from experiments that every irrelevant slot on that first screen reduces session length, so P@5 is their north-star offline metric: it measures exactly the surface the user looks at.

Precision@K is the right metric when the number of displayed items is fixed and each slot has a visible cost.

Two rankers each put 3 relevant items in their top 5. Ranker A places them at positions 1, 2, 3; Ranker B at positions 3, 4, 5. Both score P@5 = 0.60.

Users scan top-down and click early, so Ranker A is clearly better, but Precision@K cannot see it. If order inside the window matters, you need MRR, MAP or NDCG.

Precision@K is a set metric wearing a ranking costume: it only asks what got in, not where.

A search team reports P@1 = 0.81, P@3 = 0.70, P@10 = 0.52. The drop is normal: the first result is easiest to get right, and precision decays as you go deeper.

Which K to track depends on the interface. On mobile, three results are visible without scrolling, so P@3 is the primary metric. On desktop, ten are visible, so P@10 matters too. Reporting several cutoffs shows whether a model change helped the visible slots or only the tail.

Never quote Precision@K without the K.

A two-stage search system retrieves 200 candidates per query, then a neural ranker orders them. Offline, raters marked an average of 14 relevant documents per query. On average 11.2 of them appear in the 200 candidates. Recall@200 = 0.80.

The remaining 20% of relevant documents can never be shown, no matter how good the ranker is. Improving retrieval recall from 0.80 to 0.90 lifted final NDCG@10 by more than any ranker change that quarter.

Recall@K at the candidate stage is a ceiling on everything downstream.

In offline evaluation, each user’s last 20 purchases are hidden. The model produces 50 recommendations per user. On average 6 of the 20 hidden purchases appear in the 50. Recall@50 = 0.30.

That sounds low, but with a 2-million-item catalog, random recommendations would score about 0.0005. Recall@K needs to be judged against catalog size and the number of relevant items, not against 1.0.

Also note that “relevant” here means “was purchased”, which undercounts: users might have loved items they never saw. Recall@K in recommenders is a lower bound.

Increasing K raises recall and lowers precision. For one system: R@10 = 0.35, P@10 = 0.60; R@50 = 0.62, P@50 = 0.21; R@200 = 0.85, P@200 = 0.07.

The retrieval team picks K = 200 because they want high recall and precision is the ranker’s job. The results-page team cares about K = 10 where precision dominates. The same system is judged by different metrics at different stages.

Plotting recall against K shows where the marginal candidate stops adding value, which sets the retrieval budget.

A streaming service hides the next show each user watched and asks the model for 10 recommendations. For 68,000 of 100,000 users, the hidden show is somewhere in the 10. Hit Rate@10 = 0.68.

This is the simplest ranking metric to explain to stakeholders: two-thirds of the time, the thing the user wanted was on the screen. It is common in session-based recommendation and next-basket prediction where there is exactly one ground-truth item.

Its bluntness is the point: no graded relevance, no position weighting, just “was it there”.

Two models both achieve Hit Rate@10 = 0.68. Model A tends to place the hit at rank 1–2; Model B tends to place it at rank 8–10. Users of Model B scroll more and click less.

Hit Rate treats these identically. Reporting MRR alongside it exposes the difference: Model A’s MRR is 0.55, Model B’s is 0.12.

Use Hit Rate@K to answer “is it reachable” and pair it with MRR or NDCG to answer “is it prominent”.

For a product search engine: Hit Rate@1 = 0.42, @3 = 0.61, @5 = 0.70, @10 = 0.79, @20 = 0.86.

The curve flattens after 10, so extending the results page from 10 to 20 items would help only 7% more users find their item, while doubling page load. The team keeps the page at 10 and invests in moving hits from rank 5–10 up to rank 1–3 instead.

Hit Rate at several K values maps directly onto UI decisions like page size and infinite scroll.

A knowledge-base search is tested on 4 questions. The correct article appears at rank 1, 3, 2, and not at all. Reciprocal ranks: 1.00, 0.33, 0.50, 0. MRR = 1.83 / 4 = 0.46.

MRR fits because each question has one right answer and the user stops looking once they find it. A second correct article at rank 7 would add nothing to MRR, which matches how the user behaves.

The steep discount (rank 2 is worth half of rank 1) reflects how strongly users prefer the top slot.

A search box shows 5 suggestions as the user types. Over a week, the suggestion the user eventually selected appeared at rank 1 for 55% of sessions, rank 2 for 20%, rank 3 for 8%, rank 4–5 for 7%, and not at all for 10%.

MRR ≈ 0.55·1 + 0.20·0.5 + 0.08·0.33 + 0.07·0.22 + 0.10·0 = 0.69. A new model that moves 5% of sessions from rank 2 to rank 1 would raise MRR by 0.025, which the team has correlated with a measurable drop in keystrokes per search.

For known-item lookup, MRR tracks the user’s effort more faithfully than NDCG.

A travel site’s hotel search returns many acceptable options per query. Model A puts one good hotel at rank 1 and junk below it; Model B puts good hotels at ranks 1 through 8. Both have MRR = 1.0.

Users comparing hotels want a full page of good options, so Model B is far better, and MRR cannot tell them apart. MAP or NDCG would.

MRR assumes the user needs one answer. If the user is browsing or comparing, choose a metric that credits every relevant result.

A query has 3 relevant documents. The ranker returns 5 results; relevant ones are at ranks 1, 3 and 5.

AP = (1.00 + 0.67 + 0.60) / 3 = 0.76. If the third relevant document had never been retrieved, AP would be (1.00 + 0.67 + 0) / 3 = 0.56: the missing item still counts in the denominator.

Averaging AP across all test queries gives MAP.

Two rankings each retrieve all 3 relevant documents in the top 6. Ranking A places them at 1, 2, 3: AP = (1 + 1 + 1) / 3 = 1.00. Ranking B places them at 4, 5, 6: AP = (0.25 + 0.40 + 0.50) / 3 = 0.38.

Recall@6 is identical (1.0) for both, and Precision@6 is identical (0.5). MAP is the metric that sees the difference, because precision is sampled at each relevant hit.

Use MAP when several relevant items exist and you want all of them near the top, with binary relevance.

An e-commerce search has graded labels: exact match (3), close substitute (2), loosely related (1), irrelevant (0). To compute MAP the team collapses grades 1–3 into “relevant”.

Two rankings now score the same MAP even though one puts exact matches first and the other puts loosely-related items first. The collapse threw away the information that mattered most.

When relevance has degrees, NDCG is the natural upgrade from MAP.

Graded labels: 3 = perfect, 2 = useful, 1 = somewhat, 0 = irrelevant. A ranker returns items with grades [2, 3, 0] in its top 3.

DCG@3 = 7.42. The ideal order is [3, 2, 0]: IDCG@3 = 7/1 + 3/1.58 + 0 = 8.89. NDCG@3 = 7.42 / 8.89 = 0.83.

Swapping the first two results would give NDCG = 1.0. The metric penalised putting the merely-useful item above the perfect one.

With linear gain (rel instead of 2^rel−1), a grade-3 item is worth three grade-1 items. With exponential gain, it is worth seven. Placing one perfect result at rank 1 then outweighs several mediocre ones.

For a search team, this matches user behaviour: a single exact answer at the top satisfies the query, whereas three vaguely related results do not. For a browsing feed where variety matters, some teams use linear gain instead.

The gain function is a modelling choice; make it explicit in the metric definition you publish.

A search team requires any ranker change to show NDCG@10 ≥ +0.5% offline before an A/B test. A candidate model scores +1.2% NDCG@10 but −0.3% NDCG@3.

The split tells them the model improved the lower half of the page while slightly hurting the top three slots, which is where most clicks happen. They ship it to a small A/B test rather than a full rollout, and indeed CTR on rank 1 dips slightly.

Report NDCG at more than one cutoff; different cutoffs answer different product questions.

A store forecasts daily demand for five products. Actuals vs predictions: (120, 110), (80, 95), (200, 190), (50, 45), (300, 280). Absolute errors: 10, 15, 10, 5, 20. MAE = 60 / 5 = 12 units.

The number is immediately meaningful to a planner: “we are off by about 12 units per product per day”. That legibility is MAE’s main advantage.

It treats over- and under-forecasting equally, which is fine if both cost the same; if stockouts cost more than overstock, a weighted or quantile loss is better.

Ten delivery-time predictions have errors of 3 minutes each, except one that is off by 90 minutes. MAE = (9·3 + 90) / 10 = 11.7 minutes. RMSE for the same errors is √((9·9 + 8100)/10) = 28.6 minutes.

The one outlier more than doubled RMSE but only moved MAE from 3 to 11.7. If the 90-minute miss was a data error (a courier forgot to scan), MAE gives the fairer picture of typical performance.

Choose MAE when large errors are noise you want to downweight, not disasters you want to punish.

A revenue model reports MAE = $4,200. For enterprise accounts averaging $250,000, that is 1.7% and excellent. For SMB accounts averaging $6,000, the same MAE is 70% and useless.

A single pooled MAE hid that the model was effectively only working on large accounts. Segmenting MAE by account size, or reporting MAE divided by mean actual (WAPE), made the problem visible.

Always read MAE against the typical magnitude of what is being predicted.

An ad platform predicts daily spend so campaigns end the month on budget. Being $50 off on a $1,000 day is harmless; being $2,000 off wrecks the month. Over 30 days the model’s errors are mostly $30–$80 with two days at $1,500 and $2,200.

MAE = $180. RMSE = $490. The gap between them is a signal: a few very large errors are present. Because those large misses are precisely the ones that matter here, RMSE is the metric the team optimises.

When the cost of an error grows faster than the error itself, RMSE aligns with the business.

Two models forecast weekly sales. Model A: MAE 40, RMSE 45. Model B: MAE 35, RMSE 70.

Model B is better on the typical week (lower MAE) but has some weeks with very large misses (RMSE much larger than MAE). If the RMSE/MAE ratio is close to 1, errors are uniform; if it is large, errors are spiky.

Which model to deploy depends on whether the business can tolerate occasional big misses in exchange for better average accuracy. Reporting both numbers makes that a decision rather than a surprise.

Most regression models are trained by minimising squared error, so RMSE on the training set naturally looks good. On the holdout set, a house-price model shows RMSE = $58,000 while training RMSE was $22,000.

The large gap signals overfitting. The team added regularisation and the holdout RMSE fell to $41,000 while training RMSE rose to $35,000: a healthier balance.

Always compare RMSE on held-out data, and treat the train/holdout gap as a diagnostic in its own right.

A dwell-time model has R² = 0.72 on the test set. The sum of squared errors is 28% of the variance you would get by predicting the average dwell time for everyone. The model explains 72% of the variation.

Whether that is good depends on the domain. For a physical process it would be weak; for predicting human behaviour from sparse signals it is strong. R² is a relative measure, not an absolute one.

Its unit-free nature makes it convenient for comparing models across targets with different scales.

A model trained on last year’s traffic is applied to this year’s, after a site redesign. Test R² = −0.35.

Negative means the model’s errors are larger than the errors of simply predicting this year’s mean. The model is actively worse than a constant. This usually signals distribution shift or a broken pipeline rather than a subtly bad model.

A negative R² in production is an alarm bell, not a low score.

Adding more features raises training R² from 0.61 to 0.79. Adjusted R² (which penalises extra features) rises only to 0.63, and holdout RMSE gets slightly worse.

Plain R² never decreases when you add features, so it rewards complexity. Adjusted R² and held-out error are the honest checks.

Use R² for communication, but make deployment decisions on MAE/RMSE measured out of sample.

A retailer forecasts three products: actual 1,000 predicted 950 (5% error); actual 100 predicted 110 (10%); actual 10 predicted 13 (30%). MAPE = (5 + 10 + 30) / 3 = 15%.

MAPE lets a planner compare accuracy across products regardless of scale, which MAE cannot. That is why it is common in supply-chain reporting.

But note the 3-unit miss on the small product contributed twice as much to MAPE as the 50-unit miss on the big one. MAPE overweights small-volume items.

A product sells 0 units on a Sunday; the model predicted 2. The percentage error is |0 − 2| / 0, which is undefined. On Monday it sells 1 and the model predicted 4: 300% error from a 3-unit miss.

A handful of such days can push MAPE into the hundreds of percent and make the report meaningless. Teams typically switch to WAPE, which sums absolute errors and divides by the sum of actuals: WAPE = Σ|y − ŷ| / Σy. Over a month with total actual 3,000 and total absolute error 240, WAPE = 8%, stable and interpretable.

If your target can be zero, do not use MAPE.

Actual demand is 100. Predicting 50 gives 50% error; predicting 150 also gives 50% error. So far symmetric. But now actual is 50 and the prediction is 100: error is 100%. Same 50-unit miss, double the penalty.

Because the denominator is the actual, over-forecasting is penalised more than under-forecasting of the same size. Minimising MAPE therefore biases models toward predicting low.

SMAPE uses the average of actual and prediction as the denominator to soften this. Whichever you choose, know that percentage errors carry a hidden bias.