Data Science Model Evaluation Metrics Dashboard

Metric Selection Guide (Flowchart‑style)

High‑level guide for picking suitable metrics:

Problem type?
- Binary / multiclass classification → use metrics in Performance (Classification).
- Regression → use metrics in Regression & Correlation.
- Hypothesis / A/B test → use Inference & Hypothesis Testing.
Is the dataset imbalanced?
- Yes → emphasise recall, precision, F1, PR AUC, MCC; avoid relying on plain accuracy.
- No → accuracy, ROC AUC and F1 may all be useful, but choose them from the error costs, probability/ranking needs and deployment objective rather than class balance alone.
Threshold‑free ranking vs threshold‑specific performance?
- Ranking quality → ROC AUC / PR AUC.
- Specific operating point → confusion matrix, precision/recall/F1 at that threshold.
Regression goals?
- Penalise big errors heavily → RMSE / MSE.
- Interpret “average absolute deviation” → MAE.
- Multiplicative/log-scale discrepancies matter on a non-negative target → RMSLE or log-scale modelling; target skew or a wide range alone is not sufficient.
Need interpretability?
- Global feature importance → permutation importance, SHAP.
- Local explanations → SHAP, LIME.

Go Beyond Definitions: Focus on Pitfalls & Robustness

This resource prioritizes practical experience, highlighting the critical errors that lead to faulty models and bad decisions. Learn to avoid common pitfalls like:

The Accuracy Trap when working with imbalanced datasets.
Misinterpreting the p-value – it is not the probability that the null hypothesis is true.
Data leakage from incorrectly applying cross-validation to time-series or temporally ordered data.

Data Science Model Evaluation Metrics: Condensed Reference Sheet

Quick overview of what to use, when to use it, and what to watch out for.

Metric Selection Guide

Problem Type → Metric Category
Binary/Multiclass Classification → Performance (Classification) metrics
Regression → Regression & Correlation metrics
Hypothesis/A/B Testing → Inference & Hypothesis Testing
Dataset Imbalance?
Yes → Focus on Recall, Precision, F1, PR AUC, MCC; avoid plain Accuracy
No → Accuracy, ROC AUC and F1 may all be useful; choose among them from the deployment objective and error costs
Ranking vs. Threshold Performance
Ranking quality → ROC AUC / PR AUC
Specific operating point → Confusion matrix, Precision/Recall/F1 at threshold
Regression Goals
Heavily penalize big errors → RMSE / MSE
Interpret “average absolute deviation” → MAE
Multiplicative/log-scale discrepancies matter on a non-negative target → RMSLE or log-scale modelling; target skew or a wide range alone is not sufficient
Need Interpretability?
Global feature importance → Permutation importance, SHAP
Local explanations → SHAP, LIME

Classification Metrics (Quick Reference)

Interpretation rule: there is no universal “good/fair/poor” score for most classification metrics. Compare against a relevant baseline, report uncertainty, and choose operating thresholds from error costs and deployment constraints.

Binary formulas

Accuracy  = (TP + TN) / (TP + FP + TN + FN)
Precision = TP / (TP + FP)
Recall    = TP / (TP + FN)
F1        = 2 × (Precision × Recall) / (Precision + Recall)
Balanced accuracy = (Recall + Specificity) / 2
MCC       = (TP×TN − FP×FN) / √[(TP+FP)(TP+FN)(TN+FP)(TN+FN)]

For multiclass problems, balanced accuracy is the macro-average recall across classes. Precision/recall/F1 require an averaging convention such as macro, weighted or micro. The binary MCC formula above has a standard multiclass generalisation.

Metric	Range / anchor	Interpretation	When useful
Accuracy	0–1	Compare with majority/simple baselines and class balance.	Balanced classes and symmetric error costs.
Precision	0–1	Choose target from false-positive cost and required recall.	False positives costly.
Recall	0–1	Choose target from false-negative cost and corresponding precision/FPR.	False negatives costly.
F1	0–1	Compare models at relevant thresholds; no universal cutoff.	Precision and recall are similarly weighted.
Balanced accuracy	0–1	Average class recall; inspect per-class recalls too.	Class imbalance.
ROC AUC	0.5 random, 1 perfect for ordinary binary ranking	Ranking quality; usefulness is domain-specific.	Threshold-free ranking.
AP / PR curve	No-skill level tied to prevalence	Compare with prevalence and the relevant recall region.	Imbalanced positive class.
MCC	−1 to 1	0 is chance-like; ±1 are perfect/directly inverted in binary classification.	Balanced summary using all confusion cells.

Regression Metrics (Quick Reference)

Interpretation rule: error magnitudes are scale- and application-dependent. Compare them with a simple out-of-sample baseline and with the size of errors the application can tolerate.

When to use:

RMSE → large errors deserve extra weight.
MAE → absolute errors map naturally to the decision problem.
RMSLE → non-negative targets where multiplicative/log-scale error is meaningful.

Metric	Reference	Interpretation	Characteristics
R²	1 perfect; 0 mean baseline; <0 worse than mean baseline	No universal “good” cutoff.	Can be negative out of sample.
Adjusted R²	Same response/data	Higher can support added predictors, but is not predictive validation.	Penalises added degrees of freedom.
RMSE	Baseline RMSE + domain tolerance	Lower is better for the same target/sample.	Weights large residuals strongly.
MAE	Baseline MAE + domain tolerance	Lower is better for the same target/sample.	Linear error penalty.

Statistical Testing Essentials

Hypothesis Testing Flow

State H₀ & H₁ → Pre-specify α (for example 0.05 when justified) → Compute p-value → Compare:

p < α → Reject H₀ (statistically significant)
p ≥ α → Fail to reject H₀; this does not prove H₀ is true

Common Tests

Test	Use Case	Key Assumptions
t-test	Compare 2 group means	For independent-samples tests: independent observations/groups and an appropriate sampling model. Student’s t-test assumes equal variances; Welch’s t-test does not.
ANOVA	Compare ≥ 3 group means	Independent observations and an appropriate residual/sampling model. Classical one-way ANOVA assumes equal variances; Welch ANOVA relaxes that assumption.
Chi-square	Test independence in categorical data	Adequate expected counts for the χ² approximation; use exact/simulation methods when sparse
Permutation Test	Randomisation / exchangeability-based test	The permutation scheme must be valid under the null and preserve pairing, clustering, time or other dependence structure as required.
Kolmogorov–Smirnov (KS)	Compare a sample to a reference distribution (1-sample) or compare two samples (2-sample)	For a one-sample KS test, the continuous reference CDF should be fully specified independently of the tested sample, unless an adjusted procedure is used when parameters are estimated. For the two-sample form, samples should be independent under the standard setup.
Levene’s test	Test equality of variances across groups (do we trust “equal variance” for t-test / ANOVA?)	Groups independent. Works reasonably well even when data are not normal.

Multiple Testing Corrections

Method	Controls	When to Use
Bonferroni	FWER (strict)	Few tests, false positives very costly
Benjamini–Hochberg	FDR (less strict)	Many tests (genomics, feature screening)

Model Selection Criteria

Rule-of-thumb only: small ΔAIC/ΔBIC values indicate similar support under their respective criteria, but conventional bands are approximations and AIC/BIC target different model-selection goals.

Criterion	Preference	Best For
AIC	Lower is better	Information-theoretic comparison of candidate models balancing fit and complexity
BIC	Lower is better	Parsimonious selection under BIC's asymptotic assumptions
Cross-validated Error	Lower is better	Resampling-based estimate of generalization error; use nested or separate evaluation when the same CV is used for tuning
Adjusted R²	Higher is better	Regression with multiple predictors

Interpretability Methods

These describe fitted-model behaviour / attribution, not causal effects.

Method	Scope	Key Insight
Permutation Importance	Global	Feature importance by performance drop
SHAP Values	Global + Local	Additive feature contributions
LIME	Local	Local surrogate model explanations

Top 10 Common Pitfalls

Using Accuracy on imbalanced data
Ignoring false negatives in medical/safety applications
Optimizing only Precision or Recall (neglecting the other)
Treating p-value as probability H₀ is true
Not correcting for multiple testing
Comparing R² across different datasets
Using RMSE when outliers are unimportant
Interpreting SHAP/LIME as causal effects
Selecting models based on tiny metric differences
Ignoring confidence intervals for metrics

Quick Decision Checklist

Before choosing metrics

What’s the business objective?
Balanced or imbalanced data?
Cost of false positives vs false negatives?
Need probability rankings or binary decisions?
Require interpretability?

After model evaluation

Check multiple metrics (not just one)
Examine confusion matrix / error patterns
Validate on holdout / test set
Consider confidence intervals
Compare to reasonable baselines

Essential Python Snippets

Classification Metrics

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score

acc     = accuracy_score(y_true, y_pred)
prec    = precision_score(y_true, y_pred)
rec     = recall_score(y_true, y_pred)
f1      = f1_score(y_true, y_pred)
roc_auc = roc_auc_score(y_true, y_proba)

Regression Metrics

from sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score

mae  = mean_absolute_error(y_true, y_pred)
rmse = root_mean_squared_error(y_true, y_pred)
r2   = r2_score(y_true, y_pred)

Statistical Tests

from scipy import stats

# Welch t-test (two independent groups; does not assume equal variances)
t_stat, p_value = stats.ttest_ind(group1, group2, equal_var=False)

# Chi-square test of independence for a contingency table
chi2, p_chi, dof, expected = stats.chi2_contingency(contingency_table)

# Normality: Shapiro–Wilk (combine with a Q–Q plot)
w_stat, p_shapiro = stats.shapiro(sample)

# 1-sample KS compares against a FULLY SPECIFIED continuous reference distribution.
# These parameters are pre-specified, not estimated from `sample`.
mu0, sigma0 = 0.0, 1.0
reference_cdf = stats.norm(loc=mu0, scale=sigma0).cdf
ks_stat, p_ks = stats.kstest(sample, reference_cdf)

# 2-sample KS: compare two independent samples' continuous distributions
ks2_stat, p_ks2 = stats.ks_2samp(sample1, sample2)

# Levene’s test for equality of variances
lev_stat, p_lev = stats.levene(group1, group2, group3)

Key Takeaways

No single metric tells the whole story → always use multiple.
Context is everything → choose metrics aligned with business goals.
Visualize → confusion matrices, ROC/PR curves, residual plots.
Uncertainty matters → report confidence intervals, not just point estimates.
Baseline comparison → always compare to simple benchmarks.

Based on the “Data Science Model Evaluation Metrics Dashboard” condensed sheet.

Validation, Uncertainty & Deployment

Core rule: a metric is only as trustworthy as the evaluation design that produced it. Choose the split, resampling scheme, threshold-selection procedure and uncertainty method to match how the model will actually be used.

1. Match validation to the data-generating structure

Ordinary i.i.d.-like prediction: stratified splits/CV can preserve class proportions when appropriate.
Grouped or repeated observations: keep related observations in the same fold.
Temporal prediction: train on the past and validate on the future; use rolling/blocked or otherwise time-aware splits rather than random leakage-prone folds.
Spatial/geographical generalisation: consider spatial blocking or genuinely separate regions.
Model/hyperparameter selection: use nested CV or a distinct validation layer when extensive tuning would otherwise contaminate the performance estimate.
External validity: when possible, evaluate on a genuinely separate population, site, period or acquisition process.

2. Keep every learned preprocessing step inside training folds

Imputation, scaling, feature selection, target encoding, oversampling/resampling, dimensionality reduction and learned feature engineering must be fitted using training data only, ideally inside a pipeline. Otherwise validation performance can be optimistically biased.

3. Select thresholds before the final test evaluation

Tune a decision threshold on training/validation data using the relevant error costs or operating constraints, then lock it before evaluating the final untouched test set. Selecting the threshold on the final test set makes the reported test performance optimistic.

4. Quantify uncertainty with a design-appropriate method

For i.i.d.-like test data, a stratified bootstrap is useful for many metrics; grouped or serially dependent data need cluster/block/time-aware resampling.
ROC AUC uncertainty is commonly estimated with bootstrap or DeLong-type methods; use a method compatible with the sampling design.
Repeated/CV score distributions can describe resampling variability, but ordinary fold-to-fold SD is not automatically a standard error or confidence interval.
Time-to-event metrics require censoring-aware uncertainty procedures and explicit time horizons.

5. Check subgroups, data quality and post-deployment shift

Inspect performance and uncertainty for substantively relevant subgroups where sample size permits. Also investigate label quality, missingness, duplicates/near-duplicates, sampling bias, prevalence changes, calibration drift and covariate/concept shift. Fairness criteria encode different normative goals and should be chosen from the application and harms, not mechanically.

How to Read the Color Bars

The colors are visual cues for direction or screening regions under the stated metric-specific guidance; they are not universal pass/fail grades.

Green: a more favourable direction or reference region under the stated context or heuristic.

Yellow: an intermediate or cautionary region that requires context.

Red: a less favourable or flagged region under the stated context or screening rule.

Important: most numeric bands on this dashboard are pedagogical rules of thumb, not universal acceptance thresholds. Interpret metrics relative to a relevant baseline, uncertainty, prevalence, decision costs and the standards of your domain. For hypothesis tests, a small p-value is evidence against a specified null hypothesis, not an intrinsically “good” outcome; a large p-value does not prove the null or an assumption true.

Performance (Classification)

Metric	Decision Criterion (Value Range)	Purpose	Description	Working Mechanism	Example	Limitations
Accuracy	No universal accuracy cutoff. Compare with a relevant baseline (for example majority-class accuracy), class balance, uncertainty, and the costs of the different errors. High accuracy can still be useless on an imbalanced task.	Gauge overall classification success rate. Useful for quick assessment on balanced data, but unreliable alone on strongly imbalanced data.	Proportion of all predictions that are correct: \[ \text{Accuracy} = \frac{TP + TN}{TP+FP+TN+FN}. \]	Counts correct vs total predictions with all errors weighted equally.	90 correct predictions out of 100 → accuracy = 0.90.	Can be very misleading for imbalanced data – a majority‑class predictor can have high accuracy but be useless. Common pitfalls: Comparing models by accuracy without checking class balance or baseline (e.g. majority‑class accuracy). Interpreting small accuracy differences as meaningful without considering confidence intervals or variability.
Precision (Positive Predictive Value, PPV)	Higher precision means fewer false positives among predicted positives, but there is no universal “good” cutoff. Choose the operating target from prevalence, false-positive cost, and the recall required at the deployed threshold.	Measures how reliable positive predictions are, crucial when false positives are costly.	\[ \text{Precision} = \frac{TP}{TP+FP}. \]	Improves as false positives decrease, even if recall suffers.	If 100 flagged spam emails contain 90 real spam, precision = 0.90.	Can be gamed by predicting very few positives; must be balanced with recall. Common pitfalls: Optimising precision alone and ending up with a model that rarely predicts positives (very low recall). Comparing precision across datasets with very different prevalence without context.
Recall (Sensitivity / True Positive Rate)	Higher recall means fewer true positives are missed, but there is no universal “good” cutoff. Choose the required recall from the cost of false negatives and inspect the corresponding precision and false-positive rate.	Measures completeness of positive detection; key when missing positives is costly.	\[ \text{Recall} = \frac{TP}{TP+FN}. \]	Improves when false negatives decrease.	Recall = 0.95 means 95% of true positives are found.	Ignores false positives; predicting everything positive yields recall 1.0. Common pitfalls: Maximising recall at the expense of an unacceptably high false‑positive rate. Interpreting high recall as “good model” without looking at precision or class prevalence.
Specificity (True Negative Rate)	Higher specificity means fewer false positives among actual negatives, but there is no universal “good” cutoff. Choose the target from false-alarm costs and the corresponding sensitivity/recall requirement.	Measures ability to correctly identify negatives; important when false positives are costly.	\[ \text{Specificity} = \frac{TN}{TN+FP}. \]	Complement of false positive rate: FPR = 1 − specificity.	Specificity 0.98 means 98% of real negatives are correctly left unflagged.	Trivially high if model predicts almost everything negative. Common pitfalls: Quoting high specificity while recall on the positive class is extremely low. Confusing specificity with NPV or accuracy when explaining results to stakeholders.
False Positive Rate / False Negative Rate (FPR / FNR)	Lower is better only relative to the application's error costs; there is no universal acceptable rate. Report both rates at the deployed threshold and show the threshold trade-off. Acceptable FPR/FNR can differ by orders of magnitude across applications.	Quantify false alarms (FPR) and missed positives (FNR).	\[ \text{FPR} = \frac{FP}{FP+TN}, \quad \text{FNR} = \frac{FN}{FN+TP}. \]	Threshold choice trades FPR vs FNR; ROC and PR curves illustrate this trade‑off.	A cancer test may tolerate FPR 0.15 for FNR 0.01 (very few missed cases).	Need domain‑specific cost trade‑offs; no single “correct” target. Common pitfalls: Optimising only one of FPR or FNR without considering the business/clinical cost of the other side. Reporting FPR without stating the decision threshold and the evaluation population/data distribution.
F₁ Score	Higher F1 is better for a fixed positive-class definition and thresholding setup, but there is no universal F1 quality threshold. Compare models at relevant operating thresholds. F1 weights precision and recall equally and does not use true negatives.	Single number that balances precision and recall, especially on imbalanced datasets.	\[ F_1 = 2 \frac{\text{Precision} \times \text{Recall}} {\text{Precision} + \text{Recall}}. \]	Harmonic mean, so dominated by the smaller of precision and recall.	Precision 0.9, recall 0.9 → F1 0.9; precision 0.9, recall 0.1 → F1 ≈ 0.18.	Weights precision and recall equally; may not match domain priorities. Common pitfalls: Using F1 when precision and recall have very different real‑world costs (e.g. medical diagnosis). Comparing F1 scores across datasets with very different class imbalance.
ROC AUC	For ordinary binary ranking, 0.5 is the random-ranking expectation and 1.0 is perfect ranking; values below 0.5 can occur by chance or indicate reversed ranking. Beyond those anchors, usefulness is domain-specific. Report uncertainty and evaluate the operating region that matters.	Measures how well model ranks positives above negatives across thresholds.	Equivalent to the probability a random positive has a higher score than a random negative, with score ties contributing half credit in the usual empirical AUC.	Integrates TPR vs FPR curve over all thresholds.	AUC 0.85 → across random positive–negative pairs, correct ordering receives about 85% average credit when ties count as half.	Doesn’t reflect calibration; can look good on heavily imbalanced data even if practical performance is poor. Common pitfalls: Using ROC AUC on extreme class imbalance where PR AUC is more informative. Assuming high AUC always implies good performance at the specific threshold used in production.
Negative Predictive Value (NPV)	Higher NPV means predicted negatives are more often truly negative, but there is no universal cutoff. NPV depends strongly on prevalence, so evaluate it in the deployment population and alongside sensitivity/specificity.	Probability that a predicted negative is truly negative; key when false negatives are costly.	\[ \text{NPV} = \frac{TN}{TN+FN}. \]	Negative predictive counterpart to precision/PPV.	In disease screening with low prevalence, NPV often very high even for moderate models.	Strongly dependent on prevalence; high NPV doesn’t automatically imply a good model. Common pitfalls: Interpreting high NPV as strong evidence of good model quality in very low‑prevalence settings where almost everyone is negative. Confusing NPV with specificity when explaining metrics.
Balanced Accuracy	Higher is better, but there is no universal quality cutoff. For K classes: \[\text{Balanced Accuracy}=\frac{1}{K}\sum_{k=1}^{K}\text{Recall}_k.\] For binary classification this is $(\text{TPR}+\text{TNR})/2$.	Summarise class-wise recall without letting a large majority class dominate.	Average the recall obtained on each class; valid for binary and multiclass classification.	Each class contributes equally regardless of prevalence. A chance-adjusted variant can map random performance to 0.	Binary TPR = 0.9 and TNR = 0.5 gives balanced accuracy 0.7; inspect both components because the average hides which side is weak.	No scalar reveals which classes are weak. “Chance = 0.5” is a binary special case, not a universal multiclass baseline.
Brier Score	Lower is better, but absolute values require a reference forecast. \[\text{Brier}=\frac{1}{N}\sum_i(\hat p_i-y_i)^2.\] A useful relative comparison is \[BSS=1-\frac{BS_{model}}{BS_{reference}}.\] BSS > 0 improves on the chosen reference; BSS = 0 matches it.	Evaluate overall quality of probabilistic binary predictions with a proper scoring rule.	Mean squared probability error; rewards accurate probabilities and penalises confident mistakes.	The Brier score can be decomposed into reliability (calibration), resolution and uncertainty; a low score alone is not proof of good calibration.	If prevalence is 10%, always predicting 0.10 has Brier 0.09. Judge a model against that or another relevant reference, not a universal threshold.	Depends on outcome prevalence and reference problem; use calibration plots when calibration itself is the question and ranking metrics for discrimination. Common pitfalls: Calling a fixed Brier value “good” without a baseline. Assuming low Brier guarantees either good calibration or good ranking.
Calibration Error (ECE)	Lower is better only for a specified ECE estimator/binning scheme; there is no universal ECE cutoff. Report the estimator definition, sample size and a calibration plot, and compare models on the same evaluation data.	Measures mismatch between predicted probabilities and observed frequencies.	Bins predictions by confidence; compares average predicted probability to empirical frequency in each bin, then averages absolute differences.	A low ECE means small average bin-level calibration discrepancy under the chosen estimator; coarse bins or limited data can still hide local miscalibration.	Used for risk models in credit, medicine, etc. as a complement to AUC.	Depends on binning; can be unstable with small sample sizes. Common pitfalls: Using a single ECE value without inspecting calibration plots per probability region. Ignoring that ECE can be low even when model ranking (AUC/PR AUC) is poor.
Confusion Matrix	A confusion matrix is not a scalar score, so there is no universal good/bad diagonal ratio. Inspect class-specific counts and row/column-normalised rates, especially for minority classes and cost-asymmetric errors.	Summarise classification results in terms of true positives, false positives, false negatives, and true negatives for each class.	Matrix whose rows are actual classes and columns are predicted classes (or vice versa); each cell counts how often that combination occurs.	From predictions and labels, fill contingency table; derived metrics like precision, recall, F1, MCC are computed from its cells.	A binary confusion matrix with large TP and TN and very small FP/FN indicates a strong classifier; see interactive explorer below.	Not a single scalar; becomes large for many classes and may be hard to read without normalisation. Common pitfalls: Looking only at totals without normalising per row/column, which hides minority‑class errors. Comparing confusion matrices across datasets with different sizes without converting to rates.
Precision–Recall Curve (PR curve; AP / PR AUC summaries)	Interpret relative to positive-class prevalence π and the recall region that matters operationally. A random/no-skill ranking has expected precision near π. Average Precision (AP) and trapezoidal PR AUC are different summaries.	Assess the precision–recall trade-off across thresholds, especially with an imbalanced positive class.	The PR curve plots precision against recall as the decision threshold changes.	AP weights precision by increments in recall. Trapezoidal PR AUC numerically integrates the curve with linear interpolation; name the summary explicitly.	With 0.5% positives, AP = 0.65 can be very strong relative to a no-skill level near 0.005.	Strongly prevalence-dependent. Do not compare across differently sampled datasets without context, and do not label AP as “PR AUC” without saying which summary is used.
Cohen’s Kappa	-1 0 0.6 1 Agreement beyond chance (−1 to 1). The labels below are a traditional descriptive convention, not universal quality thresholds. ≤ 0 = no better (or worse) than random. 0.01–0.40 = slight–fair agreement. 0.41–0.60 = moderate. > 0.60 = substantial to almost perfect agreement.	Measure inter‑rater reliability or agreement between model predictions and labels while adjusting for agreement expected by chance.	Compares observed accuracy $p_o$ to expected accuracy $p_e$ under random agreement given class marginals.	\[ \kappa = \frac{p_o - p_e}{1 - p_e}. \] Large κ implies agreement much higher than chance, small κ close to or below 0 implies near‑random agreement.	Often used to compare two human labelers, or model vs clinician, in medical imaging or annotation tasks.	Sensitive to prevalence and marginal distributions; κ can be low even with high observed accuracy in imbalanced datasets. Common pitfalls: Interpreting low κ as “bad model” without considering skewed class frequencies or label noise. Applying generic qualitative cutoffs (e.g. “good” at 0.6) without domain‑specific context.
Matthews Correlation Coefficient (MCC)	−1 ≤ MCC ≤ 1. +1 = perfect agreement, 0 = chance-like/no association, and −1 = perfectly inverted binary prediction. Magnitude requirements are application-specific.	Balanced summary of classification agreement using the full confusion structure.	In the binary case MCC is the phi correlation between predicted and true labels.	\[ \text{MCC}=\frac{TP \cdot TN-FP \cdot FN} {\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}. \] A standard multiclass generalisation also exists.	Useful when accuracy is dominated by a large majority class.	The displayed TP/TN/FP/FN formula is specifically binary; multiclass MCC uses the full confusion matrix.

Interactive Confusion Matrix Explorer

Adjust TP / FP / FN / TN to see how the main metrics change (toy calculator only).

	Predicted Positive	Predicted Negative
Actual Positive	TP:	FN:
Actual Negative	FP:	TN:

Derived metrics from the current confusion matrix:

Accuracy: –
Precision (PPV): –
Recall (TPR): –
Specificity (TNR): –
F₁ score: –
False positive rate (FPR): –
False negative rate (FNR): –
Balanced accuracy (binary): –
Matthews correlation coefficient (MCC): –

Survival / Time-to-Event Performance

Use these metrics when the outcome is a time until an event (death, relapse, failure, churn) and some observations are censored (we only know the event has not happened yet by the end of follow-up).

When to use

Outcomes like “time from diagnosis to death”, “time to device failure”, “time until customer churn”.
Many people are still event-free at the end of the study (right-censoring).
We care about risk over time, not just a yes/no label at a fixed date.

Key tools & metrics

Kaplan–Meier curve – step-shaped curve showing the fraction still event-free over time. Great for visualising survival patterns and comparing groups.
Log-rank test – tests whether two or more Kaplan–Meier curves are systematically different over time.
Cox proportional hazards model – regression model for time-to-event data that estimates hazard ratios for predictors. Hazard ratios are model-based associations; a causal interpretation requires an appropriate causal design and additional assumptions.
C-index (concordance index) – rank-based discrimination based on concordance among comparable subject pairs. The exact estimator determines how censoring and ties are handled; for common formulations, 0.5 is often a no-discrimination reference and 1.0 is perfect concordance.
Time-dependent ROC / AUC(t) – ROC-style discrimination at specific time points (e.g. 1-year, 5-year AUC).
Brier score over time – censoring-aware mean squared error of predicted event probabilities at an explicit time horizon. Compare against a reference forecast; an integrated Brier score summarises performance across a chosen time interval, and time-dependent calibration plots complement the score.

Key assumptions and interpretation checks

Proportional hazards – standard Cox hazard-ratio interpretation assumes covariate hazard ratios are approximately constant over time; check this rather than assuming it.
Censoring mechanism – standard survival methods generally require censoring to be non-informative, often conditionally on included covariates.
Log-rank weighting – ordinary log-rank is most naturally aligned with proportional-hazards-type differences; crossing hazards can make a single p-value hard to interpret.
Metric definition – C-index and time-dependent AUC/Brier scores have censoring-aware variants; state the estimator and time horizon used.

Common pitfalls

Ignoring censoring – treating censored cases as if the event never happened biases estimates. Always use survival-aware methods (Kaplan–Meier, Cox, etc.).
Immortal-time bias – giving people “risk-free” time because they must survive long enough to receive a treatment or enter a group.
Competing risks – if other events (e.g. death from another cause) prevent the event of interest, ordinary Kaplan–Meier treatment of the event of interest can overstate cumulative incidence. Use competing-risk methods appropriate to the question; cause-specific and subdistribution hazards answer different questions.
Too short follow-up – if few events occur, any metric (C-index, Brier, log-rank) will be noisy and underpowered.

Inference & Hypothesis Testing

Metric / Test	Decision Criterion (Test Ranges)	Purpose	Description	Working Mechanism	Example	Limitations
p-value	Compare p with a pre-specified significance level α only when a dichotomous decision is required. p < α → reject H₀ under the chosen testing procedure. p ≥ α → fail to reject H₀; this does not establish that H₀ is true or that an effect is absent.	Quantify how incompatible the observed test statistic (or a more extreme one) is with a specified null model.	Probability, assuming H₀ and the test assumptions hold, of obtaining a test statistic at least as extreme as observed.	Derived from the null distribution; interpret alongside effect size, uncertainty, study design and multiplicity.	p = 0.03 may justify rejecting H₀ at α = 0.05, but does not mean H₀ has 3% probability or that the effect is important.	Does not measure effect size, practical importance or the probability a hypothesis is true; strongly affected by sample size and analysis choices. Common pitfalls: Interpreting p as the probability that H₀ is true. Equating “not significant” with “no effect”, or p < 0.05 with an intrinsically desirable outcome.
Adjusted α (Bonferroni Correction)	For m tests, Bonferroni uses α_adj = α/m for each test. Reject a null only when its valid p-value meets that threshold. By the Bonferroni inequality this guarantees FWER ≤ α without requiring independence. “Significant after correction” is evidence relative to the null/testing procedure, not an intrinsically good outcome.	Control family‑wise probability of any false positive across multiple tests.	Divides α by number of tests; each test uses α_adj as threshold.	Simple and conservative; Bonferroni controls FWER under arbitrary dependence among valid tests. Dependence can make the bound conservative.	20 tests with α=0.05 → α_adj=0.0025; only very small p‑values survive.	Can be overly conservative for large m, greatly reducing power. Common pitfalls: Applying Bonferroni mechanically in exploratory analyses where some false positives are acceptable. Treating α as the exact post-correction FWER rather than an upper bound; dependence can make the procedure substantially conservative.
Benjamini–Hochberg FDR	Choose a target FDR level q, sort p-values p₍₁₎ ≤ … ≤ p_(m), and find the largest k with p_(k) ≤ (k/m)q. Under the standard conditions for the BH procedure, this controls the expected false discovery proportion (FDR) at or below the target level; the realised fraction of false discoveries in one analysis can be above or below q.	Control the expected proportion of false discoveries among rejected hypotheses.	A step-up multiple-testing procedure that is typically less conservative than family-wise-error control.	Classical BH guarantees depend on the p-values and their dependence structure; arbitrary dependence may require a more conservative method such as Benjamini–Yekutieli.	Often used for high-dimensional screening where some false discoveries can be tolerated and quantified at the procedure level.	FDR is not the probability that any particular reported result is false, and q is not a guarantee about the realised false-discovery fraction in every dataset.
Type I & Type II Error	α is the Type-I error rate specified by the testing procedure; β depends on the true effect and design, with power = 1−β. Values such as α = 0.05 or power = 0.80/0.90 are common conventions, not universal optima. Choose them from error costs, minimum relevant effect, multiplicity, sample size and regulatory/domain requirements.	Frame trade‑off between false positives (Type I) and false negatives (Type II) when designing tests and studies.	Type I: reject true H₀. Type II: fail to reject false H₀. Power = 1 − β.	Power analysis couples α, β, effect size, and sample size.	A clinical trial might fix α=0.025 (one‑sided) and β=0.10 (90% power).	Reducing α without increasing sample size generally increases β. Common pitfalls: Choosing a very small α without increasing sample size, making studies underpowered. Focusing only on Type I error while ignoring the cost of missed true effects (Type II).
Statistical Power	Power is defined for a specified alternative/effect size and analysis plan. 80% or 90% are common planning conventions, not universal thresholds. Report the effect size the study was powered to detect and avoid “observed/post-hoc power” as an interpretation of a completed test.	Probability a test detects a true effect of given size.	Depends on α, effect size, variability, and sample size.	Higher n or larger effect sizes raise power.	Power 0.8 means 80% chance to detect the specified effect if it exists.	Post‑hoc power is usually uninformative; better to plan it a priori. Common pitfalls: Doing “observed power” calculations after non‑significant results and over‑interpreting them. Confusing low power with a higher nominal Type-I error rate. Low power mainly reduces sensitivity to a specified true effect; in selective literatures or screening settings, low power combined with low prior odds and significance filtering can still worsen the reliability and exaggeration of reported findings.
Z / t Tests	Interpret the statistic through its null distribution, degrees of freedom (for t), alternative hypothesis and p-value. A larger \|statistic\| is farther from the null in standard-error units, but absolute cutoffs such as 2 or 3 are not universal and do not measure practical effect size.	Test if a mean / difference / regression coefficient differs from a null value.	Statistic = (estimate − null) / SE, compared against normal or t distribution.	Large \|statistic\| means estimate is many SE away from null.	\|t\| = 4 with df≈50 usually implies p < 0.001 (strong evidence).	Assumes approximate normality and independence; sensitive to outliers. Common pitfalls: Using a Z‑test instead of a t‑test with small samples or unknown population variance. Ignoring multiple‑testing corrections when running many t‑tests in parallel.
Chi-Square Test (χ²)	Interpret χ² relative to its degrees of freedom and null distribution, not an absolute “high/low” scale. A small p-value is evidence against the specified null; report an effect size such as Cramér's V when magnitude matters.	Test independence (contingency tables) or goodness‑of‑fit of categorical data to expected counts.	\[ χ^2 = \sum \frac{(O_i - E_i)^2}{E_i}. \]	Large χ² means observed counts deviate strongly from expectations.	Used for testing independence between categorical variables or Mendelian ratios in genetics, etc.	With very large n, tiny practical differences become significant. Needs effect size (e.g. Cramér’s V) for magnitude. Common pitfalls: Applying the asymptotic χ² approximation to sparse expected counts without checking whether an exact or simulation-based method is more appropriate. Interpreting a significant χ² as evidence of large practical effect without reporting effect size.
ANOVA F-test	Interpret F using its numerator and denominator degrees of freedom and corresponding p-value; there is no universal F=2/3/5 significance scale. A significant omnibus result says at least one mean differs under the model assumptions; add effect sizes and multiplicity-aware follow-ups as needed.	Test if means of 3+ groups are all equal vs at least one differs.	Compares between‑group variance to within‑group variance.	Large F implies between‑group differences are large relative to noise.	Follow significant F with post‑hoc tests to identify which groups differ.	Requires an appropriate independent-observation/sampling structure and is sensitive to strong non-normality/outliers in small samples. Classical one-way ANOVA assumes equal variances; Welch ANOVA relaxes that equal-variance assumption. The omnibus test does not identify specific groups or effect sizes by itself. Common pitfalls: Stopping at a significant overall F without reporting effect sizes or post‑hoc comparisons. Ignoring heteroscedasticity or unbalanced designs where standard ANOVA assumptions fail.
Confidence Interval (e.g. 95% CI)	Interpret both location and width relative to meaningful values. For a two-sided 95% CI constructed from the same model/test, excluding the null (0 for a difference, 1 for a ratio) corresponds to rejection at α≈0.05. “Narrow” is scale- and decision-dependent.	Provide a range of plausible values for a parameter.	Usually estimate ± critical value × standard error.	Reflects both effect size and uncertainty.	Difference 5 with 95% CI [2, 8] suggests a clearly positive but moderately uncertain effect.	Relies on model assumptions; misinterpreted as containing the true value with 95% probability (frequentist CIs don’t strictly mean that).
Permutation Test	Interpret the permutation p-value under the exchangeability scheme actually used. Pre-specify α when a decision is needed. A small p-value is evidence against the permutation null, not an intrinsically “good” result; preserve pairing, clustering, time or other required dependence.	Randomisation/exchangeability-based significance test using an allowed permutation scheme.	Permute labels many times; recompute statistic to get null distribution and p_perm.	Useful for complex statistics where analytic null distributions are hard to derive.	Accuracy 0.8 vs permutation null mean 0.5 with p_perm=0.01 is strong evidence of real signal.	Computationally heavy; must respect structure (e.g. grouping or time). Common pitfalls: Permuting labels in time series or clustered data where observations are not exchangeable. Using too few permutations, leading to coarse p‑value resolution and unstable conclusions.
Effect Size (Cohen’s d)	0 0.2 0.5 0.8 >1 Common descriptive convention only: \|d\| < 0.2 = negligible. 0.2–0.5 = small. 0.5–0.8 = medium. > 0.8 = large effect.	Quantify standardized mean differences independent of sample size.	\[ d = \frac{\bar{x}_1 - \bar{x}_2}{s_{pooled}}. \]	Helps separate statistical from practical significance.	d = 0.6 means the group means differ by 0.6 pooled-SD units; practical importance depends on the outcome and domain.	Assumes similar SDs; thresholds are rough and context‑dependent. Common pitfalls: Using generic “small/medium/large” thresholds without considering what effect size is practically meaningful in context. Ignoring unequal variances where standard pooled‑SD formula is inappropriate.
Cliff’s Delta	-1 -0.5 0 0.5 1 Common descriptive convention only: \|δ\| < 0.147 = negligible. 0.147–0.33 = small. 0.33–0.474 = medium. > 0.474 = large effect (strong dominance).	Non‑parametric effect size based on ranks; robust to non‑normal data.	δ = P(X>Y) − P(Y>X), where X/Y from two groups; ranges −1..1.	Equivalent to rank‑biserial correlation; sign shows direction, magnitude shows strength.	With negligible ties, δ = 0.5 corresponds to P(X>Y)=0.75; interpret the practical importance in context.	Summarises ordering, not magnitude of differences; can be less intuitive than differences in means. Common pitfalls: Reporting δ without clarifying the underlying direction (which group is X vs Y). Assuming δ behaves like Pearson r or Cohen’s d in terms of interpretation thresholds.

Real‑World Scenarios for Common Tests

t‑test: compare mean blood pressure in treatment vs control groups in a clinical trial.
Paired t‑test: before/after measurements of the same patients after an intervention.
Chi‑square test: association between smoking status and lung‑disease incidence.
ANOVA F‑test: compare average click‑through rate across several ad creatives.
Permutation test: evaluate whether model accuracy exceeds chance by shuffling labels.
KS test: detect distribution shift between training and production feature distributions.

Statistical Tests & Assumptions – Quick Reference

What each test is asking, when to use it, and what can go wrong.

Big picture

Every test asks a question.
“Are these means equal?”, “Are these variances equal?”, “Do these two distributions look the same?”, “Is there autocorrelation?”.
p-value is not the effect size.
A tiny p-value can correspond to a tiny, unimportant effect if the sample is huge.
Assumptions matter.
Many tests assume things like normal residuals, equal variances, or independent observations. If those are badly violated, the p-values can be misleading.
Multiple testing inflates false positives.
If you run many tests, you need FWER/FDR control (Bonferroni, Holm, BH).

Distribution shape & normality

These tests check whether data or residuals look like they come from a particular distribution (usually normal). They are sensitive to sample size and to outliers.

Test	Main question	Typical use	Notes & pitfalls
Shapiro–Wilk	“Do these data look roughly normal?”	Small to moderate samples; for very large samples rely heavily on Q–Q plots and practical relevance	Powerful for normality; very sensitive to even small deviations in large samples. Always combine with plots (Q–Q plot, histogram). SciPy specifically cautions that for N > 5000 the W statistic is accurate but its p-value may not be.
Kolmogorov–Smirnov (KS)	“Is the sample distribution different from a reference distribution (or from another sample)?”	Comparing one sample to a known distribution, or two independent samples.	Works for continuous data; most sensitive near the centre, less in the tails. With estimated parameters, classical p-values need corrections.
Anderson–Darling	“Do these data follow a given distribution, especially in the tails?”	Checking normality with more tail focus than KS.	Gives more weight to tails than KS. As with other tests, large n ⇒ tiny deviations become “significant”.

Equality of variances

Many tests (for example classical t-test, ANOVA) assume similar variances across groups. These tests check that.

Test	Main question	Data type	Notes & pitfalls
Levene’s test	“Do these groups have equal variances?”	Continuous outcome, groups categorical	More robust to non-normal data than classical tests. A small p-value suggests at least one group has a different variance.
Brown–Forsythe	Levene’s test variant using medians instead of means.	Continuous outcome, heavy tails or outliers	Even more robust when distributions are skewed. Often preferred if outliers are expected.

Comparing means

The parametric tests below target group means under their model assumptions. Rank-based procedures such as Mann–Whitney and Kruskal–Wallis use orderings/ranks and generally test rank/distribution hypotheses; they are not simply “mean tests without normality”. Median or pure location-shift interpretations require additional shape/location assumptions.

Test	Main question	Design	Notes & pitfalls
t-test (independent)	“Are the means of two independent groups equal?”	Two groups, continuous outcome	Assumes normal residuals and (often) equal variances. For unequal variances, use Welch’s t-test.
t-test (paired)	“Is the mean difference between paired measurements zero?”	Before/after, matched pairs	Applied to differences. Assumes differences are roughly normal.
One-way ANOVA	“Are all group means equal?”	3+ groups, continuous outcome	Global test; if significant, follow with post-hoc comparisons (and multiple-testing correction). Requires independent observations under the study design; classical ANOVA assumes approximately normal residuals and equal variances, while Welch ANOVA relaxes equal variances.
Mann–Whitney U	“Do two independent groups differ in stochastic ordering / rank distribution?”	Two independent groups, ranked/continuous outcome	Non-parametric alternative to the independent t-test. Tests distribution shift, not strictly medians.
Kruskal–Wallis	“Do 3+ groups differ in their distributions (ranks)?”	3+ independent groups, ranked/continuous outcome	Non-parametric analogue of one-way ANOVA. If significant, follow with pairwise rank tests + multiple-testing correction.

Categorical data & independence

These tests work on counts in contingency tables and compare observed counts with expectations under a specified null model, such as independence or a specified categorical distribution.

Test	Main question	Typical table	Notes & pitfalls
Chi-square test of independence	“Are two categorical variables independent?”	R × C contingency table (for example treatment × outcome)	The χ² approximation needs adequate expected counts; for sparse tables use an exact, Monte Carlo, or otherwise appropriate small-sample method rather than a rigid one-number rule. Large samples make tiny deviations “significant”.
Chi-square goodness-of-fit	“Do observed category frequencies match a specified distribution?”	1 × C table (observed vs expected counts)	Used to compare observed counts to a theoretical or historical pattern.
Fisher’s exact test	“Is there association in a 2×2 table?”	2 × 2 table with small counts	For the usual 2×2 formulation, Fisher’s test conditions on fixed margins and avoids the large-sample χ² approximation. It is useful for sparse 2×2 tables, but it is not an automatic replacement for every sparse table; two-sided conventions can differ across implementations, and effect estimates/confidence intervals still matter.

Time-series residuals & autocorrelation

For time-ordered data, errors often correlate over time. These tests check whether residuals look “independent” or show systematic patterns.

Test	Main question	Typical use	Notes & pitfalls
Durbin–Watson	“Is there first-order autocorrelation in regression residuals?”	Linear regression on time-ordered data	Values near 2 ≈ no autocorrelation; near 0 ≈ strong positive autocorrelation; near 4 ≈ strong negative. Not designed for complex time-series models.
Ljung–Box	“Are a set of autocorrelations jointly zero?”	Checking whether residuals from a time-series model look like white noise.	Tests several lags jointly. A small p-value suggests remaining serial structure; choose lags deliberately and, for fitted time-series models, account for estimated model degrees of freedom where appropriate.

Summary: how to think about tests

Always pair tests with plots. QQ-plots, residual plots and histograms often tell the story faster than p-values.
Large samples detect tiny issues. A “significant” deviation may be practically irrelevant.
Small samples lack power. A non-significant result does not prove that assumptions are perfect or effects are zero.
Match the method to the violated assumption and estimand. Unequal variances often call for Welch’s t-test / Welch ANOVA rather than automatically switching to a rank test. Mann–Whitney and Kruskal–Wallis answer distribution/rank questions that are not identical to mean comparisons.
Remember multiple testing. Running many tests on the same data requires FWER/FDR control, otherwise false positives accumulate fast.

Robustness & Resampling

Robustness Playground: Mean vs Median & Outliers

Type any numbers, then drag the outlier slider. Watch how the mean swings while the median and IQR stay more stable. This is what “robustness to outliers” looks like in practice.

Data points (comma or space separated):

Extra outlier: 0

Try: 1, 2, 3, 4, 5 then add an outlier like +30. The mean moves a lot; the median barely moves. This playground illustrates how the mean and median react differently to extreme values. The median is more resistant to isolated outliers, while the mean uses every magnitude and estimates a different population quantity. Neither is universally "better"; the appropriate measure depends on the estimand, data-generating process and decision problem. In robust statistics, an estimator is considered more robust when a limited amount of contamination or a few extreme observations have less influence on its value. The mean is less resistant to isolated extreme values than the median, but that does not make it an inferior estimator in general.

Resampling Stability Playground (Bootstrap vs Jackknife)

This playground simulates a simple linear model with one true signal feature and one noise feature. Play with sample size, noise, and signal strength, then compare how a single fit, bootstrap, and jackknife disagree about the coefficient. Watch how CI width and sign stability change.

Sample size 100

Noise level (σ) 1.0

Signal strength (β₁) 1.5

Model: y = β₀ + β₁·x₁ + ε with an extra noise feature x₂ (true β₂ = 0).

Look for: when noise is high or n is small, single fit can be very misleading. Bootstrap and jackknife show how uncertain the coefficient really is.

How to read this playground

This Resampling Stability Playground compares two resampling methods – Bootstrap and Jackknife – for a simple linear model. It answers: “How stable is my estimated coefficient under resampling?”

Purpose

The model is y = β₀ + β₁·x₁ + ε with one true signal feature (x₁) and one pure noise feature (x₂, true β₂ = 0). By changing sample size, noise level, and signal strength, you can see when estimates are stable vs. when they are fragile.

How to use the controls

Sample size – more observations usually mean tighter intervals.
Noise level – higher noise makes estimates wobble more.
Signal strength – stronger β₁ is easier to detect reliably.
Generate new data – redraws a fresh dataset and new resamples.

Reading the plot

The vertical green dashed line shows the true β₁.
The orange dot is the single-fit estimate on the full sample.
The blue bar is the Bootstrap 95% CI for β₁.
The purple bar is the Jackknife normal-approximation 95% CI for β₁, using the standard delete-1 jackknife SE.

Reading the table

CI width – how wide the interval is (narrow = more precise).
Sign stable – fraction of resamples that keep the same sign as the mean.
The green/red tags use a custom teaching heuristic combining interval width and sign consistency; they are not formal statistical thresholds.

Use this to build intuition for resampling stability and precision: agreement between methods and narrower intervals can indicate less sampling sensitivity under the chosen resampling design. It does not rule out bias, leakage, dependence violations or model misspecification.

Bootstrap 95% CI Jackknife normal-approximation 95% CI Single-fit estimate

Demo: more stable / Demo: less stable use a custom visual heuristic, not a formal test.

Metric / Method	Decision Criterion	Purpose	Description	Working Mechanism	Example	Limitations
Cross-Validation Mean & Std (k-fold CV)	Compare the mean score with relevant baselines and competing models. Fold-to-fold standard deviation describes split sensitivity, but has no universal “stable” cutoff and is not automatically a standard error/CI because folds overlap and scores are dependent.	Estimate generalisation performance under a specified resampling scheme.	Repeatedly fit on training folds and evaluate on held-out folds.	The split design must match deployment: stratify when appropriate, keep groups together, use time-aware splits for temporal prediction, and put preprocessing/feature selection inside the CV pipeline.	Report individual/repeated-CV scores and compare paired results; use nested CV or an untouched test set when model/hyperparameter selection is substantial.	Ordinary random k-fold assumes exchangeable/i.i.d.-like observations. Fold SD alone is not an inferential confidence interval.
Bias–Variance Tradeoff	underfit balanced overfit High bias (underfit) → high error on train & test. Balanced bias–variance → low and similar train/test error. High variance (overfit) → very low train error, high test error.	Conceptual tool for selecting model complexity.	This is the classical intuition: insufficient flexibility can create bias and excessive effective flexibility can create variance. The relationship is not universally monotone in modern regularised/overparameterised models.	Analyse learning curves vs model capacity to find sweet spot.	Deep tree that fits training perfectly but fails on test is high‑variance.	Not a single numeric statistic; patterns can be subtle for deep models. Common pitfalls: Assuming that increasing model capacity always improves performance without monitoring overfitting. Using training error alone as proxy for generalisation error.
Jackknife Variability	For a smooth estimator, the delete-1 jackknife can estimate a standard error using the dispersion of leave-one-out estimates. Interpret the SE on the estimator's scale; there is no universal “low/high” cutoff. Large leave-one-out changes also reveal sensitivity to particular observations.	Approximate sampling variability and diagnose observation-level sensitivity for suitable estimators.	Recompute the estimator n times, each time omitting one observation.	The standard delete-1 jackknife SE rescales the dispersion of the leave-one-out estimates by $\sqrt{(n-1)/n}$; raw percentiles of those leave-one-out estimates are not a standard jackknife CI.	If omitting individual observations barely changes a smooth regression coefficient, its jackknife SE will usually be small relative to the coefficient's practical scale.	Can perform poorly for non-smooth statistics and dependent observations unless the resampling scheme is adapted. A small jackknife SE is not proof that the model is correctly specified.
Bootstrap CI Width	narrow medium wide A narrower bootstrap CI indicates greater precision only relative to the parameter’s meaningful scale and under a valid resampling design. Intermediate width = intermediate precision; “acceptable” is context-dependent. Very wide or irregular CI can flag high uncertainty or an irregular/unstable sampling distribution; inspect the resampling distribution rather than applying a universal width cutoff.	Non‑parametric uncertainty quantification for statistics and model parameters.	Resample with replacement, recompute estimator, and use empirical distribution.	Percentile or BCa intervals reflect sampling variability without assuming normality.	Narrow [4.1, 4.2] CI is very precise; [0, 20] shows extreme uncertainty.	Expensive for large models; assumes sample is representative. Common pitfalls: Bootstrapping data with temporal or grouped dependence without respecting structure. Using too few bootstrap samples, resulting in noisy interval estimates.
Feature Stability Across Resamples	Report selection/importance frequency for the specified resampling and selection procedure. Cutoffs such as 0.5 or 0.8 are project-specific heuristics, not universal statistical guarantees. Formal stability-selection methods have their own assumptions and error-control results.	Check whether feature conclusions are sensitive to sampling variation.	Repeat the complete selection/importance procedure over valid resamples and record selection or importance frequency.	Stability is conditional on the model, hyperparameters, threshold and resampling design.	A feature selected in 95/100 resamples is more repeatable under that procedure than one selected 20/100 times, but correlated features may swap roles.	Do not convert a selection frequency into causal importance or a universal false-discovery guarantee.

Regression & Correlation

Metric	Decision Criterion	Purpose	Description	Working Mechanism	Example	Limitations
R² (Coefficient of Determination)	R² is not restricted to 0–1. R² < 0 = worse than predicting the evaluation-set mean under the usual definition. R² = 0 = equal to the mean-prediction baseline. R² → 1 = predictions increasingly match observed outcomes. There is no universal weak/moderate/strong cutoff; acceptable R² depends on domain, noise and evaluation design.	Compare squared prediction error with the error from a mean-prediction baseline.	\[ R^2 = 1 - \frac{SS_{res}}{SS_{tot}}. \]	R² = 1 is perfect; R² = 0 matches the mean baseline; negative R² is worse than that baseline.	R² = 0.85 means residual sum of squares is 15% of the mean-baseline total sum of squares on that evaluated dataset.	Can be inflated in-sample by overfitting; does not imply causality or good out-of-sample performance. It can be negative, especially on held-out data. Common pitfalls: Equating high R² with causal explanation. Applying universal “good R²” thresholds across domains with different irreducible noise.
Adjusted R²	Adjusted R² can be negative. Compare it only among models fitted to the same response/data. Higher adjusted R² means the fit improvement offset the degrees-of-freedom penalty, but it is not an out-of-sample performance guarantee.	Compare models with different numbers of predictors while penalising complexity.	Adjusts R² downward for each extra degree of freedom.	Only increases when added variables meaningfully reduce residual variance.	If R² rises but adjusted R² falls when adding variables, they’re probably not helpful.	Can’t compare across different datasets; still doesn’t guarantee predictive performance. Common pitfalls: Using small differences in adjusted R² as decisive evidence between models. Ignoring other diagnostics (residual plots, multicollinearity) when adjusted R² looks acceptable.
RMSE (Root Mean Square Error)	Lower is better for the same target, units and evaluation sample. Interpret RMSE against a simple out-of-sample baseline and domain error tolerance; there is no universal fraction of target SD/range that is “good”.	Square root of the mean squared prediction error, expressed in the target's original units.	\[ \text{RMSE} = \sqrt{\frac{1}{n}\sum (y_i - \hat{y}_i)^2 }. \]	Squares errors (emphasising large ones), then square‑roots.	On the same held-out housing dataset, RMSE \$20k is lower than \$55k, but neither value is universally “good” or “poor”; compare with a baseline and the application’s error tolerance.	Highly sensitive to outliers; needs baseline to interpret magnitude. Common pitfalls: Comparing RMSE across targets with different scales instead of using relative metrics. Optimising RMSE when extreme outliers are less important than typical error magnitude (where MAE may be better).
MAE (Mean Absolute Error)	Lower is better for the same target, units and evaluation sample. Compare with a naive baseline and a domain-defined tolerable error; percentages of the target mean are not universal quality thresholds.	Average absolute prediction error; more robust than RMSE.	\[ \text{MAE} = \frac{1}{n}\sum \|y_i - \hat{y}_i\|. \]	Each error contributes linearly.	MAE of 1.5k means the average absolute error is 1.5k; whether that is acceptable depends on a relevant baseline and the application’s error tolerance.	Doesn’t strongly penalise rare huge errors. Common pitfalls: Using MAE when large outliers are mission‑critical, under‑penalising them. Interpreting MAE without relating it to the typical value or variance of the target.
MSE (Mean Squared Error)	Lower is better for the same target, units and evaluation sample, but there is no universal quality cutoff. Compare MSE with a suitable baseline and the application's error costs. Remember that MSE is expressed in squared target units.	Measure average squared prediction error when large residuals deserve extra weight.	\[ \text{MSE} = \frac{1}{n}\sum_i (y_i-\hat y_i)^2. \]	Squares each residual before averaging, so a few large errors can dominate.	On one fixed test set, MSE 400 is lower than MSE 900; whether either is acceptable depends on the target scale, baseline and decision problem.	Squared units are less directly interpretable than MAE/RMSE, and the score is sensitive to extreme residuals. Do not compare raw MSE values across differently scaled targets.
RMSLE (Root Mean Squared Logarithmic Error)	Lower is better only for the same non-negative target, evaluation sample and log1p definition; there is no universal cutoff. Use it when discrepancies on a multiplicative/log scale are substantively meaningful, not merely because the target is skewed.	Evaluate prediction error on a log1p scale for non-negative outcomes.	\[ \text{RMSLE} = \sqrt{\frac{1}{n}\sum_i\left[\log(1+\hat y_i)-\log(1+y_i)\right]^2}. \]	Transforms actual and predicted values with log1p, computes squared differences, averages them and takes the square root.	Useful for non-negative outcomes where a multiplicative discrepancy is more meaningful than an additive error in the original units.	Negative targets or predictions are invalid under the common definition. RMSLE is not a percentage error and should not be compared numerically with MSE/MAE/RMSE on the original scale.
Pearson Correlation (r)	-1 -0.5 0 0.5 1 Interpreting \|r\| using common rough conventions only (field- and task-dependent): < 0.3 = weak. 0.3–0.5 = moderate. > 0.7 = strong linear association.	Strength and direction of linear association between two numeric variables.	Standardised covariance: \[ r = \frac{\text{Cov}(X,Y)}{\sigma_X\sigma_Y}. \]	Ranges −1..1; sign gives direction, magnitude gives strength.	Height vs weight often r≈0.7–0.8.	Very sensitive to outliers; misses non‑linear relationships. Common pitfalls: Interpreting correlation as causation or assuming no hidden confounders. Quoting r without visualising scatter plots to check linearity and outliers.
Spearman Correlation (ρ)	-1 -0.5 0 0.5 1 Interpreting \|ρ\| using common rough conventions only (field- and task-dependent): < 0.3 = weak monotonic association. 0.3–0.5 = moderate. > 0.7 = strong monotonic relationship.	Correlation on ranks; robust to outliers and non‑linear but monotonic trends.	Compute ranks of X and Y, then Pearson r on ranks.	Captures relationships where one variable consistently increases/decreases with the other.	Useful for ordinal data or non‑linear monotonic relationships.	Near zero for non‑monotonic relationships (e.g. U‑shaped). Common pitfalls: Assuming Spearman detects arbitrary non‑linear patterns; it only captures monotonic tendencies. Not accounting for many ties in ranked data, which can affect estimates.
Partial Correlation	-1 -0.5 0 0.5 1 Interpreting \|r_partial\| using rough descriptive conventions only: ≈ 0 = little remaining association beyond controls. ≈ 0.3–0.5 = moderate residual link. > 0.5 = strong association beyond controlled factors.	Measure association between two variables while controlling for others.	Regress each variable on controls, then correlate residuals.	Describes the remaining linear association after adjusting for specified controls; it does not by itself identify direct or causal effects.	Correlation between exercise and blood pressure may shrink after controlling for age.	Only removes linear effects; can be unstable with many correlated controls. Common pitfalls: Interpreting partial correlation as proof of direct causal influence. Including too many collinear controls, leading to noisy or unstable estimates.
Regression Coefficients + CI	Interpret the coefficient on its scale and report its confidence interval. For a two-sided interval/test from the same model, a 95% CI excluding 0 corresponds to rejection at about α=0.05, but statistical significance is not practical importance and a wide interval can include materially important effects.	Interpret predictor effects and uncertainty in regression models.	Coefficients describe expected change in response for unit change in predictor, holding others fixed; CI shows uncertainty.	Based on estimated SEs and t / normal critical values.	“Each extra year of experience adds \$2k (95% CI \$1.5k–\$2.5k)” is clear and interpretable.	Interpretation assumes correct model form and no severe multicollinearity. Common pitfalls: Interpreting coefficients from poorly specified models (e.g. missing confounders) as causal. Ignoring the width of CIs and focusing only on significance.
Durbin–Watson Test	0 2 4 < 1.5 = likely positive autocorrelation (bad for OLS SEs). 1.5–2.5 = a common rough screening band around 2; formal interpretation depends on the design and critical values. > 2.5 = possible negative autocorrelation.	Detect first‑order serial correlation in regression residuals.	DW ≈ 2(1 − ρ₁), where ρ₁ is lag‑1 autocorrelation.	Values far from 2 suggest residual dependence.	DW = 0.9 suggests strong positive autocorrelation; consider time‑series models or GLS.	Primarily detects first‑order correlation; interpretation uses critical value tables or approximations. Common pitfalls: Applying Durbin–Watson to models with lagged dependent variables where its distribution changes. Ignoring serial correlation even when DW indicates strong dependence, leading to underestimated SEs.
Breusch–Pagan Test	0 0.05 1.0 p < 0.05 → reject homoscedasticity; heteroscedastic errors (bad for standard OLS SEs). p ≥ 0.05 → no strong evidence of non‑constant variance.	Test for heteroscedasticity (non‑constant variance) in regression.	Regress squared residuals on predictors; statistic ~χ² under constant variance.	Significant p suggests need for robust SEs, transforms, or alternate models.	Often used in linear regression diagnostics.	Power depends on auxiliary regression specification; may miss complex patterns. Common pitfalls: Assuming homoscedasticity solely because the Breusch–Pagan p‑value is slightly above 0.05. Using standard OLS SEs despite strong evidence of heteroscedasticity.
Shapiro–Wilk Normality Test (on residuals)	0 0.05 1.0 Null: residuals are normal. p < 0.05 → reject normality (assumption violation). p ≥ 0.05 → no strong evidence against normality.	Check normality assumption for regression residuals.	Statistic W measures agreement between ordered residuals and expected normal order statistics.	Used with residual plots to assess normality assumption for t‑based inference.	p = 0.4 gives no strong evidence against residual normality; p = 0.001 suggests heavy tails / skew.	Very powerful for large n (tiny deviations flagged); doesn’t tell how residuals deviate. Common pitfalls: Overreacting to tiny deviations from normality in large samples where CLT‑based inference is still robust. Relying solely on the test without inspecting Q–Q plots.

Comparative Table – When to Use MAE vs RMSE

Scenario	Prefer MAE	Prefer RMSE
Robustness to outliers	Yes – if occasional extreme errors are not critical.	Usually less robust – RMSE gives disproportionately more weight to large residuals.
Penalising large errors heavily	No – treats all deviations linearly.	Yes – squared errors heavily punish large mistakes.
Interpretability	“On average, we are off by …” is intuitive.	Less intuitive, but mathematically convenient for optimisation.
Gradient‑based optimisation	Non‑differentiable at 0 but workable.	MSE is the smooth quadratic loss and is strongly convex in the predictions. RMSE is a monotone transformation of MSE with the same fixed-sample minimiser, but it is not itself the quadratic loss.
Highly skewed targets	Sometimes combined with median‑based models.	Consider a log transform/RMSLE only when a multiplicative error scale and non-negative target are substantively appropriate.

Multicollinearity Diagnostics

Metric	Decision Criterion	Purpose	Description	Working Mechanism	Example	Limitations
Variance Inflation Factor (VIF)	1 5 10 20+ Common screening heuristics only: VIF ≈ 1 → little linear redundancy with the other predictors. 5–10 = moderate concern. > 10 = serious multicollinearity.	Quantify how much variance of a coefficient is inflated by linear dependence with other predictors.	\[ \text{VIF}_j = \frac{1}{1-R_j^2} \] where R_j² from regressing predictor j on others.	Large R_j² → large VIF → unstable coefficient for that predictor.	VIF = 12 suggests coefficient may be poorly estimated and highly sensitive to small data changes.	Doesn’t indicate which predictors are collinear with each other; only that some redundancy exists. Common pitfalls: Dropping variables solely because VIF is above a rule‑of‑thumb threshold without considering domain meaning. Confusing VIF with scale-dependent conditioning diagnostics: with an intercept, ordinary VIF is invariant to simple centering/rescaling of predictors.
Condition Number	1 10 30 60+ Common scale-dependent screening heuristic: < 10 = relatively low condition number on the chosen scaling. 10–30 = moderate. > 30 = severe (near singular matrix).	Measure overall multicollinearity of predictor matrix.	Ratio of largest to smallest singular value (or sqrt of eigenvalue ratio).	Large condition number means XᵀX is ill‑conditioned; coefficient estimates can be unstable.	On an appropriately scaled design, condition number ≈ 50 can indicate severe ill-conditioning and possible strong collinearity among predictors.	Scaling affects value; interpret with standardised predictors. Does not pinpoint which variables are problematic. Common pitfalls: Ignoring collinearity when condition number is large but VIFs seem moderate. Comparing condition numbers across models where predictors are scaled differently.

Outlier & Distribution Metrics

Metric/Test	Decision Criterion	Purpose	Description	Working Mechanism	Example	Limitations
Skewness (Distribution Asymmetry)	-3 -1 0 1 3 Descriptive heuristic only: Between -1 and +1 = modest skew; whether that matters depends on the analysis/model. Between -2 and -1 or 1 and 2 = moderate skew. < -2 or > 2 = strong skew; consider transform or robust methods.	Quantify asymmetry of a distribution (left vs right tail).	Third standardized moment; sign indicates direction of long tail.	Positive skew → long right tail; negative → long left tail.	Incomes are typically right‑skewed; log‑incomes are closer to symmetric.	Unstable in small samples; easily influenced by a few extreme points. Common pitfalls: Using skewness alone to justify transformations without visual inspection. Interpreting minor non‑zero skewness in large samples as serious model violation.
Kurtosis (Tailedness)	-2 0 2 (Using excess kurtosis, normal ≈ 0.) Descriptive heuristic only: Excess kurtosis near 0 = a normal-like fourth moment; it does not by itself establish normal tails. Between -2 and -0.5 = somewhat light‑tailed. > 2 = heavy tails / many extreme values.	Describe how heavy the tails are compared to normal.	Fourth standardized moment minus 3.	High kurtosis indicates variance dominated by rare large deviations.	Financial returns often show high positive kurtosis.	Very sensitive to outliers; hard to interpret without skewness and plots. Common pitfalls: Attributing all high kurtosis to “fat tails” instead of checking for data quality or structural breaks. Using kurtosis for tiny samples where estimates are extremely noisy.
Shapiro-Wilk Test (Normality test)	0 0.05 1.0 Null: data are normal. p < 0.05 → reject normality. p ≥ 0.05 → no strong evidence against normality.	Formal test of normality for small–moderate samples.	Statistic W measures agreement between ordered data and expected normal quantiles.	p-value derived from W; small p indicates deviation from normality.	p=0.08 → no strong evidence against normality; p=0.001 → strong evidence of deviation.	Very sensitive with large n; use with plots and domain context. Common pitfalls: Automatically transforming data because p < 0.05 even when deviations are minor and models are robust. Using the test on discrete or heavily censored data where normality is impossible.
Z-score Outlier Detection	-3 -2 0 2 3 Under an approximately normal reference model, these are common screening conventions rather than automatic outlier rules: \|z\| < 2 = not unusually far from the mean. 2 ≤ \|z\| ≤ 3 = borderline outlier. \|z\| > 3 = potential outlier.	Flag univariate outliers relative to mean and SD.	\[ z = \frac{x - \mu}{\sigma}. \]	Extremely large \|z\| values are unlikely under a normal model.	z = 5 is extremely unusual (probability < 10⁻⁶ under normal).	Assumes normality; heavy‑tailed data produce many \|z\|>3 that are not truly abnormal. Mean/SD can be distorted by the outliers. Common pitfalls: Using z‑score thresholds on clearly non‑normal data (e.g. Pareto‑like heavy tails). Removing points based only on z‑scores without investigating data quality or domain context.
IQR Method (Tukey’s Fences)	LF Q1 Med Q3 UF Within [Q1−1.5·IQR, Q3+1.5·IQR] = not flagged by the standard Tukey-fence rule. Outside 1.5·IQR but within 3·IQR = moderate outlier. Beyond 3·IQR fences = extreme outlier.	Non‑parametric, robust rule of thumb for univariate outliers.	Uses quartiles and interquartile range (IQR = Q3–Q1).	Points outside whiskers in a boxplot correspond to Tukey outliers.	Common default rule in statistical software boxplots.	Skewed distributions may produce many flagged points; rule is heuristic and dimension‑wise only. Common pitfalls: Applying standard 1.5×IQR rule to strongly skewed data without adjustment. Dropping all outliers automatically instead of investigating their cause.
Robust Z-score (MAD)	-3 -2 0 2 3 For a roughly symmetric unimodal reference distribution, common screening heuristics include: \|z_MAD\| < 2.5 = not strongly flagged. 2.5–3.5 = borderline. > 3.5 = strong outlier candidate.	Detect outliers robustly using median and median absolute deviation.	Let MAD_raw = median(\|x − median\|). Under approximate normality, 1.4826·MAD_raw estimates σ, so a normal-consistent robust score is (x − median)/(1.4826·MAD_raw).	More stable than classical z‑score in presence of outliers.	More resistant to isolated extremes than mean/SD-based z-scores, but a symmetric MAD rule can still misclassify observations in strongly skewed or multimodal data.	Still assumes a roughly unimodal distribution; threshold choices are heuristic. Common pitfalls: Using robust z‑scores but still computing MAD on a mixture of very different populations. Believing robust methods remove the need for visual inspection or domain knowledge.
Cook’s Distance	0 0.5 1 2+ Cook's D uses screening heuristics, not a universal deletion threshold. Values near/above 1 are often inspected; rules such as 4/n are also used. Investigate influential observations instead of deleting them mechanically.	Measure influence of each observation on regression fit.	Combines leverage and residual size to approximate change in all fitted values when a point is removed.	Large Cook’s D means the observation strongly affects estimates.	One point with D=1.5 while others < 0.1 indicates a dominating data point.	Thresholds are rules of thumb; influential points may be valid data, not necessarily errors. Common pitfalls: Automatically deleting points with Cook’s D above a threshold without checking if they are legitimate. Ignoring the leverage–residual decomposition that explains why points are influential.
Mahalanobis Distance	0 χ²_p,0.975 χ²_p,0.99 extreme For p dimensions: MD² ≤ χ²_p,0.975 = inside main cloud. Between χ²_p,0.975 and χ²_p,0.99 = potential multivariate outlier. MD² > χ²_p,0.99 = strong multivariate outlier.	Detect multivariate outliers accounting for correlations between variables.	\[ MD(x) = \sqrt{(x-\mu)^\top \Sigma^{-1} (x-\mu)}. \] Squaring MD gives a χ² statistic under multivariate normality.	Points with large MD² lie far from the multivariate mean in whitened space.	Useful for anomaly detection in multi‑feature settings.	Requires good estimates of μ and Σ; classical covariance is itself distorted by outliers; robust covariance estimators may be needed. The simple χ² reference is most direct under multivariate normality with fixed population parameters; estimating μ and Σ from the same finite sample changes the exact reference distribution. Common pitfalls: Computing Mahalanobis distance with non‑invertible or poorly conditioned covariance matrices. Applying χ² cutoffs to data that are far from multivariate normal.
Kolmogorov–Smirnov Test (KS)	0 0.05 1.0 Two‑sample KS for distribution shift: p < 0.05 → distributions differ significantly (potential shift / mismatch). p ≥ 0.05 → no strong evidence of difference. Larger D (0–1) = stronger discrepancy.	Compare empirical distributions (e.g. train vs production) or sample vs theoretical distribution.	KS statistic D = sup \|F₁(x) − F₂(x)\| over x.	p‑value derived from D; sensitive to location and shape differences.	Useful in monitoring feature distribution drift over time.	More sensitive near median than in tails; assumes continuous data and independent samples. Common pitfalls: Using KS on discrete or heavily binned data without an appropriate variant. Interpreting a non‑significant KS as proof that distributions are identical (rather than “no strong evidence of difference”).

Influence & Robustness Lab (Interactive Playgrounds)

These playgrounds show how single points, noise and multiple comparisons can quietly break models, even when headline metrics still look good.

Outlier Impact 2.0 – How one point can twist a regression line

Controls

Sample size (base points) 30

Noise level (σ) 1.0

Outlier height 6.0

True model: y = 1 + 1.2·x. Outlier shares the same x-range, but you can move it far up or down.

How to read this playground

This playground shows how a single outlier can dramatically change an ordinary least-squares regression line.

Purpose

We simulate a simple linear model and compare two fits: one without the outlier (baseline) and one with the outlier. If a single point can flip the slope or change it a lot, your model is fragile.

How to use the controls

Sample size – more points ⟹ harder to twist the line.
Noise level – more noise hides the clean relationship.
Outlier height – drag far up/down and watch the orange line tilt.
Regenerate – new random base cloud with same settings.

Interpretation

Blue dots: base data. Red dot: outlier. Blue line: fit without outlier.
Orange line: fit including outlier.
In the table, Δ slope and the custom visual tag fragile/stable tell you how influential the point is.

Base points Outlier Fit without outlier Fit with outlier

Model	Slope	Intercept	R²	Δ slope	Stability

If one point can flip the conclusion, you don’t have a stable finding – you have an anecdote dressed up as a model.

Cook’s Distance & Leverage – Influence of a single high-leverage point

Controls

Leverage (x position) 2.5

Residual offset (signed) 2.0

Base model: same y = 1 + 1.2·x with noise. The purple point is the high-leverage candidate.

How to read this playground

Cook’s Distance combines two ideas: leverage (how unusual x is) and residual (how badly the point is fit). A point with both high leverage and large residual is highly influential.

What you see

Blue dots: regular data used to fit the baseline regression line.
Purple dot: candidate point at the chosen x-position and residual.
The table shows leverage, an internally studentized residual, Cook’s Distance, and qualitative screening flags (OK vs influential).

Heuristics

Common screens include Cook’s D > 4/n (worth inspection) and D around/above 1 (often large); neither is an automatic deletion rule.
High leverage with tiny residual can still be dangerous if a future small mistake there would flip the slope.

Base points Candidate point Baseline fit Fit with candidate

Quantity	Value	Interpretation

The candidate is included in the full model; Cook’s Distance then asks how much that fitted model would change if the candidate were deleted. High leverage plus a large full-model residual produces the strongest influence.

Noise Injection Playground – How noise quietly kills stability

Noise controls

Gaussian feature noise 0.3

Random label flips 0.05

Feature noise (jitter) 0.2

# Irrelevant features 5

This panel is an illustrative response-surface toy: it does not fit a classifier or run cross-validation. Its numbers are synthetic intuition scores, not estimated accuracy, CV variance or a validated stability metric.

Effect on performance & stability

Illustrative clean-data score 0.90

Illustrative train score 0.90

Illustrative held-out score 0.86

Illustrative variability score 0.01

Custom toy stability heuristic (0–1) 0.80

Verdict moderate robustness

How to read this playground

Increase noise and watch train accuracy stay high while CV accuracy drops and variance inflates.
Many irrelevant features increase overfitting pressure even if the core signal is unchanged.
The displayed stability number is a custom visual heuristic combining the toy held-out score and variability term; it is not a published stability estimator.

Moral: always think in terms of “signal vs noise”. Robust models maintain high CV accuracy and low instability even when noise rises.

Bonferroni & VIF – Why many tests and correlated features can fool you

In real projects we almost never test just one thing. We try many features, model variants, time points, segments, and outcomes. Every extra test is another chance to see a “significant” result that is actually just noise.

1. Multiple testing & Bonferroni

For a valid test whose null hypothesis is true, testing at α = 0.05 gives a 5% Type-I error probability under the testing procedure. If many true null hypotheses are tested at the same uncorrected threshold, the chance of making at least one false rejection can grow quickly.

1 valid test of a true null at α = 0.05 → 5% Type-I error probability.
100 independent valid tests with all nulls true at α = 0.05 → about 99.4% chance of at least one false rejection.

What is family-wise error (FWE / FWER)?

Family-wise error (FWE), often called the family-wise error rate (FWER), is the probability of making at least one Type I error (false positive) when performing a “family” of statistical tests. When you run many tests, this probability increases. FWE / FWER is used to quantify and control this risk, usually by adjusting the significance level or using corrections such as Bonferroni or Holm.

What Bonferroni does

Bonferroni is a conservative safety brake:

New per-test α = original α ÷ number of tests.

Example: with α = 0.05 and 100 tests, the Bonferroni-corrected threshold becomes 0.05 ÷ 100 = 0.0005 for each test. The Bonferroni inequality guarantees FWER ≤ 5% for valid tests without requiring independence, though the procedure may be conservative.

Pros: simple; guarantees FWER ≤ α for valid tests under arbitrary dependence.
Cons: conservative; with many tests it can hide real signals.

The playground shows how FWER explodes with many tests when you don’t correct, and how Bonferroni pulls it back under control.

2. VIF – When features tell the same story

In the right-hand panel, we are not testing many hypotheses, but we are using many correlated features in a regression. VIF explains how this hurts the stability of your coefficients.

Intuition:

The slider "Correlation between similar features" (ρ) says how strongly a group of features move together.
The slider "# of similarly correlated features" (k) says how many features sit in that correlated pack.

VIF is defined as VIF = 1 / (1 − R²) when regressing one feature on the others. It tells you how much the variance of a coefficient is inflated because of collinearity.

VIF ≈ 1 – little linear redundancy with the other predictors.
VIF 5–10 – a common screening range that often motivates closer inspection; context and model purpose matter.
VIF > 10 – a common severe-screening heuristic, not an automatic variable-deletion rule.

If ρ and k are high, the model cannot cleanly separate which feature carries the signal. Coefficients may swing wildly, flip sign, or become impossible to interpret, even though the underlying relationship hasn’t changed.

3. Big picture

Both panels demonstrate the same core idea in different ways:

Multiple testing – too many chances to “win” by noise → false discoveries.
Collinearity – too many overlapping features → unstable estimates.

In short: noise + many decisions = unreliable statistics. The corrections and diagnostics you see here (Bonferroni, FWER, VIF) are tools to keep that under control.

Bonferroni & VIF Intuition – Multiple tests and collinearity

Bonferroni Correction – Family-wise error

# of tests (m) 20

Per-test α (uncorrected) 0.05

FWER before correction (if tests independent) 0.64

Bonferroni α_corr = α / m 0.0025

Bonferroni FWER upper bound (≤ α) 0.05

Verdict illustrative screen

Every extra test is another chance to see “significance” by luck. Bonferroni shrinks the per-test α so FWER is bounded at or below the chosen family-wise α. The “before” formula shown here assumes independent tests; the Bonferroni bound itself does not.

Try m = 1 vs m = 100 with α = 0.05 and feel how uncorrected FWER explodes.

VIF Intuition – How correlation inflates variance

Correlation between similar features (ρ) 0.7

# of similarly correlated features (k) 3

R² for the equicorrelated predictor setup 0.61

VIF = 1 / (1 − R²) 2.58

Interpretation demo screen

VIF tells you how much the variance of a coefficient is inflated by collinearity with other predictors.

VIF ≈ 1 – little linear redundancy with the other predictors.
VIF 5–10 – a common screening range that often motivates closer inspection; context and model purpose matter.
VIF > 10 – a common severe-screening heuristic; investigate the design/features rather than deleting variables mechanically.

This toy setup assumes the focal predictor and all k companion predictors are mutually equicorrelated at ρ. Increase ρ or k and watch coefficient variance inflate.

Model Selection Criteria

Metric	Decision Criterion	Purpose	Description	Working Mechanism	Example	Limitations
Akaike Information Criterion (AIC)	0 2 10 20+ Compare ΔAIC relative to best (lowest). ΔAIC < 2 = essentially equally good. 2–10 = some to strong evidence against. > 10 = much worse than best model.	Trade off fit vs complexity for out‑of‑sample prediction quality.	\[ \text{AIC} = 2k - 2\log(L), \] k = parameters, L = likelihood.	Lower AIC preferred among models fit to same data/response.	Models with ΔAIC < 2 are often considered similarly plausible.	Asymptotic; for small n, AICc is preferable. Not directly interpretable in absolute terms. Common pitfalls: Treating small AIC differences (< 2) as meaningful when they are usually negligible. Comparing AIC values across different datasets or response variables.
Bayesian Information Criterion (BIC)	0 2 10 20+ ΔBIC vs best: < 2 = weak evidence against. 2–6 = positive evidence against. 6–10 = strong. > 10 = very strong evidence against model.	More heavily penalise complexity, tending to pick more parsimonious models as n grows.	\[ \text{BIC} = k \log(n) - 2\log(L). \]	Approximate Bayes factor under certain priors; lower is better.	BIC often selects smaller subset of predictors than AIC in large samples.	Assumes true model is in candidate set; may underfit when prediction, not true model recovery, is goal. Common pitfalls: Using BIC to choose between models when the goal is pure prediction rather than parsimony. Comparing BIC across datasets with different numbers of observations.
Mallows’ C_p	Let p denote the number of fitted parameters (including the intercept under the usual convention). Look for small C_p values near p. C_p much larger than p suggests appreciable bias/lack of fit. Values below p can occur from sampling variation and are not, by themselves, evidence of overfitting.	Screen candidate subset linear-regression models for a bias–variance trade-off relative to a full-model error-variance estimate.	Compares a subset model's residual sum of squares with an error-variance estimate from the full candidate model.	Under the usual derivation, an approximately unbiased candidate has expected C_p near p; the full model itself has C_p near/equal to p by construction and should not be selected for that reason alone.	If several subsets have small C_p near p, inspect predictive validation, diagnostics, simplicity and subject-matter plausibility rather than choosing mechanically.	Depends on the full candidate model and its variance estimate; definitions/counting conventions for p vary across software; it is a screening criterion, not an automatic proof of the best model.
Cross-validated Log-Loss / Deviance	Lower is better for the same outcome definition, weighting and evaluation scheme. Compare against a simple probabilistic baseline and resampling uncertainty. There is no universal percentage difference that is automatically meaningful.	Compare probabilistic predictive models out of sample using likelihood-based scoring.	Log-loss is average negative log predictive probability. Model deviance is closely related to negative log-likelihood but may differ by constants or scale depending on the model/software, so name the reported quantity precisely.	Evaluate the score on held-out folds generated by a leakage-safe split/pipeline; confident wrong probabilities receive a large penalty.	For binary classification, compare CV log-loss with a prevalence-only or other domain-relevant probabilistic baseline.	Can be dominated by a few extremely confident errors; values are not directly comparable when observation weights, outcome definitions or likelihood conventions differ.
Regularisation Strength (λ)	too small balanced too large Very small λ → unregularised, risk of overfitting. λ chosen inside a properly nested/tuned CV workflow → a data-driven candidate; final performance still requires honest out-of-sample evaluation. Very large λ → coefficients shrunk too much (underfitting).	Control complexity of models like ridge, lasso, elastic‑net.	Penalise large coefficients (L2) or enforce sparsity (L1) to improve generalisation.	Typically chosen via cross‑validated performance curves over λ.	L1 (lasso) can produce sparse models; L2 (ridge) stabilises coefficients with multicollinearity.	Interpretation depends on scaling; different λ scales across algorithms and implementations. Common pitfalls: Using default λ without checking for under/over‑regularisation via validation curves. Comparing λ values across models with different feature standardisation or penalty definitions.

Advanced & Research Diagnostics

These methods extend the classical metrics above. Some are established specialist tools, while others remain research-heavy. Always check the exact definition used by the paper or software package: similarly named calibration, drift and reclassification measures are not automatically interchangeable.

Calibration diagnostics specialist / practice
Adaptive Calibration Error (ACE), classwise/static calibration variants, calibration curves, Brier Score and Brier Skill Score. Binning-based errors depend on binning choices and sample size.
Test-Based Calibration Error (TCE) research
A research calibration framework based on statistical tests of calibration. Do not confuse the abbreviation with an informal “thresholded calibration error”.
Bayesian predictive model comparison specialist / practice
WAIC, PSIS-LOO, expected log predictive density (ELPD) and Pareto-k diagnostics.
Distribution-shift diagnostics specialist / practice
KL divergence, Maximum Mean Discrepancy (MMD), Wasserstein distance and Population Stability Index (PSI). PSI is a monitoring heuristic, not a universal significance test.
Risk / survival discrimination specialist / practice
Concordance index, Somers’ D, time-dependent AUC and time-dependent Brier scores. IDI and NRI are specialised model-comparison measures and need careful interpretation.
Predictive uncertainty research → practice
Negative log-likelihood, proper scoring rules, prediction intervals / sets and, where supported by the model, aleatoric versus epistemic uncertainty decomposition.
Robustness and influence research → practice
Influence functions, resampling stability, sensitivity analysis and perturbation tests. Project-specific stability indices should be labelled explicitly as custom heuristics.
Fairness diagnostics research → practice
Equalized odds, demographic parity, predictive parity and calibration within groups. These criteria can conflict; metric choice is a normative as well as technical decision.
Interaction-aware interpretability specialist
SHAP interaction values and related diagnostics describe model behaviour, not causal effects.
Leakage checks practice
Use time/group-aware validation, pipeline isolation, feature provenance checks and shuffled-target sanity checks. Do not present a project-specific leakage score as an established standard metric without an external definition.

Model Interpretability & Explainability

Method / Metric	Decision Criterion	Purpose	Description	Working Mechanism	Example	Limitations
Permutation Feature Importance	0 medium large Large performance drop when permuted = strong importance. Near‑zero drop = little contribution (under current model).	Global importance ranking for features in any black‑box model.	Measures how much a model’s performance metric worsens when the values of a feature are randomly permuted.	Break the relationship between a feature and target by shuffling that feature; recompute performance and compare to baseline.	On held-out data, repeat each permutation several times and report the distribution of score drops; correlated features may share or mask importance.	Correlated features can share importance (each appears less important alone). Requires many evaluations of the model; may be expensive. Common pitfalls: Interpreting low importance as “irrelevant” when it may be redundant with correlated variables. Permuting time‑series or grouped features independently, breaking their structure.
SHAP Values	unstable ok stable Check attribution fidelity, stability and sensitivity to the background/masking assumptions. Large \|SHAP\| means large model-attribution magnitude for that prediction; agreement with prior beliefs is not itself validation.	Provide theoretically grounded, additive feature attributions for individual predictions and global patterns.	Based on Shapley values from cooperative game theory; each feature gets a contribution to pushing the prediction away from a baseline.	Approximate each feature’s marginal contribution by averaging over many coalitions of features; specialised fast algorithms exist for tree‑based models.	Global SHAP summary plots highlight which variables drive model predictions overall; local plots explain single predictions (e.g. why a loan was rejected).	Computationally expensive for complex models without specialised approximations; explanations can be misread as causal rather than correlational. Common pitfalls: Over‑interpreting SHAP values as causal effects instead of associations. Ignoring the impact of feature collinearity on SHAP attribution.
LIME (Local Interpretable Model‑agnostic Explanations)	unstable ok faithful Good explanations are locally faithful (approximate the model well near the point of interest) and sparse enough to be interpretable.	Explain individual predictions by fitting a simple surrogate model around a local neighbourhood.	Samples points near the instance, queries the black‑box model, and fits an interpretable model (e.g. linear or small tree) to those local outputs.	Weights samples by proximity to the instance; the surrogate’s coefficients are reported as local feature importance.	Useful for debugging why a specific prediction was made, especially in regulated domains where explanations must be human‑readable.	Local surrogate may be unstable (different runs give different explanations); only valid near the point; can be misleading globally. Common pitfalls: Treating LIME’s local linear model as a global explanation. Ignoring randomness in neighbourhood sampling, leading to inconsistent explanations.

Python Code Snippets (scikit‑learn / SciPy / SHAP / LIME)

Classification Metrics & Confusion Matrix



import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    confusion_matrix, accuracy_score, balanced_accuracy_score,
    precision_score, recall_score, f1_score, roc_auc_score,
    average_precision_score, auc, precision_recall_curve,
    matthews_corrcoef, cohen_kappa_score
)

X, y = make_classification(
    n_samples=800, n_features=10, n_informative=6, n_redundant=2,
    weights=[0.75, 0.25], class_sep=1.2, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, stratify=y, random_state=42
)

model = LogisticRegression(max_iter=1000, random_state=42)
model.fit(X_train, y_train)
y_proba = model.predict_proba(X_test)[:, 1]
y_pred = (y_proba >= 0.5).astype(int)

cm = confusion_matrix(y_test, y_pred)
acc = accuracy_score(y_test, y_pred)
bal_acc = balanced_accuracy_score(y_test, y_pred)
prec = precision_score(y_test, y_pred, zero_division=np.nan)
rec = recall_score(y_test, y_pred, zero_division=np.nan)
f1 = f1_score(y_test, y_pred, zero_division=np.nan)
roc_auc = roc_auc_score(y_test, y_proba)
average_precision = average_precision_score(y_test, y_proba)  # AP
pr_precision, pr_recall, _ = precision_recall_curve(y_test, y_proba)
trapezoidal_pr_auc = auc(pr_recall, pr_precision)
mcc = matthews_corrcoef(y_test, y_pred)
kappa = cohen_kappa_score(y_test, y_pred)

print("Confusion Matrix:\n", cm)
print("Accuracy:", acc)
print("Balanced accuracy:", bal_acc)
print("Precision:", prec)
print("Recall:", rec)
print("F1-score:", f1)
print("ROC AUC:", roc_auc)
print("Average Precision (AP):", average_precision)
print("Trapezoidal PR AUC:", trapezoidal_pr_auc)
print("Matthews Corr. Coeff.:", mcc)
print("Cohen's Kappa:", kappa)

Statistical Tests (t-test, ANOVA, chi-square)



import numpy as np
from scipy import stats

rng = np.random.default_rng(42)

# Two independent groups: Welch t-test is a robust default when equal variances
# are not established.
group1 = rng.normal(loc=0.0, scale=1.0, size=40)
group2 = rng.normal(loc=0.5, scale=1.4, size=45)
t_result = stats.ttest_ind(group1, group2, equal_var=False)
print("Welch t-test:", t_result)

# One-way ANOVA. SciPy >= 1.16 supports Welch ANOVA via equal_var=False.
group_a = rng.normal(loc=0.0, scale=1.0, size=30)
group_b = rng.normal(loc=0.5, scale=1.5, size=35)
group_c = rng.normal(loc=1.0, scale=0.8, size=28)
anova_classical = stats.f_oneway(group_a, group_b, group_c, equal_var=True)
anova_welch = stats.f_oneway(group_a, group_b, group_c, equal_var=False)
print("Classical one-way ANOVA:", anova_classical)
print("Welch ANOVA:", anova_welch)

# Chi-square test of independence. Inspect expected counts before trusting the
# large-sample approximation.
contingency_table = np.array([[30, 20], [15, 35]])
chi2, p_chi, dof, expected = stats.chi2_contingency(contingency_table)
print("Chi-square:", chi2, p_chi, "dof =", dof)
print("Expected counts:\n", expected)

# Fisher's exact test is available for a 2x2 table and is useful for sparse counts.
odds_ratio, p_fisher = stats.fisher_exact(np.array([[1, 9], [8, 2]]))
print("Fisher exact:", odds_ratio, p_fisher)

# Normality: Shapiro-Wilk is a test, not proof of normality; combine it with
# a Q-Q plot and practical/model-specific robustness considerations.
sample = rng.normal(size=100)
w_stat, p_shapiro = stats.shapiro(sample)
print("Shapiro-Wilk:", w_stat, p_shapiro)

# 1-sample KS: compare to a FULLY SPECIFIED continuous reference distribution.
# Do not estimate mu0/sigma0 from the same sample and then use the ordinary
# known-parameter KS p-value as if they had been fixed in advance.
mu0, sigma0 = 0.0, 1.0
reference_cdf = stats.norm(loc=mu0, scale=sigma0).cdf
ks_stat, p_ks = stats.kstest(sample, reference_cdf)
print("One-sample KS:", ks_stat, p_ks)

# Two-sample KS compares two independent continuous distributions.
sample2 = rng.normal(loc=0.3, size=120)
ks2_stat, p_ks2 = stats.ks_2samp(sample, sample2)
print("Two-sample KS:", ks2_stat, p_ks2)

# Brown-Forsythe form of Levene's test uses group medians.
lev_stat, p_lev = stats.levene(group_a, group_b, group_c, center="median")
print("Brown-Forsythe / median Levene:", lev_stat, p_lev)

Regression Metrics (MAE vs RMSE)


from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score

# Separate regression example — do not reuse binary classification labels.
X_reg, y_reg = make_regression(
    n_samples=500,
    n_features=6,
    n_informative=4,
    noise=20.0,
    random_state=42,
)

X_train_reg, X_test_reg, y_train_reg, y_test_reg = train_test_split(
    X_reg, y_reg, test_size=0.25, random_state=42
)
reg_model = LinearRegression()
reg_model.fit(X_train_reg, y_train_reg)
y_pred_reg = reg_model.predict(X_test_reg)

mae = mean_absolute_error(y_test_reg, y_pred_reg)
rmse = root_mean_squared_error(y_test_reg, y_pred_reg)
r2 = r2_score(y_test_reg, y_pred_reg)

print(f"Mean Absolute Error (MAE): {mae:.4f}")
print(f"Root Mean Squared Error (RMSE): {rmse:.4f}")
print(f"R²: {r2:.4f}")

SHAP / LIME Basics



import numpy as np
import pandas as pd
import shap
from lime.lime_tabular import LimeTabularExplainer
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=600, n_features=8, n_informative=5, n_redundant=1, random_state=42
)
feature_names = [f"feature_{i}" for i in range(X.shape[1])]
X_df = pd.DataFrame(X, columns=feature_names)

X_train_df, X_test_df, y_train, y_test = train_test_split(
    X_df, y, test_size=0.25, random_state=42, stratify=y
)
model = LogisticRegression(max_iter=1000, random_state=42)
model.fit(X_train_df, y_train)

# SHAP: use training/background data; explain held-out rows.
shap_explainer = shap.Explainer(model, X_train_df, algorithm="linear")
shap_values = shap_explainer(X_test_df)
shap.plots.beeswarm(shap_values, show=False)

# LIME: training data define its perturbation distribution.
lime_explainer = LimeTabularExplainer(
    training_data=X_train_df.to_numpy(),
    feature_names=feature_names,
    class_names=["negative", "positive"],
    mode="classification",
    random_state=42,
)

# LIME supplies NumPy perturbations; restore feature names for sklearn.
def predict_proba_array(a):
    return model.predict_proba(pd.DataFrame(a, columns=feature_names))

i = 0
exp = lime_explainer.explain_instance(
    X_test_df.iloc[i].to_numpy(), predict_proba_array, num_features=5
)
print("LIME local fidelity R²:", exp.score)
print(exp.as_list())

Selected Authoritative References & Documentation

These are starting points for definitions, APIs and evaluation workflow; individual methods may require more specialised methodological literature.

scikit-learn: Model selection and evaluation — cross-validation, threshold tuning, metrics, baselines and model selection.
scikit-learn: Probability calibration — reliability diagrams, calibration and proper scoring-rule context.
SciPy: Statistical functions — current statistical-test and distribution APIs used in the examples.
Lundberg & Lee: SHAP / unified feature attribution.
Ribeiro, Singh & Guestrin: LIME.
Binder: Configure the user interface — documented JupyterLab launch-link form used by the companion notebooks.

Limits of Models, Metrics & Science

This dashboard summarises many of the tools we currently use to make sense of data – from classical statistics to modern ML metrics. They are powerful, but they are not the whole story.

Specialised scope: forecasting, recommender/information-retrieval ranking, clustering, multilabel learning, causal inference and reinforcement learning each have their own evaluation families and validation designs. Examples include MASE/pinball loss/CRPS for forecasting, NDCG/MAP@K/MRR for ranking, and silhouette or Davies–Bouldin diagnostics for clustering. They are intentionally not compressed into generic “good/bad” bands here; use a domain-specific reference when those are the primary task.

Gödel’s incompleteness theorems show, roughly, that any consistent, effectively axiomatized formal system expressive enough to represent elementary arithmetic is incomplete: there are statements in its language that it can neither prove nor refute from its own axioms. This is a theorem about formal systems; the modelling point made here is only an analogy, not a statistical consequence.

By analogy, no modelling framework or collection of metrics can ever fully capture the processes we study. We always work with:

finite, noisy, and biased data
simplified models of complex systems
metrics that highlight some aspects while ignoring others
assumptions that are never perfectly satisfied

The goal here is not to pretend that AUC, RMSE, p-values, or any “cutting-edge” metric delivers final truth. Instead, this dashboard makes our tools explicit – showing where they are informative, where they are fragile, and where they leave important questions unanswered.

In other words: this is a map, not the territory. The map is useful, and it keeps improving – but if we ever forget that it is incomplete, we stop doing science.

Thanks for your attention.
— manu

Email: x34mev@proton.me • GitHub: https://github.com/slashennui