The payments company will switch a fraud model on at the point of sale next month. You are the one who says which model, at what threshold, at what review volume, and what it will and won't catch — in a memo the operations lead can act on, backed by a notebook you wrote. Complex from the start: the first checkpoint already needs the data loaded, the baseline computed, and the metric argued.
A starter notebook (project-starter.ipynb)
with load_fraud(), load_and_recode() + IDS_mapping merge (for the leak-audit
callback), standardise, recall_at_flagged_rate, bootstrap_metric already
written, a time_split_3way(df, train_frac, tune_frac) stub (signature only — s4-2
taught it, the body is yours), a business_cost stub too (s4-1 taught it), and one empty labelled cell
per checkpoint.
Each checkpoint is one observable artifact you build yourself. Checking all six flips this module's mastery to "project complete" in the tracker. No auto-grading — you mark it once the artifact exists.
Target, metric, and the success sentence written before any model cell runs; the do-nothing baseline and total fraud dollars at stake computed.
Metric is recall at a flagged rate (or dollars recovered) with the review volume stated as the desk's constraint, not the model's; accuracy explicitly rejected with the 99.83% number.
A time-ordered three-way split with the cut points stated; a one-line leak verdict for each column class
(Time, Amount, V1..V28) and for one constructed aggregate feature you build
and then remove.
Tune slice never used for scoring; the constructed leak's inflated (or non-inflated — report what you
measure) score shown next to the honest one; the s2-1-style "when is this written?" argument applied to
Time.
≥3 models from the Module 03/04 toolbox on identical time-ordered test rows: PR-AUC, recall at flagged rate, fit and single-row predict wall-clock, each with a bootstrap interval; the tie (or the real winner) declared either way.
The latency constraint measured, not guessed; the interval on the difference between the top two reported; no accuracy column.
The four-cell cost matrix in dollars with the review cost justified in writing; the tuned threshold from
TunedThresholdClassifierCV on the tune slice; test-row cost at 0.5 vs. tuned; flag count at the tuned
threshold; the threshold and saving re-run under at least two other review-cost assumptions.
Folds respect the clock; the saving reported as a range with its assumption; the flag count compared to what the desk can staff.
A limits section: what two days cannot show (with s5-1's failure mode named), the leak classes checked, one monitoring rule for the live stream, and the one finding that would change the recommendation.
Does not claim the model will keep working; names the seen/unseen-attack lesson as the analogy; the monitor is concrete (what is compared, how often).
≤1 page, plain language, no code: which model, what threshold, what review volume, what it catches (with intervals), what it costs, what it won't do — answer first. Plus a ≤10-line sketch on the churn brief (never loaded): target, metric, split, first model, first pitfall, honest success.
Leads with the decision; every number traceable to a checkpoint; a tie (if it is one) or a real winner (if it is one) stated as a finding with precedent; the churn sketch derives its metric from "500 calls a month."
Class weighting or resampling for the minority class (class_weight="balanced", or an oversampling
scheme — neither was taught; you're on your own here, which is the point) and whether the tuned threshold or
the intervals move.