This is the course where we get into the architectures — not just the tools.
It's four sessions, designed for senior PMs and product leaders who've been working with data for several years and want to sharpen the systems through which they think about metrics, not just the individual analyses.
Session 1: North Star Frameworks
Not "which metric should we use?" but "how do we build a metric hierarchy that captures both near-term health and long-term value creation?"
The classic North Star framing is useful but incomplete. We work through three companies — a ride-hailing platform, a subscription media product, and a social commerce app — and build their North Star metric from scratch. The goal isn't to arrive at the right answer. It's to surface all the implicit trade-offs in the metric choice: what you're rewarding, what you're ignoring, and what you'll discover six months from now that you should have thought about today.
Session 2: Diagnostic Analysis
When a metric moves unexpectedly — up or down — most product teams run the same analysis: segment cuts by geography, platform, cohort, feature. They find the segment where the movement is concentrated and declare the diagnosis.
This is frequently wrong.
We cover a structured approach to diagnostic analysis that explicitly tests for: data artifacts (is the metric actually moving, or is something broken in the tracking?), supply-side changes (did inventory, pricing, or algorithm change?), demand-side changes (did user behavior change, or did the user population change?), and interference effects (is this metric responding to something that happened elsewhere in the product?).
The dating app matchmaking case illustrates all four simultaneously.
Session 3: Experimentation Design at Scale
Running A/B tests is easy. Running A/B tests that produce actionable, trustworthy results at scale is genuinely difficult.
We work through the specific failure modes of experimentation at scale: network effects that violate the independence assumption, geographic clustering that produces false significance, hold-out groups that drift over time. The ride-hailing dynamic pricing case is the main vehicle — it's a product where almost every classic experimental assumption breaks.
Session 4: Turning Analysis into Organizational Action
This is the session that surprises students most. Everything before it is about analysis. This session is about what happens after the analysis — when you've found something real, and you need to make the organization act on it.
Analysis that doesn't change decisions is noise. The gap between "we know this" and "we changed behavior because of this" is often larger in organizations than the gap between "we don't know" and "we know." We work through case studies of PM teams who had the right analysis and couldn't act on it — and what they did or didn't do about it.