LLMs from First Principles · Class 1

Shipping nondeterministic software

A three-hour, lifecycle-driven class mapping predictive ML deployment to LLM applications and agents—from versioned evidence and offline gates through staged release, online evaluation, drift, and incident-driven iteration.

View slides + presenter notes → Download the PowerPoint ↓ Open the BugBoard lab → Download the lab data ↓

The engineering outcome

Specify

Turn product intent into representative tasks, critical slices, observable invariants, and a versioned failure taxonomy.

Measure

Choose graders deliberately, preserve paired comparisons, report uncertainty, and inspect regressions before averages.

Release + learn

Stage exposure, join online outcomes, confirm drift, contain failures, and promote incidents into future release gates.

Three-hour run of show

0:00When evidence is captured, how blast radius grows, and the eight-stage deployment loop.

0:30Predictive ML, LLM application, and agent corollaries at every lifecycle stage.

0:55Outcomes, risks, representative evidence, slices, failure modes, and holdouts.

1:20Version the full system, construct offline evaluation, and write the release gate first.

2:00BugBoard decision, error analysis, staged exposure, online outcomes, and delayed labels.

2:38Drift detection, controlled response, incident-to-eval learning, and homework.

The case that carries the class

BugBoard is a fictional issue-triage agent. Version B improves overall success from 82% to 86%, yet security containment falls from 90% to 65% while latency and cost rise. Students must discover why the headline is insufficient and turn product judgment into a no-go gate.

Teaching-data boundary: every BugBoard ticket, metric, and result is fictional and illustrative. None of it represents WHOOP data or production results.

Built for retrieval, not passive listening

Predict

Students commit to a release decision before the security slice is revealed.

Investigate

Pairs interrogate stored A/B results without needing credentials or a model API.

Defend

Each pair writes thresholds and explains what evidence could change its decision.