Working worksheet
Fine-tuning intake
A fine-tune is justified by a measured behavior gap and a credible experiment—not by the availability of a training API.
Decision rule: if a deterministic guarantee belongs in ordinary code, put it in code. If the problem is missing or changing knowledge, test retrieval. If the problem is legal output structure, use constrained decoding and validation. Fine-tune when changing learned behavior is both necessary and measurable.
Interview questions
- What exact behavior is failing?
Name the input conditions, current output, desired output, and why the difference matters to a user. - What evidence establishes the baseline?
Provide representative examples, current metrics, behavioral slices, cost, and latency. - Why are simpler interventions insufficient?
Evaluate ordinary code, prompt changes, better context, retrieval, schemas, tools, and workflow redesign. - What examples exist?
Identify their source, volume, coverage, freshness, ownership, consent, privacy constraints, and labeling process. - What is the training objective?
Specify the input, target tokens, masked regions, system messages, tool schemas, and expected inference format. - How will leakage be prevented?
Split by user, organization, time, conversation, or near-duplicate cluster when random row splitting would leak identity or content. - Which metrics and slices define success?
Separate product outcomes from token loss. Name critical regressions that block release. - What training plan is proportionate?
Choose base model, full or parameter-efficient tuning, learning rate, batch strategy, epochs, checkpoints, and early stopping. - How will the change reach production?
Define offline gates, canary scope, monitoring, drift signals, incident ownership, and rollback mechanics. - What result would make us stop?
State cost, latency, quality, safety, or data findings that invalidate the experiment.
One-page experiment plan
| Behavior hypothesis | The fine-tune will improve ______ for ______ without regressing ______. |
|---|---|
| Baseline | Current model, prompt, workflow, metrics, slices, cost, and latency. |
| Data | Source, rights, coverage, filtering, label quality, and split unit. |
| Training | Objective, masks, parameter strategy, hyperparameters, checkpoints, and stopping rule. |
| Evaluation | Held-out tasks, graders, human review, statistical comparison, and critical slices. |
| Deployment | Gate, canary, monitoring, rollback threshold, and owner. |