One cumulative bug-triage agent

Homework 1

Build a miniature eval harness

Students receive a task set and stored outputs from two agent versions. They must define success, calculate results, expose a slice-level regression, and recommend whether to release.

def schema_score(output, schema): ...
def task_success(example, output): ...
def summarize_results(results): ...
def paired_difference(before, after): ...
def confidence_interval(outcomes): ...

Homework 2

Train a tiny language model

The required path is intentionally small enough to run in NumPy. Every function corresponds to a term used in production fine-tuning conversations.

• Numerically stable softmax
• Cross-entropy loss
• Gradient of a linear classifier
• Batch averaging and one SGD update
• Shifted next-token labels
• Padding-mask-aware sequence loss
• Finite-difference gradient check

Optional extension: train a character-level model, generate samples before and after training, plot training and validation loss, and deliberately overfit a tiny dataset.

Homework 3

Design and defend a fine-tuning experiment

Students receive traces, a proposed dataset, and an underspecified request to “make the model better.” They must determine whether tuning is the right intervention and design the test.

• Diagnose the error distribution and name the target behavior.
• Choose between ordinary code, prompting, retrieval, constraints, or fine-tuning.
• Clean and split data without identity, time, or near-duplicate leakage.
• Construct masked training labels.
• Interpret training and validation curves.
• Define critical behavioral slices and unacceptable regressions.
• Write deployment, canary, monitoring, and rollback criteria.