Turn examples into a learning signal

An example is an input paired with a desired response

A JSONL file stores one JSON object per line. Our prepared examples have messages with three roles: system instructions specify the SQL contract, the user supplies schema and question, and the assistant supplies the demonstrated SQL response.

system: Return one SQL query using the supplied column identifiers.
user:   col0 = Driver, col2 = Laps
        How many laps did Ricardo Zonta have?
assistant: SELECT SUM(col2) FROM data WHERE col0 = 'ricardo zonta';

This shortened display uses the development example from lesson 1 to explain the format. It is not one of the examples used for updates. The downloaded records retain the full instructions, schema, source-record index, and reference answer.

We reuse the pretrained tokenizer, which turns text into token IDs, and its chat template, which adds the model's role and message-boundary tokens. Tokens need not be whole words. Training and generation must use the same conventions.

Inspect a real training record

This example really supplied updates: source-record index 41941 in the official training split, table 2-1140101-6. The question asks which constructor is associated with the Goodwood circuit; col1 is Circuit and col4 is Constructor.

{
  "messages": [
    {
      "role": "system",
      "content": "Translate the question into one SQLite SELECT query. Use only the table data and the column identifiers col0, col1, etc. shown below. Select one column, optionally using MAX, MIN, COUNT, SUM, or AVG. Conditions may use =, >, or < joined by AND. Do not use joins, sorting, grouping, or LIMIT. Text values in the database are lowercase; use lowercase string literals. Return only SQL, with no explanation or Markdown."
    },
    {
      "role": "user",
      "content": "Table: data\nColumns:\ncol0 (text): \"Race Name\"\ncol1 (text): \"Circuit\"\ncol2 (text): \"Date\"\ncol3 (text): \"Winning driver\"\ncol4 (text): \"Constructor\"\ncol5 (text): \"Report\"\n\nQuestion: Which Constructor has the goodwood circuit?"
    },
    {
      "role": "assistant",
      "content": "SELECT col4 FROM data WHERE col1 = 'goodwood';"
    }
  ]
}

After the actual chat template, this record contains 175 prompt tokens followed by 15 response tokens, 190 total. The response starts after the assistant-role prefix and includes its ending token and trailing newline. The loss boundary follows the tokenized template, not a character count or the visible word “assistant”.

Predict the response one token at a time

During training, the input contains the prompt followed by the demonstrated SQL. At each response position the causal model sees the prompt and earlier correct SQL tokens, then predicts the next token. Later answer tokens remain hidden from that position. At generation time there is no reference SQL in the input: each next prediction follows the model's own earlier output.

The loss is a prediction penalty. Cross-entropy is smaller when the model assigns more probability to the demonstrated next token. We average those penalties over real response targets, including the ending token. This is next-token learning applied to reviewed responses.

Prepare data without weakening the evaluation

Keep the official train/development/test assignments. Check for duplicate or near-duplicate questions and table contents, not only identical IDs. Our experiment verified disjoint table IDs; it did not complete that deeper duplicate audit or prove the public data was absent from pretraining.

Our sequence limit was 512 tokens. Preparation checked lengths rather than silently cutting off answers. One oversized training candidate was skipped; no development or test examples were dropped. A cutoff that depends on the expected answer can bias evaluation, so a stronger future evaluator keeps all chosen test cases and reports unsupported lengths explicitly.

In this experiment, preprocessing read test labels to construct reference SQL and validate execution, and a raw test record was inspected during format checks. Test answers did not enter weight updates or generation prompts. This was held out from training and tuning, not an access-controlled blind test.

Inspect the code that establishes these boundaries

prepare.py joins questions to schemas, preserves split membership, applies the chat template, and writes provenance and lengths. loss.py excludes prompt and padding positions from loss. We corrected an exclusive-end masking error before the reported run; the tests verify that the first padding position contributes nothing.

Before a job, inspect several training records for complete responses, sensible labels, and the exact generation format. An ambiguous or incorrect demonstration becomes a training target too. Keep final-test labels in a separate evaluator when designing a new experiment.