# Local WikiSQL fine-tuning results — September 29, 2026

The small-model approach works as a laptop teaching exercise. A single pass over
2,000 WikiSQL training examples took **4 minutes** on this M3 Max and raised
held-out answer accuracy to **54%**. The result is useful for demonstrating SFT
and inspecting failures; it is not accurate enough to execute business queries
without review.

## Final comparison, protocol frozen before model evaluation

All three conditions received the same 200 questions from 198 test tables.
None of these table IDs appeared in the training or validation splits. Greedy
decoding and the maximum of 128 output tokens were held constant. The adapter
checkpoint was the preselected final step 2,000, not a checkpoint chosen using
test scores.

| Condition | Accepted SQL within the task grammar | Correct executed answer | Normalized exact SQL |
|---|---:|---:|---:|
| Original model, instructions only | 61/200 (30.5%) | 3/200 (1.5%) | 1/200 (0.5%) |
| Original model, three reviewed training examples in the prompt | 178/200 (89%) | 11/200 (5.5%) | 2/200 (1%) |
| Fine-tuned adapter, instructions only | 198/200 (99%) | **108/200 (54%)** | 96/200 (48%) |

Removing an enclosing Markdown fence did not change these answer scores. The
gains are not explained by Markdown formatting alone. Excluding empty/all-null
gold answers leaves 187 questions: the adapter gets 101 right (**54.0%**), versus
2 for the original model and 6 for the three-example prompt.

The gap between 99% accepted SQL and 54% correct answers is the most important
teaching result. Syntax and schema validation catch only some failures. A query
can reference real columns and execute successfully while answering the wrong
question.

## Time and memory

Hardware: Apple M3 Max, 14 CPU cores, 30 GPU cores, 36 GiB unified RAM.

| Measurement | Observed value |
|---|---:|
| Main training process, including periodic validation | 239.35 seconds |
| Training wall time including guard/process startup | 240.24 seconds |
| Peak MLX allocations | 1.87 GB / 1.74 GiB |
| Peak sampled whole-process footprint | 2.20 GB / 2.05 GiB |
| Minimum free system memory during training, including speculative pages | 1.38 GiB |
| New swap-outs / increase in occupied swap | **0 / 0 bytes** |
| Adapter size | 11,754,630 bytes, about 12 MB |
| Original-model final-test evaluation | 41.57 seconds |
| Three-example-prompt final-test evaluation | 49.75 seconds |
| Adapter final-test evaluation | 49.60 seconds |

The download of the model and dataset took approximately 36 seconds on this
connection. Installation, data preparation, initial calibration, and development
work are excluded from the four-minute training measurement. Compilation caches
were already warm from calibration. This is a measurement on this Mac, not a
runtime guarantee for every MacBook. No inference API or paid cloud compute was
used. The model processes exited after their runs; weights were not left resident.

The final adapter is additionally preserved outside the cache at
`~/models/dougdoes-wikisql-sft-2026-09-29/adapter/`, with a base-model revision and
SHA256 provenance file. Large weights and databases are not committed to Git.

## What changed and what still fails

An illustrative validation question asks: **“How many laps did Ricardo Zonta
have?”** The columns include `col0: Driver` and `col2: Laps`.

Original model:

```sql
SELECT COUNT(*) FROM data WHERE col0 = 'Ricardo Zonta'
```

Adapter:

```sql
SELECT col2 FROM data WHERE col0 = 'ricardo zonta';
```

The latter returns **53**. It retrieves the requested numeric field instead of
counting matching rows, and follows the dataset's lowercase convention.

Failures remain. For “What is the lowest game number on 20 July 2008?”, the adapter
uses `col1 = 'july 2008'`, dropping the day and failing to match the stored date.
Another answer adds an unnecessary condition after otherwise selecting the right
columns. Include both wins and failures in the course.

These are illustrative validation examples chosen after inspecting outputs; the
200-question final-test scores above measure aggregate performance. The large
gain includes learning the dataset's column IDs and literal conventions. It does
not establish a comparable gain in general SQL reasoning.

## Experiment decisions and qualifications

- Fixed pretrained checkpoint: Qwen2.5-Coder-0.5B-Instruct, 494 million parameters,
  in its original 16-bit representation. Ordinary LoRA; no quantization.
- Rank-eight adapters in the final 16 transformer blocks; 2,932,736 trainable
  parameters. Batch one, Adam at 0.0001, gradient checkpointing, one pass over
  2,000 training examples. Only response positions contribute to loss.
- Prepared examples are at most 512 tokens. One oversized training candidate was
  skipped before selecting the fixed 2,000 examples; no targets were truncated.
- Validation uses 100 examples from 97 tables. Correct answer counts were 4 for
  the original model, 6 with the reviewed examples, and 50 for the final adapter.
  Corrected validation loss fell from 0.7906 to 0.1426, measured before the last
  update at step 1,999.
- The installed MLX-LM 0.31.3 trainer's default loss included the first padding
  position. The first main attempt was stopped; the reported adapter was trained
  fresh with the exclusive-end mask in `loss.py`. The installed package was not
  edited. Earlier calibration/aborted-run losses are not mixed with this curve.
- A preliminary three-example prompt chosen mechanically scored 1/100 on
  validation. Inspecting those examples exposed ambiguous wording. The final
  prompting baseline uses three clearer training examples covering lookup, SUM,
  and multiple conditions. Both development results are retained. This is a
  small prompting comparison, not an exhaustive search for the best prompt.
- These are local execution scores under the documented restricted SQL contract,
  not official WikiSQL leaderboard results. There is one seed, a small test
  sample, potentially ambiguous labels, and no claim that WikiSQL was absent
  from pretraining. Execution agreement can also occur coincidentally.

Test labels were read during preparation to build reference SQL, validate execution,
and check sequence lengths; a raw test record was also inspected during format
checks. No test answers supplied updates or inference context, and no test
examples were dropped. This was held out from training and tuning, not a blinded
evaluation. The published test sample is now historical evidence; further recipe
changes should use development data, with a fresh check for new transfer claims.
An independent challenge evaluation has not yet been run.

## Recommendation for the course

Use this configuration for the first practical SFT lab. The training time and
memory fit the goal, and the remaining mistakes make evaluation necessary and
concrete. Keep a saved adapter and saved predictions available so learners can
follow the lesson before installing dependencies.

Build the five lessons in the [course walkthrough](https://dougdoes.ai/courses/llms-from-first-principles/tracks/fine-tuning/):
baseline, examples and token boundaries, adapters, local training, and reloading
with evaluation. Explain the executed training steps at a practical level and
link the deeper linear algebra rather than making it a prerequisite.

Before publishing a downloadable learner lab, repeat the run on a representative
16 GiB Mac and simplify the entry commands. Improvements in answer reliability
should be separate measured experiments on data quantity/quality, model size,
or value context; do not quietly change several factors and attribute the result
to one fine-tuning technique.

## Inspect the evidence

- [Frozen protocol and source hashes](results/2026-09-29/protocol.json)
- [Aggregate results and environment versions](results/2026-09-29/results.json)
- [Training measurements](results/2026-09-29/training-metrics.jsonl)
- [Training memory trace](results/2026-09-29/training-memory.json)
- [Original-model test outputs](results/2026-09-29/base-test.predictions.jsonl)
- [Three-example-prompt test outputs](results/2026-09-29/fewshot-test.predictions.jsonl)
- [Adapter test outputs](results/2026-09-29/trained-test.predictions.jsonl)

Four focused tests cover response masking, SQL restrictions, read-only execution,
quoting, numeric normalization, and answer multiplicity. All 600 final-test scores
were recomputed from the saved raw outputs, and source hashes matched the protocol
frozen before final testing.