Decide whether fine-tuning helped

Reload and compare under fixed conditions

Reloading in a fresh process checks that the saved files are sufficient for inference. On development data, compare the original model, the original model with three training examples in the prompt, and the model with the adapter. Use the same questions, schema information, and generation settings.

python guard.py --budget-gib 2 --log "$WIKISQL_HOME/runs/adapter-valid.log" -- \
  python -u run_experiment.py eval --model "$WIKISQL_HOME/model" \
  --data "$WIKISQL_HOME/data" --source "$WIKISQL_HOME/wikisql/data" \
  --adapter "$WIKISQL_HOME/runs/trained" \
  --out "$WIKISQL_HOME/runs/adapter-valid" --split valid

For the prompting comparison, omit --adapter, add --few-shot 3, and choose a fresh output path. The three demonstrations come from training data. In our experiment, reviewing their wording improved development performance; this was a modest baseline effort, not an exhaustive search for the best prompt.

Once the recipe is fixed, --split test reproduces the published comparison. Our generations used greedy decoding and at most 128 output tokens. Because the course already exposes this test, reproducing its score does not create a new blind evaluation.

Inspect a failure before celebrating a percentage

For “What is the lowest game number on 20 July 2008?”, the reference uses the complete date:

SELECT MIN(col0) FROM data WHERE col1 = '20 july 2008';

The adapter instead produced:

SELECT MIN(col0) FROM data WHERE col1 = 'july 2008';

It dropped the day, found no matching row, and returned null instead of 1. The query has valid columns and syntax. It is still wrong. This is a selected development example; the aggregate test reports both successes and failures.

Avoid teaching to the benchmark

Training on WikiSQL's official training split is legitimate. It teaches both useful mappings and benchmark conventions such as lowercase literals and column IDs. The gain does not establish a comparable improvement on a company's database, and we cannot rule out WikiSQL appearing in the base model's pretraining.

Keep further tuning on development data. Do not repeatedly adjust prompts, epochs, or filtering in response to the published test score. Treat that test as historical evidence once it guides your decisions.

For a new transfer claim, reserve a small, independently reviewed set of questions against fresh tables before choosing the next recipe. Include paraphrases, reordered columns, dates, several filters, and count-versus-value questions. Keep related variants together in one split. Keep answers behind the final evaluator and report failures rather than dropping them. That independent challenge evaluation has not been run here.

A few dozen challenge questions can expose weaknesses; they cannot certify production reliability. Stronger claims need more representative data, uncertainty estimates, duplicate checks, and repeated training runs. Unsupported joins should be treated separately from the single-table task rather than silently changing what the score means.

Make a decision from the evidence you have

For this narrow lab, the adapter improves held-out answer agreement substantially, takes a short local training run, and leaves meaningful errors. It is useful for learning SFT and testing further ideas. The evidence does not justify executing its outputs against a business database without review.

A practical report includes the base model and data revisions, split rules, prompt attempts, training settings, answer scores, examples of failures, timing, and memory. Compare a stronger prompt or simple input normalization on development data before deciding that more training is the next investment.

Download the source and recorded results, or read the complete measured report. The records preserve the initial unsuccessful prompting attempt as well as the final comparison.