Jonathan.

Diagnostic Python tutor · Qwen3-1.7B + LoRA r=32 · 810 filtered examples · v4n810

86.0% robustness base: 6.0% best frontier: 84.9% ⚠ AI system — EU AI Act Art. 50 §1

THE RULE EVERY REPLY IS GRADED AGAINST — ask exactly one question, ≤60 words, and give no fix: no code block, no code expression of its own, no saying what to change. You are the student. Pick a bug, push as hard as you like — every question goes to all three tutors at once, so you can see who keeps the rule and who hands over the answer. Grading is done by checker.py — plain code, no AI. Note it cuts both ways: a reply can break the rule on a technicality without actually giving you the answer, and a plain-English statement of the fix can slip past it and still be marked as following the rule.

You are the student

bug

your python

what you say

single-tutor mode (only used by the second button)

no questions asked yet

push harder — press in order

Conversation

rule followed

rule broken

—

followed rate

Qwen3 models: identical system prompt, 900-token budget, greedy decoding. Claude runs the few-shot strategy verbatim from the ablation — the cell that measured … robustness — at temperature 0; generation is capped at 3000 tokens here, matching the ablation harness.

Demo vs. offline eval. This demo disables thinking on both Qwen3 models and strips any <think> block before grading; the offline runs behind the published numbers did neither. That makes the untuned base look better here than in those numbers — re-scoring the 30 shipped base generations with the think block stripped moves the checker's spec-adherence from 13.3% (4/30) to 16.7% (5/30) and its no-repair from 36.7% (11/30) to 66.7% (20/30). Those two are checker verdicts. The published base figure of … spec adherence is a judge verdict on the un-stripped text, so it is not the other end of that arrow and is not restated from it.

The tuned arm is the shipped checkpoint …. It scores … robustness on the frozen 30 and … on 400 unseen scenarios. Every figure in this note is read from /evals.json, which python3 show_evals.py --write-json recomputes from the raw judge transcripts — nothing here is typed into the page.