Tuning the dynamic-course generator

The dials you turn when v2 output is wrong, and what each one actually does.

This is the companion to apps/skillnet-api/scripts/quality_bench.py, which is the only honest way to tell whether a change helped. The loop is: turn one dial, run the bench, read the failure dump. A percentage without the failures in front of it does not tell you what to fix.

cd apps/skillnet-api
uv run python scripts/quality_bench.py --offline     # no key, checks the harness works
uv run python scripts/quality_bench.py --repeat 3    # real provider from .env
uv run python scripts/quality_bench.py --only extintor --model groq/openai/gpt-oss-120b

The bench runs the real pipeline (run_node_render → build_node_graph() → the eight nodes of src/agents/runtime/nodes.py). It fakes exactly three seams — the DB session, SSE, and a wrapper under litellm.acompletion that adds a User-Agent and backs off on 429. It reports first-try / repaired / fallback / error rates, p50 and p95 latency, tokens and USD, diffs against the previous run, and dumps every failed output with the validator’s reason. Useful flags: --repeat, --only, --offline, --out, --compare, --pause, --model / --model-fast / --model-heavy, --price-in / --price-out, --dump-prompts, --list, --seed.

Every value below was read out of the module named above it. There are deliberately no line numbers: the column existed until 2026-08-27, when it was checked against the code and 28 of 45 citations were wrong — and a stale line number does not land nowhere, it lands somewhere real and plausible-looking. The module path stays, because a path is stable and a constant name is a stable, unique, greppable address inside it.

What we measured, and what it means for tuning

Measured against real Groq on 2026-07-27 (groq/llama-3.1-8b-instant as the fast tier, groq/openai/gpt-oss-120b as the heavy one): sub-second to about 3 s per render, at roughly 0.0008 USD per render, with token accounting populated.

The “20-30 second generation” problem the research phase assumed does not exist on this stack. The 60-150 s figures in the internal sources came from a 7B model on local CPU. This matters for tuning because it removes the reason to pre-generate, to add waiting layers, or to trade quality for latency: there is no latency budget under pressure. Spend the dials on correctness, not on speed.

The one real operational constraint is that Groq’s free tier returns 429 readily. Any measurement run needs backoff — the bench has it built in (RATE_LIMIT_MAX_RETRIES = 5, RATE_LIMIT_BASE_DELAY = 4.0, RATE_LIMIT_MAX_DELAY = 90.0, quality_bench.py:119-121), and it counts the waiting separately in rate_limit_seconds so a rate limit can never be scored as a quality failure. If you write your own harness, do the same or your numbers are noise.

1. Which model, and how much room it gets

src/config.py

src/agents/runtime/router.py

2. Prompts and budgets

src/llm/prompts/runtime.py

The dialect fragment is never written by hand. ui_generator_system() is src.render.prompt.render_prompt() — the artefact generated from the frontend kit — plus the answer-key protocol. Do not paste component signatures into this module; that is the drift tests/test_render_prompt_artifact.py exists to catch.

3. The gate: what gets refused before parsing

src/render/gate.py. These are safety limits, not quality dials — but a program rejected here shows up in the bench as a repair or a fallback, so they read like quality problems.

The reactivity check is deliberately light and textual: it blanks every string literal first, then checks the remaining skeleton against the alphabet the frozen grammar can produce. That order is the whole trick — a keyword grep over raw text has measured false positives on legitimate prose (“en SQL una Query() se escribe con SELECT”, “cuesta $300”).

4. The mastery rule

src/services/mastery_service.py. Everything in this module is pure — no DB, no LLM, no clock you did not pass in — because the rule that decides what a certificate says has to be testable case by case. Changing any of these changes what “mastered” means, so change them with the unit tests open.

One dial was removed rather than retuned (2026-08-28). REPROBE_COOLDOWN_DAYS (7) gated the re-probe on “only from needs_review, and only after this long”. Migration 0033 removed the needs_review state, so the first half of that gate can never hold again and a cooldown on its own decides nothing — keeping it would have widened the rule into “anyone may re-probe a node they already mastered, one week later”. probe_service._authorize_reprobe now refuses unconditionally, and which condition should replace the state is a product decision, not a dial.

5. Runtime graph behaviour

src/agents/runtime/nodes.py

src/services/learner_profile_service.py

The bench corpus deliberately includes learners with nodes_completed < 3 (extintor, prevencion-riesgos) precisely so this branch is measured.

6. Working without an API key

src/config.py

FixtureLLMService resolves by exact hash of (system, user), so a new corpus has no recordings and would fail on the first call. Note the consequence for tuning: changing a prompt invalidates every fixture. Re-record, or use --offline, which self-seeds.

Where the numbers live afterwards

• llm_usage_log — per call: use_case, purpose, model, tier, tokens_in, tokens_out, duration_ms. This is what answers “is the fast/heavy ratio really 90/10”.
• node_renders — per render: tokens_* and duration_ms for the whole render, both calls and every retry included.
• Token counts are None, never 0, when nobody counted. 0 claims the call was free; coalescing the two would make an unmeasured cost model look measured.