POC: AI model version selector with per-project settings - #255
POC: AI model version selector with per-project settings#255lucashuizinga wants to merge 1 commit into
Conversation
DataForSEO's llm_responses catalog moved on (claude-sonnet-5, gpt-5.5, gemini-3.6) while the model_name pins stayed frozen, so AI visibility checks measured answers from superseded models. This makes the version visible and selectable instead of hardcoded: - Per-provider version dropdown in Prompt Explorer, encoded in the URL (mv=claude:claude-sonnet-4-6) so shared links and history reproduce the exact run - Settings > AI Models: linkable per-project defaults page backed by a project_ai_models table (SQLite + Postgres); resolution is per-run choice -> project setting -> app default - Response cache now keys on the resolved model version, so a version switch can't serve week-old answers from a different model - Claude app default bumped to claude-sonnet-5 (the version consumer Claude serves; also cheaper per token than 4.5); allowlist synced with the live catalog and a test enforces every selectable version stays on it, since DataForSEO bills rejected model names Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015W5xJH8uH6VmH5jH6ee2nD
![chatgpt-codex-connector[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7f6be25ea5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| projectId, | ||
| trimmedPrompt, | ||
| urlState.models.toSorted().join(","), | ||
| modelVersionsKey, |
There was a problem hiding this comment.
When a project default is changed and a prompt has no explicit mv override, this key remains unchanged because it includes only urlState.modelVersions. Returning to the same prompt within the five-minute staleTime therefore reuses the response from the old model even though the form displays the new default, which can silently corrupt comparisons. Include the resolved project defaults in the key (and wait for them to load), or invalidate the project's Prompt Explorer queries whenever settings change.
Useful? React with 👍 / 👎.
1 participant
Proof of concept for #254 — I understand external PRs aren't being merged right now; this exists to make the issue concrete, per CONTRIBUTING.md.
What it does:
mv=claude:claude-sonnet-4-6, so shared links and history reproduce the exact run. The results card already showedmodel_name, so which version answered is visible./p/:projectId/settings/ai-models), backed by aproject_ai_modelstable (SQLite + Postgres mirror, parity test passes). Resolution is per-run choice → project setting → app default; a choice equal to the app default is stored as "no choice" so projects keep tracking app-level bumps.Invalid Field: 'model_name', so an unlisted version must never dispatch.claude-sonnet-5(what consumer Claude serves; also cheaper per token than 4.5). ChatGPT/Gemini defaults are left as-is — only the selectable lists grew.Tested:
tscclean, 230 tests pass (including schema parity and the two new test files), and the full flow verified in a browser — save a project default, see Prompt Explorer pick it up, override per run, URL round-trip.🤖 Generated with Claude Code
https://claude.ai/code/session_015W5xJH8uH6VmH5jH6ee2nD