[AI-3] Probe models for embedding support automatically - #24886
Conversation
|
Caution The provided work package version does not match the core version Details:
Please make sure that:
|
|
Caution The Enterprise plan field is not set on the work package Details:
Please make sure that:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e475d8679
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
6e475d8 to
87cc4ee
Compare
87cc4ee to
c83347e
Compare
c83347e to
82901e2
Compare
Adds the one behavioural probe worth writing: a single embeddings request per candidate model, whose 200 response also carries the vector dimension count. Probing every listed model would be wrong, since a gateway can list hundreds and some providers bill per request, so the background pass after a credential change is capped at ten models whose names suggest they embed. The name is only a hint for where to spend a probe, never a verdict in itself. A definite refusal records unsupported, a vector records supported, and anything else records nothing at all, because servers silently drop unknown parameters and a 200 alone proves nothing. Administrator assertions are never overwritten. Part 8 of the AI-3 stack. https://community.openproject.org/work_packages/66020
82901e2 to
d6182c5
Compare
|
Warning Flaky specs
🤖 Ask Copilot to investigateCopy the prompt below into a new comment on this PR to delegate the investigation to GitHub Copilot. It will look into the flakiness and open a separate pull request with you as reviewer. |
Ticket
AI-3
What are you trying to accomplish?
PR 8 of 11 in the AI-3 stack. The one behavioural probe worth writing: a single embeddings request per candidate model, whose 200 response also carries the vector dimension count. The background pass after a credential change is capped at ten models whose names suggest they embed, because a gateway can list hundreds and some providers bill per request. A definite refusal records unsupported, a vector records supported, and anything else records nothing, because servers silently drop unknown parameters and a 200 alone proves nothing. Administrator assertions are never overwritten.
Merge checklist
llm_connectionfeature flagStacked on #24885.