Skip to content

[AI-3] Probe models for embedding support automatically - #24886

Open
tangopium wants to merge 1 commit into
feature/66020-pr07-model-curationfrom
feature/66020-pr08-embedding-detection
Open

[AI-3] Probe models for embedding support automatically#24886
tangopium wants to merge 1 commit into
feature/66020-pr07-model-curationfrom
feature/66020-pr08-embedding-detection

Conversation

@tangopium

@tangopium tangopium commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

Ticket

AI-3

What are you trying to accomplish?

PR 8 of 11 in the AI-3 stack. The one behavioural probe worth writing: a single embeddings request per candidate model, whose 200 response also carries the vector dimension count. The background pass after a credential change is capped at ten models whose names suggest they embed, because a gateway can list hundreds and some providers bill per request. A definite refusal records unsupported, a vector records supported, and anything else records nothing, because servers silently drop unknown parameters and a 200 alone proves nothing. Administrator assertions are never overwritten.

Merge checklist

  • Added/updated tests
  • Behind the llm_connection feature flag

Stacked on #24885.

@github-actions

Copy link
Copy Markdown

Caution

The provided work package version does not match the core version

Details:

Please make sure that:

  • The work package version OR your pull request target branch is correct

@github-actions

Copy link
Copy Markdown

Caution

The Enterprise plan field is not set on the work package

Details:

Please make sure that:

  • The work package Enterprise plan field is set

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e475d8679

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread app/services/llm_connections/detect_capabilities_service.rb Outdated
Comment thread app/services/llm_connections/detect_capabilities_service.rb Outdated
Comment thread app/services/llm_connections/detect_capabilities_service.rb Outdated
Comment thread app/services/llm/probes/embeddings_probe.rb Outdated
@tangopium
tangopium force-pushed the feature/66020-pr08-embedding-detection branch from 6e475d8 to 87cc4ee Compare August 22, 2026 15:05
@tangopium tangopium changed the title [#66020] Probe models for embedding support automatically [AI-3] Probe models for embedding support automatically Aug 22, 2026
@tangopium
tangopium force-pushed the feature/66020-pr08-embedding-detection branch from 87cc4ee to c83347e Compare August 22, 2026 15:26
@tangopium
tangopium force-pushed the feature/66020-pr08-embedding-detection branch from c83347e to 82901e2 Compare August 22, 2026 15:51
Adds the one behavioural probe worth writing: a single embeddings request per candidate model, whose 200 response also carries the vector dimension count. Probing every listed model would be wrong, since a gateway can list hundreds and some providers bill per request, so the background pass after a credential change is capped at ten models whose names suggest they embed. The name is only a hint for where to spend a probe, never a verdict in itself.

A definite refusal records unsupported, a vector records supported, and anything else records nothing at all, because servers silently drop unknown parameters and a 200 alone proves nothing. Administrator assertions are never overwritten.

Part 8 of the AI-3 stack.

https://community.openproject.org/work_packages/66020
@tangopium
tangopium force-pushed the feature/66020-pr08-embedding-detection branch from 82901e2 to d6182c5 Compare August 22, 2026 16:16
@github-actions

Copy link
Copy Markdown

Warning

Flaky specs

  • rspec ./spec/features/work_packages/table/switch_types_spec.rb[1:1:1]
🤖 Ask Copilot to investigate

Copy the prompt below into a new comment on this PR to delegate the investigation to GitHub Copilot. It will look into the flakiness and open a separate pull request with you as reviewer.

@copilot The following spec(s) are flaky in CI (first seen on PR #24886, linked for reference only):

- `rspec ./spec/features/work_packages/table/switch_types_spec.rb[1:1:1]`

Treat this as a standalone task, unrelated to PR #24886. Create a new branch from origin/dev and open a new pull request targeting dev — do not stack it on PR #24886 or reuse that branch.

Follow the playbook in docs/development/testing/handling-flaky-tests/README.md to find the root cause and fix the underlying race — do not skip, delete, or weaken the spec to make it pass; disabling is a last resort per the playbook, and only with a bug ticket. Verify the fix by running the spec(s) repeatedly (e.g. `script/bulk_run_rspec --run-count 10`).

If you cannot reproduce the flake or are not confident in a fix after reasonable investigation, do not fabricate a change or skip the spec to force CI green. Instead, leave the pull request in draft and document what you tried, the suspected cause, and any leads in its description, then assign @tangopium to take over.

Once the fix is verified, title the PR after the spec(s) it fixes, and use the PR description to explain the root cause, how the change resolves it, and the before/after results. Label the PR `flaky-spec`, assign @tangopium, and request a review from @tangopium.
On every commit, set @tangopium as the sole co-author with a `Co-authored-by:` trailer (use their GitHub no-reply email so it links to their account), so it is traceable who dispatched the fix.

@tangopium tangopium self-assigned this Aug 22, 2026
@tangopium
tangopium requested a review from thykel August 22, 2026 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant