Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ai-proxy): enable compatibility with LLM SDKs, model selection by request-parameters #12807

Closed
wants to merge 16 commits into from
Closed

feat(ai-proxy): enable compatibility with LLM SDKs, model selection by request-parameters #12807

wants to merge 16 commits into from

Conversation

tysoekong
Copy link
Contributor

@tysoekong tysoekong commented Mar 28, 2024 •
edited by kikito
Loading

Summary

Our users have noted that they wanted to have the option to not use the plugin configured model "tuning" options, but rather use something like e.g. the OpenAI SDK, and send the options in per-request.

This function enables this.

Checklist

  • The Pull Request has tests
  • A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
  • There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

KAG-4127
https://konghq.atlassian.net/browse/KAG-4124

@tysoekong tysoekong force-pushed the feat/KAG-4127-ai-proxy-allow-override-defaults branch 2 times, most recently from a0234ae to 8227a29 Compare March 29, 2024 03:25
@pull-request-size pull-request-size bot added size/L and removed size/M labels Mar 29, 2024
@tysoekong tysoekong force-pushed the feat/KAG-4127-ai-proxy-allow-override-defaults branch 2 times, most recently from 5d1b3e6 to f2bef33 Compare March 29, 2024 03:27
@tysoekong tysoekong marked this pull request as ready for review March 29, 2024 03:27
@tysoekong tysoekong force-pushed the feat/KAG-4127-ai-proxy-allow-override-defaults branch from 699b3ff to 4b89b4e Compare April 1, 2024 01:04
@ttyS0e
Copy link
Contributor

ttyS0e commented Apr 1, 2024

This one is ready to go.

@tysoekong tysoekong force-pushed the feat/KAG-4127-ai-proxy-allow-override-defaults branch from 4b89b4e to 78d5b46 Compare April 8, 2024 13:48
@RobSerafini RobSerafini added this to the 3.7.0 milestone Apr 8, 2024
@tysoekong tysoekong changed the title feat(ai-proxy): allow overriding default model options feat(ai-proxy): enable compatibility with LLM SDKs Apr 10, 2024
@tysoekong tysoekong changed the title feat(ai-proxy): enable compatibility with LLM SDKs feat(ai-proxy): enable compatibility with LLM SDKs, model selection by request-parameters Apr 10, 2024
flrgh
flrgh reviewed Apr 10, 2024
View reviewed changes
kong/llm/drivers/azure.lua Outdated Show resolved Hide resolved
flrgh
flrgh reviewed Apr 10, 2024
View reviewed changes
kong/llm/drivers/cohere.lua Outdated Show resolved Hide resolved
flrgh
flrgh reviewed Apr 10, 2024
View reviewed changes
kong/llm/drivers/openai.lua Outdated Show resolved Hide resolved
flrgh
flrgh reviewed Apr 10, 2024
View reviewed changes
kong/llm/drivers/shared.lua Outdated Show resolved Hide resolved
@tysoekong tysoekong force-pushed the feat/KAG-4127-ai-proxy-allow-override-defaults branch from 84378da to 01b693b Compare April 11, 2024 00:24
flrgh
flrgh reviewed Apr 11, 2024
View reviewed changes
kong/llm/drivers/shared.lua Outdated Show resolved Hide resolved
flrgh
flrgh approved these changes Apr 11, 2024
View reviewed changes
Copy link
Contributor

@flrgh flrgh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've one remaining gripe about a kong.log.warn(), but we're just about good to go.

Nice work!

@tysoekong tysoekong force-pushed the feat/KAG-4127-ai-proxy-allow-override-defaults branch from 4ed111d to 3c28675 Compare April 11, 2024 11:25
@tysoekong
Copy link
Contributor Author

@flrgh Fixed it, oops

@tysoekong tysoekong force-pushed the feat/KAG-4127-ai-proxy-allow-override-defaults branch from 3c28675 to 3461c29 Compare April 11, 2024 16:57
hanshuebner
hanshuebner approved these changes Apr 15, 2024
View reviewed changes
Copy link
Contributor

@hanshuebner hanshuebner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of style comments and questions, overall looks good.

kong/llm/drivers/cohere.lua
return false, "cannot use own model for this instance"
end

-- noop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can this check be completely be removed in the cohere driver, but the anthropic driver still has a check (albeit only checking for matching names?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/Kong/kong/pull/12903/files#r1575194343

kong/llm/drivers/cohere.lua
request_table.truncate = request_table.truncate or "END"
request_table.return_likelihoods = request_table.return_likelihoods or "NONE"
request_table.p = request_table.top_p or model.options.top_p
request_table.k = request_table.top_k or model.options.top_k
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it OK to have request_table.p be false if top_p is neither set in the request nor in the options? In the other drivers, there code to merge the request table with the model options, but it is done in a per-option fashion. It might be better to have a generalized merging function that works for all drivers, in the same way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/Kong/kong/pull/12903/files#r1575191319

kong/llm/drivers/cohere.lua
-- and move the LAST message (from 'user') into "message" string
if #request_table.messages > 1 then
local chat_history = table_new(#request_table.messages - 1, 0)
for i, v in ipairs(request_table.messages) do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A numeric for loop from 1 to #request_table.messages - 1 would be better than ipairs here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/Kong/kong/pull/12903/files#r1575192689

kong/llm/drivers/llama2.lua
messages.parameters.max_new_tokens = request_table.max_tokens or (model.options and model.options.max_tokens)
messages.parameters.top_p = request_table.top_p or (model.options and model.options.top_p)
messages.parameters.top_k = request_table.top_k or (model.options and model.options.top_k)
messages.parameters.temperature = request_table.temperature or (model.options and model.options.temperature)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems being yet another pattern of merging request_table and model.options, a unification across drivers would make sense.

kong/llm/drivers/openai.lua
end

return request
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we have the merge function that all drivers should be using.

kong/llm/drivers/azure.lua
)
parsed_url = socket_url.parse(url)
end

if string.sub(parsed_url.path, 1, 1) ~= "/" then
parsed_url.path = "/" .. parsed_url.path
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A common function to prepend a slash to relative paths should be introduced and used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR was re-done, due to messy rebases from other features.

See: https://github.com/Kong/kong/pull/12903/files#r1575189707

kong/llm/drivers/anthropic.lua Outdated
if body and body.model then
return nil, "cannot use own model for this instance"
if body and body.model and (body.model ~= conf.model.name) then
return nil, "requested model does not match the configured plugin model"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it sufficient to check the name of the model for equivalence?

@tysoekong tysoekong force-pushed the feat/KAG-4127-ai-proxy-allow-override-defaults branch from 72e8164 to 8bd5fe3 Compare April 18, 2024 10:55
@tysoekong
Copy link
Contributor Author

This is going to be superseded by a different PR, closing.

@tysoekong tysoekong closed this Apr 19, 2024
@tysoekong tysoekong mentioned this pull request Apr 22, 2024
Merged
3 tasks
tysoekong added a commit that referenced this pull request Apr 24, 2024
@tysoekong
feat(ai-proxy): folded in features from #12807
d3f50cf
tysoekong added a commit that referenced this pull request Apr 25, 2024
@tysoekong
feat(ai-proxy): folded in features from #12807
8ae0d51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hanshuebner hanshuebner hanshuebner approved these changes

@flrgh flrgh flrgh approved these changes

Assignees

@tysoekong tysoekong

Labels
cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee plugins/ai-proxy plugins/ai-request-transformer plugins/ai-response-transformer size/XXL
Projects
None yet
Milestone
3.7.0
Development

Successfully merging this pull request may close these issues.
5 participants
@tysoekong @ttyS0e @hanshuebner @flrgh @RobSerafini