Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support: OpenAI base url #5

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Conversation

Tim-Saijun
Copy link

No description provided.

Copy link
Member

@xiaohk xiaohk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! This template setting was from https://github.com/mlc-ai/web-llm/blob/ca1c3486755462dfcdc32699c651275e3fd863ce/src/conversation.ts#L158. Could you please explain your fix? Web LLM might have some different template format than other frameworks.

Copy link
Member

@xiaohk xiaohk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! Thanks for the PR!

I didn't think about any other base URLs for OpenAI. What are some use cases for other base URLs?

@xiaohk
Copy link
Member

xiaohk commented Mar 20, 2024

Thanks for the PR! This template setting was from https://github.com/mlc-ai/web-llm/blob/ca1c3486755462dfcdc32699c651275e3fd863ce/src/conversation.ts#L158. Could you please explain your fix? Web LLM might have some different template format than other frameworks.

Oh I see. It seems they have changed the format at some point. Thank you so much for noticing it and adding a fix!

Can you tell me more about the base URL please?

@Tim-Saijun
Copy link
Author

Base URL Usage

OpenAI's base URL is the API endpoint used for accessing services provided by OpenAI. By configing a different base url, user can access openai's services by forwarding. In fact, it is supported in the official SDK

For example, when user cannot directly acess the offical endpoint (https://api.openai.com/v1) due to network, they can deploy their own endpoint, just like Deploy a Worker that connects to OpenAI via AI Gateway.

By using self-hosted endpoint, user can also manage their requests (quato limit, ratio limit, get an overview of usage). Whats more, some user may choose a third-party openai api service provider (lower price, company's platform or any other reasons), so they need to config the base url.

About the commit

Base url is an optional configuration. I have set the default of base url, when user did not config it , the base url is the offical one. This means the commit is not destructive and will not interfere with existing uses.

For the consistency of code processing logic, the baseURL property is also set for google's and web_llm family, even though the baseURL can only work on OpenAI family. This is not perfect and may need further refactor.

About the fix in web-llm

Actually i dont know how to fix it, changes made in web-llm is just to shut up the builder. 🤣 So further fixes are needed.

@xiaohk xiaohk mentioned this pull request May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants