Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Short term] Implement retries and waiting for concurrent API calls in the case of throttling. #1

Open
pashpashpash opened this issue Apr 16, 2024 · 0 comments

Comments

@pashpashpash
Copy link
Owner

pashpashpash commented Apr 16, 2024

I have had some people tell me that the GPT concurrent calls I am making right now are being throttled -- most likely because my Openai organization has higher concurrency limits compared to new accounts. This can be solved by

  1. Limiting max concurrent requests
  2. Retries should be implemented as well.

https://github.com/pashpashpash/manga-reader/blob/main/app.py#L47-L66

https://github.com/pashpashpash/manga-reader/blob/main/app.py#L173-L185

From someone who attempted to run the code:

openai.RateLimitError: Error code: 429 - {'error': {'message': 'Request too large for gpt-4-vision-preview in organization org-xxxxxxxxxx on tokens per min (TPM): Limit 10000, Requested 25012. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

@pashpashpash pashpashpash changed the title Implement retries and waiting for concurrent API calls in the case of throttling. [Short term] Implement retries and waiting for concurrent API calls in the case of throttling. Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant