feat: use reduced message to avoid maxing out tokens #21
Conversation
| return int((len(message) - message.count(' ')) / chars_per_token) + json_padding | ||
|
|
||
|
|
||
| def get_reduced_message_list(system_list, message_list): |
There was a problem hiding this comment.
This was largely taken from https://github.com/2uinc/ai-chatbot-backend/blob/74faef96cc3732e1369d3ad5adeae9cdd79d0bc8/xpert/pipelines/utils.py#L173. I modified a few things within the while loop (like moving a break and adding it to the outer conditional) and changing the append + reverse to an insert for the new_message_list
| """ | ||
| If messages are larger than allotted token amount, return a smaller list of messages. | ||
| """ | ||
| total_system_tokens = sum(_estimated_message_tokens(system_message['content']) for system_message in system_list) |
There was a problem hiding this comment.
This doesn't currently account for if the total_system_tokens is greater than the number of allowed tokens, but I couldn't find anything in https://github.com/2uinc/ai-chatbot-backend on the same codepath as this function that checks for that. I think it's fine as is but probably worth noting in the discovery for https://2u-internal.atlassian.net/browse/MST-2122.
MST-2134
To avoid going over the maximum amount of tokens allowed, we should reduce the user message list and send only the most recent messages that do not go over the number of allotted tokens.