How to Truncate the input prompt?

[AamodThakur]

We are using vicuna model and setting the maximum length to 4096.

llm = LLM(model="lmsys/vicuna-7b-v1.5", max_model_len=4096, max_num_batched_tokens = 4096, tensor_parallel_size=2)

We are sending input prompt of more than 10K, and wanted it to be truncated to 4096. But we are getting erorr "Input prompt (25597 tokens) is too long and exceeds limit of 4096".

How can we set truncate to true in vllm?

Output of get tokenizer:
LlamaTokenizerFast(name_or_path='lmsys/vicuna-7b-v1.5', vocab_size=32000, model_max_length=4096, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'pad_token': '<unk>'}, clean_up_tokenization_spaces=False), added_tokens_decoder={ 0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 1: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), }

Thanks.

[tomdzh]

Same question.

[chenxu2048]

You can encode prompt with tokenizer and call generate with truncated prompt tokens. The code might be:

llm = LLM(model="lmsys/vicuna-7b-v1.5", max_model_len=4096, max_num_batched_tokens = 4096, tensor_parallel_size=2)
tokenizer = llm.get_tokenizer()
prompt_token_ids = tokenizer.encode("<PROMPT>", return_tensors="pt")

# Truncate prompt_token_ids
prompt_token_ids = prompt_token_ids[-MAX_PROMPT_TOKEN:]

llm.generate(prompt_token_ids=prompt_token_ids)

[AamodThakur]

Thank for your response,

I getting error ("TypeError: unsupported operand type(s) for +: 'Tensor' and 'list'") for the above code given.
Also getting similar error at different part of code when if "prompt_token_ids" converted to list.

Thanks.

[chenxu2048]

Thank you for your correction! llm.generate accepts a List[List[int]] type instead of a torch.Tensor.

Also getting similar error at different part of code when if "prompt_token_ids" converted to list.

This code was tested on my environment with the latest main branch, and it should work.

from vllm import LLM

llm = LLM(model="lmsys/vicuna-7b-v1.5", max_model_len=4096, max_num_batched_tokens = 4096)
tokenizer = llm.get_tokenizer()
prompt_token_ids = tokenizer.encode("def main", return_tensors="pt")

# Truncate prompt_token_ids
prompt_token_ids = prompt_token_ids[-4096:]

llm.generate(prompt_token_ids=prompt_token_ids.tolist())

[AamodThakur]

Thanks for the answer, it helped a lot.

[AamodThakur]

For new users, vllm version >= 0.3.0 does not have "return_tensors="pt"".
It directly returns list. So no need to do "prompt_token_ids.tolist()"

[chenxu2048]

Thanks for pointing it out.