Skip to content

Commit

Permalink
fix: check for tokenizer eos_token in ModelInfo response (IBM#93)
Browse files Browse the repository at this point in the history
Use consistent logic to determine the eos_token_id in ModelInfo as it is
in other functions by falling back to the tokenizer's `eos_token_id`
attribute if the model config does not have an `eos_token_id`.

Fixes the behavior for a model that does not have an eos_token_id
in the model config

Resolves IBM#91

Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
  • Loading branch information
tjohnson31415 authored May 10, 2024
1 parent 2358566 commit 0734973
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion server/text_generation_server/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ async def ModelInfo(self, request: generate_pb2.ModelInfoRequest, context) -> ge
return generate_pb2.ModelInfoResponse(
model_type=ModelInfoResponse.ModelType.SEQ2SEQ_LM
if isinstance(self.model, Seq2SeqLM) else ModelInfoResponse.ModelType.CAUSAL_LM,
eos_token=self.model.config.eos_token_id,
eos_token=getattr(self.model.tokenizer, 'model_eos_token_id', self.model.tokenizer.eos_token_id),
batch_padding=not isinstance(self.model, FlashCausalLM),
memory_scaling_model=self.memory_scaling_model,
)
Expand Down

0 comments on commit 0734973

Please sign in to comment.