why ngram speculative decode got the diff result with no speculative decode

[liuteng]

In my case, ngram speculative got diff result with no speculative decode, and even diff with different ngram-prompt-lookup-min args (1 and 2)

I just run the same model with different vllm args like:
python -m vllm.entrypoints.api_server --host 127.0.0.1 --trust-remote-code --disable-custom-all-reduce --use-v2-block-manager --enable-prefix-caching --max-model-len 3072 --gpu-memory-utilization 0.7 --model /path/to/model/ --port 9122
and
python -m vllm.entrypoints.api_server --host 127.0.0.1 --trust-remote-code --disable-custom-all-reduce --use-v2-block-manager --enable-prefix-caching --max-model-len 3072 --gpu-memory-utilization 0.7 --model /path/to/model/ --port 9122 --speculative-model [ngram] --num-speculative-tokens 6 --ngram-prompt-lookup-max 5 --ngram-prompt-lookup-min 2

is it should got the same result when I use the same query with "temperature" == 0 ?

[liuteng]

I tried v0.4.2 to v0.6.2, all of them has the same problem

[liuteng]

I changed self.disable_logprobs = True to self.disable_logprobs = False in TargetModelRunner
and then got a confused log info about the prefill output, the outputs gives the right result, but sampled_token_ids has wrong:

[[SamplerOutput(outputs=[CompletionSequenceGroupOutput(samples=[SequenceOutput(parent_seq_id=0, output_token=50006, logprobs={50006: Logprob(logprob=0.0, rank=1, decoded_token=None)})], prompt_logprobs=None)], sampled_token_probs=torch.Size([1, 115584]), sampled_token_ids=[[26888]], spec_decode_worker_metrics=None)]]

maybe there is something wrong in func _sample_with_torch in sampler.py

[comaniac]

Can you try to use an open model like llama3 to see if the problem can be reproduced, and posted your command (both server and client) here to help investigation? Thanks

[liuteng]

Hi, I finally find the cause, we miss the sop process in speculative decode as it is in _build_sampler_output:

if is_prompt and prms.sop_token_id:
seq_outputs.append(
SequenceOutput(seq_id, prms.sop_token_id, {prms.sop_token_id: Logprob(0.0, 1)}))
else:
seq_outputs.append(
SequenceOutput(seq_id, next_token_id, logprobs))

and we should add this one?

[comaniac]

I don't really get it, but if it's an easy fix please feel free to directly send a PR. Thanks

[liuteng]

I'm sorry, this is a issue about our own version of vllm, not this public one

[liuteng]

but still, I has another case that is not caused by the sop_token_id
in that case, the begin part of the outputs are the same in both normal decode and ngram decode ,but got diff result from some where
maybe there are two token have the same probs I thhink as the model is train by some spec data