Running multi-gpu and replicating models

[JoJoLev]

Currently have an LLM engine built on TensorRT-LLM. Trying to evaluate different setups and gains on types.
Was trying to deploy the llama model on a multi-gpu, whereby between the 4 GPUs, I would have a copy of the model running on each.
Is this possible with NVIDIA triton inference container?

[rmccorm4]

Hi @JoJoLev, there is a guide for this exact use case here: https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/llama_multi_instance.md. Please let us know if this helps.