Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does Triton support multiple TensorFlow backends simultaneously? #7698

Open
ragavendrams opened this issue Oct 14, 2024 · 0 comments
Open

Does Triton support multiple TensorFlow backends simultaneously? #7698

ragavendrams opened this issue Oct 14, 2024 · 0 comments

Comments

@ragavendrams
Copy link

I would like to know if Triton supports multiple Tensorflow backends at the same time (e.g Tensorflow 2.13 and 2.16).

Use case:
I have an application whose v1 requires Tensorflow 2.13 and v2 requires Tensorflow 2.16. Both versions of the application are in production (using a different inference server) and I would like to support both using one triton server instance to prevent having to allocate multiple GPUs (i.e one for a triton instance with Tensorflow 2.13 backend and another for a triton instance with Tensorflow 2.16 backend).

Known solution:
I have read about Multi Instance GPUs which can can be used to split the GPU and allocate one to each instance of Triton. But this is not supported in all NVIDIA GPUs (Eg: 2080Ti). So I would like to explore other options.

Is this possible?

Thanks in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant