Inference with multiple backends?

[sabbih-shah]

Suppose that I have the following model repo:

model_repo
|
-> classifier
     |-> classifier_v0.pytorch
     |-> classifier_v1.tensorrt
-> detector
     |-> detector_v0.tensorrt
     |-> detector_v1.tensorflow
-> segementor
     |-> segementor_v0.tensorrt
     |-> segementor_v1.tensorrt

Also, the models should be executed in the following manner:

request -> classifier -> detector -> segementor -> output

Now assume that we have hundreds of model versions in different. So, by default we only load the v0 of all of the models due to gpu memory limitations. Now we use "Model Control Mode EXPLICIT" to load/unload the different model versions depending on the input request.

Do I need to write a custom backend to handle the model execution or is this achievable using Triton's built-in functionality?

[dyastremsky]

There should be no issue with running inference on models using multiple backends. Triton will load them, as needed, based on the models you load.

That said, I don't think you can do the above with most of the backgrounds. Different model versions still use the same config, so they'll need to use the same backend. It may be possible to do the above with a custom backend or the Python backend.