Dangling resources if the deploymentMode annotation is manipulated

[israel-hdez]

InferenceServices have the special annotation serving.kserve.io/deploymentMode which can be set to either ModelMesh for deploying models using ModelMesh, or be absent for deploying models for KServe.

At the moment, in ODH there is no support for converting InferenceServices between ModelMesh and KServe. So, once an InferenceService is created, it is not expected that the annotation would change. If the user wants to switch the deployment mode of a model, it is expected that another InferenceService is created, and the old one is deleted.

Though, there is an issue: currently, there is no validation for rejecting a change in the special annotation serving.kserve.io/deploymentMode, which means that the user may accidentally change it. If this happens, the model/resources may enter in an inconsistent state. Also, odh-model-controller may not do a proper clean-up of the created resources if, later, the InferenceService is deleted.

There are two possible fixes for this:

• Since ODH does not support conversion between deploymentModes, it may be OK to enter in an inconsistent state. However, if the InferenceService is deleted, clean-up must be done properly.
• Since ODH does not support conversion between deploymentModes, it makes sense to add a ValidatingWebhook to reject a change in the annotation.