Onboard Katio to Kubernetes services hosted by other cloud vendors

[Fei-Guo]

Tasks

Beta Give feedback

1. feat: [SKU modularization] sku handler interface #357
   +0
2. feat: [SKU modularization] adding azure sku handler #360
   +0
3. feat: [SKU modularization] adding aws sku handler #364
   +0
4. feat: [SKU-modularization] Get SKU Handler #518
   +0
5. feat: [SKU modularization] remove sku_config from v1alpha1 and implement skuHandler interface #602
   +0
6. Update workspace controller to use cloud-specific sku handler to retreive skus/gpu configs
7. Test: verify Kaito works on AWS using self-provisioned gpu-nodes
8. Test: verify Kaito works on AWS with Karpenter integration
9. Make updates to Kaito charts for AWS support
10. Add installation script for AWS
11. Update docs
12. add/update tests

[Fei-Guo]

We will start from deploying Kaito in AWS since AWS does support Karpenter, making the integration easier.

[javaniecampbell]

Hey all, just quick question does this feature enhancement will it include self-hosted kubernetes, I checked a few places but wasn't sure so I figured maybe this could be right place to see if this is considered?

The consideration is that some will need for self-host community, home-labs and companies, etc that need the llms to be ran locally.

[Fei-Guo]

Hey all, just quick question does this feature enhancement will it include self-hosted kubernetes, I checked a few places but wasn't sure so I figured maybe this could be right place to see if this is considered?

The consideration is that some will need for self-host community, home-labs and companies, etc that need the llms to be ran locally.

You can run Kaito in selfmanaged k8s if you already add GPU nodes in the cluster (with proper gpu driver and k8s plugin installed). In this case, you can just add those nodes in the Kaito workspace CR as preferrednodes in the Resource spec. Kaito will skip provisioning gpu nodes and just run inference workload in the existing nodes.