# Implemented #
diff --git a/search/search_index.json b/search/search_index.json
index 8531fa0d..6529b3ed 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"KubeAI: Private Open AI on Kubernetes","text":"The simple AI platform that runs on Kubernetes.
\"KubeAI is highly scalable, yet compact enough to fit on my old laptop.\" - Some Google Engineer
\u2705\ufe0f Drop-in replacement for OpenAI with API compatibility \ud83d\ude80 Serve OSS LLMs on CPUs or GPUs \u2696\ufe0f Scale from zero, autoscale based on load \ud83d\udee0\ufe0f Zero dependencies (no Istio, Knative, etc.) \ud83e\udd16 Operates OSS model servers (vLLM and Ollama) \ud83d\udd0b Additional OSS addons included (OpenWebUI i.e. ChatGPT UI) \u2709\ufe0f Plug-n-play with cloud messaging systems (Kafka, PubSub, etc.)
"},{"location":"#architecture","title":"Architecture","text":"KubeAI serves an OpenAI compatible HTTP API. Admins can configure ML models via kind: Model Kubernetes Custom Resources. KubeAI can be thought of as a Model Operator (See Operator Pattern) that manages vLLM and Ollama servers.
"},{"location":"#local-quickstart","title":"Local Quickstart","text":"Create a local cluster using kind or minikube.
TIP: If you are using Podman for kind... Make sure your Podman machine can use up to 6G of memory (by default it is capped at 2G): # You might need to stop and remove the existing machine:\npodman machine stop\npodman machine rm\n\n# Init and start a new machine:\npodman machine init --memory 6144\npodman machine start\n
kind create cluster # OR: minikube start\n
Add the KubeAI Helm repository.
helm repo add kubeai https://substratusai.github.io/kubeai/\nhelm repo update\n
Install KubeAI and wait for all components to be ready (may take a minute).
cat <<EOF > helm-values.yaml\nmodels:\n catalog:\n gemma2-2b-cpu:\n enabled: true\n minReplicas: 1\n qwen2-500m-cpu:\n enabled: true\n nomic-embed-text-cpu:\n enabled: true\nEOF\n\nhelm upgrade --install kubeai kubeai/kubeai \\\n -f ./helm-values.yaml \\\n --wait --timeout 10m\n
Before progressing to the next steps, start a watch on Pods in a standalone terminal to see how KubeAI deploys models.
kubectl get pods --watch\n
"},{"location":"#interact-with-gemma2","title":"Interact with Gemma2","text":"Because we set minReplicas: 1 for the Gemma model you should see a model Pod already coming up.
Start a local port-forward to the bundled chat UI.
kubectl port-forward svc/openwebui 8000:80\n
Now open your browser to localhost:8000 and select the Gemma model to start chatting with.
"},{"location":"#scale-up-qwen2-from-zero","title":"Scale up Qwen2 from Zero","text":"If you go back to the browser and start a chat with Qwen2, you will notice that it will take a while to respond at first. This is because we set minReplicas: 0 for this model and KubeAI needs to spin up a new Pod (you can verify with kubectl get models -oyaml qwen2-500m-cpu).
NOTE: Autoscaling after initial scale-from-zero is not yet supported for the Ollama backend which we use in this local quickstart. KubeAI relies upon backend-specific metrics and the Ollama project has an open issue: https://github.com/ollama/ollama/issues/3144. To see autoscaling in action, checkout one of the cloud install guides which uses the vLLM backend and autoscales across GPU resources.
"},{"location":"#supported-models","title":"Supported Models","text":"Any vLLM or Ollama model can be served by KubeAI. Some examples of popular models served on KubeAI include:
- Llama v3.1 (8B, 70B, 405B)
- Gemma2 (2B, 9B, 27B)
- Qwen2 (1.5B, 7B, 72B)
"},{"location":"#guides","title":"Guides","text":" - Cloud Installation - Deploy on Kubernetes clusters in the cloud
- Model Management - Manage ML models
"},{"location":"#openai-api-compatibility","title":"OpenAI API Compatibility","text":"# Implemented #\n/v1/chat/completions\n/v1/completions\n/v1/embeddings\n/v1/models\n\n# Planned #\n# /v1/assistants/*\n# /v1/batches/*\n# /v1/fine_tuning/*\n# /v1/images/*\n# /v1/vector_stores/*\n
"},{"location":"#immediate-roadmap","title":"Immediate Roadmap","text":" - Model caching
- LoRA finetuning (compatible with OpenAI finetuning API)
- Image generation (compatible with OpenAI images API)
"},{"location":"#contact","title":"Contact","text":"Let us know about features you are interested in seeing or reach out with questions. Visit our Discord channel to join the discussion!
Or just reach out on LinkedIn if you want to connect:
- Nick Stogner
- Sam Stoelinga
"},{"location":"development/","title":"Development","text":"This document provides instructions for setting up a development environment for KubeAI.
"},{"location":"development/#cloud-setup","title":"Cloud Setup","text":"gcloud pubsub topics create test-kubeai-requests\ngcloud pubsub subscriptions create test-kubeai-requests-sub --topic test-kubeai-requests\ngcloud pubsub topics create test-kubeai-responses\ngcloud pubsub subscriptions create test-kubeai-responses-sub --topic test-kubeai-responses\n
"},{"location":"development/#local-cluster","title":"Local Cluster","text":"kind create cluster\n# OR\n#./hack/create-dev-gke-cluster.yaml\n\n# When CRDs are changed reapply using kubectl:\nkubectl apply -f ./charts/kubeai/charts/crds/crds\n\n# Model with special address annotations:\nkubectl apply -f ./hack/dev-model.yaml\n\n# For developing in-cluster features:\nhelm upgrade --install kubeai ./charts/kubeai \\\n --set openwebui.enabled=true \\\n --set image.tag=latest \\\n --set image.pullPolicy=Always \\\n --set image.repository=us-central1-docker.pkg.dev/substratus-dev/default/kubeai \\\n --set replicaCount=1 # 0 if running out-of-cluster (using \"go run\")\n\n# -f ./helm-values.yaml \\\n\n# Run in development mode.\nCONFIG_PATH=./hack/dev-config.yaml POD_NAMESPACE=default go run ./cmd/main.go --allow-pod-address-override\n\n# In another terminal:\nwhile true; do kubectl port-forward service/dev-model 7000:7000; done\n
"},{"location":"development/#running","title":"Running","text":""},{"location":"development/#completions-api","title":"Completions API","text":"# If you are running kubeai in-cluster:\n# kubectl port-forward svc/kubeai 8000:80\n\ncurl http://localhost:8000/openai/v1/completions -H \"Content-Type: application/json\" -d '{\"prompt\": \"Hi\", \"model\": \"dev\"}' -v\n
"},{"location":"development/#messaging-integration","title":"Messaging Integration","text":"gcloud pubsub topics publish test-kubeai-requests \\ \n --message='{\"path\":\"/v1/completions\", \"metadata\":{\"a\":\"b\"}, \"body\": {\"model\": \"dev\", \"prompt\": \"hi\"}}'\n\ngcloud pubsub subscriptions pull test-kubeai-responses-sub --auto-ack\n
"},{"location":"model-management/","title":"Model Management","text":"KubeAI uses Model Custom Resources to configure what ML models are available in the system.
Example:
apiVersion: kubeai.org/v1\nkind: Model\nmetadata:\n name: llama-3.1-8b-instruct-fp8-l4\nspec:\n features: [\"TextGeneration\"]\n owner: neuralmagic\n url: hf://neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8\n engine: VLLM\n args:\n - --max-model-len=16384\n - --max-num-batched-token=16384\n - --gpu-memory-utilization=0.9\n minReplicas: 0\n maxReplicas: 3\n resourceProfile: L4:1\n
"},{"location":"model-management/#listing-models","title":"Listing Models","text":"You can view all installed models through the Kubernetes API using kubectl get models (use the -o yaml flag for more details).
You can also list all models via the OpenAI-compatible /v1/models endpoint:
curl http://your-deployed-kubeai-endpoint/openai/v1/models\n
"},{"location":"model-management/#installing-a-predefined-model-using-helm","title":"Installing a predefined Model using Helm","text":"When you are defining your Helm values, you can install a predefined Model by setting enabled: true:
models:\n catalog:\n llama-3.1-8b-instruct-fp8-l4:\n enabled: true\n
You can also optionally override settings for a given model:
models:\n catalog:\n llama-3.1-8b-instruct-fp8-l4:\n enabled: true\n env:\n MY_CUSTOM_ENV_VAR: \"some-value\"\n
"},{"location":"model-management/#adding-custom-models-with-helm","title":"Adding Custom Models with Helm","text":"If you prefer to add a custom model via the same Helm chart you use for installed KubeAI, you can add your custom model entry into the .models.catalog array of your existing Helm values file:
# ...\nmodels:\n catalog:\n my-custom-model-name:\n enabled: true\n features: [\"TextEmbedding\"]\n owner: me\n url: \"hf://me/my-custom-model\"\n resourceProfile: CPU:1\n
They you can re-run helm upgrade with the same flags you used to install KubeAI.
"},{"location":"model-management/#adding-custom-models-directly","title":"Adding Custom Models Directly","text":"You can add your own model by defining a Model yaml file and applying it using kubectl apply -f model.yaml.
If you have a running cluster with KubeAI installed you can inspect the schema for a Model using kubectl explain:
kubectl explain models\nkubectl explain models.spec\nkubectl explain models.spec.engine\n
"},{"location":"model-management/#model-management-ui","title":"Model Management UI","text":"We are considering adding a UI for managing models in a running KubeAI instance. Give the GitHub Issue a thumbs up if you would be interested in this feature.
"},{"location":"installation/gke/","title":"Install KubeAI on GKE","text":"TIP: Make sure you have enough quota in your GCP project. Open the cloud console quotas page: https://console.cloud.google.com/iam-admin/quotas. Make sure your project is selected in the top left.
There are 3 critical quotas you will need to verify for this guide. The minimum value here is assuming that you have nothing else running in your project.
Quota Location Min Value Preemptible NVIDIA L4 GPUs <your-region> 2 GPUs (all regions) - 2 CPUs (all regions) - 24 See the following screenshot examples of how these quotas appear in the console:
"},{"location":"installation/gke/#gke-autopilot","title":"GKE Autopilot","text":"Create an Autopilot cluster (replace us-central1 with a region that you have quota).
gcloud container clusters create-auto cluster-1 \\\n --location=us-central1\n
Define the installation values for GKE.
cat <<EOF > helm-values.yaml\nmodels:\n catalog:\n llama-3.1-8b-instruct-fp8-l4:\n enabled: true\n\nresourceProfiles:\n L4:\n nodeSelector:\n cloud.google.com/gke-accelerator: \"nvidia-l4\"\n cloud.google.com/gke-spot: \"true\"\nEOF\n
Make sure you have a HuggingFace Hub token set in your environment (HUGGING_FACE_HUB_TOKEN).
Install KubeAI with Helm.
helm repo add kubeai https://substratusai.github.io/kubeai/\nhelm repo update\n\nhelm upgrade --install kubeai kubeai/kubeai \\\n -f ./helm-values.yaml \\\n --set secrets.huggingface.token=$HUGGING_FACE_HUB_TOKEN \\\n --wait\n
"},{"location":"tutorials/langtrace/","title":"Deploying KubeAI with Langtrace","text":"Langtrace is an open source tool that helps you with tracing and monitoring your AI calls. It includes a self-hosted UI that for example shows you the estimated costs of your LLM calls.
KubeAI is used for deploying LLMs with an OpenAI compatible endpoint.
In this tutorial you will learn how to deploy KubeAI and Langtrace end-to-end. Both KubeAI and Langtrace are installed in your Kubernetes cluster. No cloud services or external dependencies are required.
If you don't have a K8s cluster yet, you can create one using kind or minikube.
kind create cluster # OR: minikube start\n
Install Langtrace:
helm repo add langtrace https://Scale3-Labs.github.io/langtrace-helm-chart\nhelm repo update\nhelm install langtrace langtrace/langtrace\n
Install KubeAI:
helm repo add kubeai https://substratusai.github.io/kubeai/\nhelm repo update\ncat <<EOF > helm-values.yaml\nmodels:\n catalog:\n gemma2-2b-cpu:\n enabled: true\n minReplicas: 1\nEOF\n\nhelm upgrade --install kubeai kubeai/kubeai \\\n --wait --timeout 10m \\\n -f ./helm-values.yaml\n
Create a local Python environment and install dependencies:
python3 -m venv .venv\nsource .venv/bin/activate\npip install langtrace-python-sdk openai\n
Expose the KubeAI service to your local port:
kubectl port-forward service/kubeai 8000:80\n
Expose the Langtrace service to your local port:
kubectl port-forward service/langtrace-app 3000:3000\n
A Langtrace API key is required to use the Langtrace SDK. So lets get one by visiting your self hosted Langtace UI.
Open your browser to http://localhost:3000, create a project and get the API keys for your langtrace project.
In the Python script below, replace langtrace_api_key with your API key.
Create file named langtrace-example.py with the following content:
# Replace this with your langtrace API key by visiting http://localhost:3000\nlangtrace_api_key=\"f7e003de19b9a628258531c17c264002e985604ca9fa561debcc85c41f357b09\"\n\nfrom langtrace_python_sdk import langtrace\nfrom langtrace_python_sdk.utils.with_root_span import with_langtrace_root_span\n# Paste this code after your langtrace init function\n\nfrom openai import OpenAI\n\nlangtrace.init(\n api_key=api_key,\n api_host=\"http://localhost:3000/api/trace\",\n)\n\nbase_url = \"http://localhost:8000/openai/v1\"\nmodel = \"gemma2-2b-cpu\"\n\n@with_langtrace_root_span()\ndef example():\n client = OpenAI(base_url=base_url, api_key=\"ignored-by-kubeai\")\n response = client.chat.completions.create(\n model=model,\n messages=[\n {\n \"role\": \"system\",\n \"content\": \"How many states of matter are there?\"\n }\n ],\n )\n print(response.choices[0].message.content)\n\nexample()\n
Run the Python script:
python3 langtrace-example.py\n
Now you should see the trace in your Langtrace UI. Take a look by visiting http://localhost:3000.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"KubeAI: Private Open AI on Kubernetes","text":"The simple AI platform that runs on Kubernetes.
\"KubeAI is highly scalable, yet compact enough to fit on my old laptop.\" - Some Google Engineer
\u2705\ufe0f Drop-in replacement for OpenAI with API compatibility \ud83d\ude80 Serve OSS LLMs on CPUs or GPUs \u2696\ufe0f Scale from zero, autoscale based on load \ud83d\udee0\ufe0f Zero dependencies (no Istio, Knative, etc.) \ud83e\udd16 Operates OSS model servers (vLLM and Ollama) \ud83d\udd0b Additional OSS addons included (OpenWebUI i.e. ChatGPT UI) \u2709\ufe0f Plug-n-play with cloud messaging systems (Kafka, PubSub, etc.)
"},{"location":"#architecture","title":"Architecture","text":"KubeAI serves an OpenAI compatible HTTP API. Admins can configure ML models via kind: Model Kubernetes Custom Resources. KubeAI can be thought of as a Model Operator (See Operator Pattern) that manages vLLM and Ollama servers.
"},{"location":"#local-quickstart","title":"Local Quickstart","text":"Create a local cluster using kind or minikube.
TIP: If you are using Podman for kind... Make sure your Podman machine can use up to 6G of memory (by default it is capped at 2G): # You might need to stop and remove the existing machine:\npodman machine stop\npodman machine rm\n\n# Init and start a new machine:\npodman machine init --memory 6144\npodman machine start\n
kind create cluster # OR: minikube start\n
Add the KubeAI Helm repository.
helm repo add kubeai https://substratusai.github.io/kubeai/\nhelm repo update\n
Install KubeAI and wait for all components to be ready (may take a minute).
cat <<EOF > helm-values.yaml\nmodels:\n catalog:\n gemma2-2b-cpu:\n enabled: true\n minReplicas: 1\n qwen2-500m-cpu:\n enabled: true\n nomic-embed-text-cpu:\n enabled: true\nEOF\n\nhelm upgrade --install kubeai kubeai/kubeai \\\n -f ./helm-values.yaml \\\n --wait --timeout 10m\n
Before progressing to the next steps, start a watch on Pods in a standalone terminal to see how KubeAI deploys models.
kubectl get pods --watch\n
"},{"location":"#interact-with-gemma2","title":"Interact with Gemma2","text":"Because we set minReplicas: 1 for the Gemma model you should see a model Pod already coming up.
Start a local port-forward to the bundled chat UI.
kubectl port-forward svc/openwebui 8000:80\n
Now open your browser to localhost:8000 and select the Gemma model to start chatting with.
"},{"location":"#scale-up-qwen2-from-zero","title":"Scale up Qwen2 from Zero","text":"If you go back to the browser and start a chat with Qwen2, you will notice that it will take a while to respond at first. This is because we set minReplicas: 0 for this model and KubeAI needs to spin up a new Pod (you can verify with kubectl get models -oyaml qwen2-500m-cpu).
NOTE: Autoscaling after initial scale-from-zero is not yet supported for the Ollama backend which we use in this local quickstart. KubeAI relies upon backend-specific metrics and the Ollama project has an open issue: https://github.com/ollama/ollama/issues/3144. To see autoscaling in action, checkout one of the cloud install guides which uses the vLLM backend and autoscales across GPU resources.
"},{"location":"#supported-models","title":"Supported Models","text":"Any vLLM or Ollama model can be served by KubeAI. Some examples of popular models served on KubeAI include:
- Llama v3.1 (8B, 70B, 405B)
- Gemma2 (2B, 9B, 27B)
- Qwen2 (1.5B, 7B, 72B)
"},{"location":"#guides","title":"Guides","text":" - Installation on GKE - Deploy on Kubernetes clusters in the cloud
- Model Management - Manage ML models
"},{"location":"#openai-api-compatibility","title":"OpenAI API Compatibility","text":"# Implemented #\n/v1/chat/completions\n/v1/completions\n/v1/embeddings\n/v1/models\n\n# Planned #\n# /v1/assistants/*\n# /v1/batches/*\n# /v1/fine_tuning/*\n# /v1/images/*\n# /v1/vector_stores/*\n
"},{"location":"#immediate-roadmap","title":"Immediate Roadmap","text":" - Model caching
- LoRA finetuning (compatible with OpenAI finetuning API)
- Image generation (compatible with OpenAI images API)
"},{"location":"#contact","title":"Contact","text":"Let us know about features you are interested in seeing or reach out with questions. Visit our Discord channel to join the discussion!
Or just reach out on LinkedIn if you want to connect:
- Nick Stogner
- Sam Stoelinga
"},{"location":"development/","title":"Development","text":"This document provides instructions for setting up a development environment for KubeAI.
"},{"location":"development/#cloud-setup","title":"Cloud Setup","text":"gcloud pubsub topics create test-kubeai-requests\ngcloud pubsub subscriptions create test-kubeai-requests-sub --topic test-kubeai-requests\ngcloud pubsub topics create test-kubeai-responses\ngcloud pubsub subscriptions create test-kubeai-responses-sub --topic test-kubeai-responses\n
"},{"location":"development/#local-cluster","title":"Local Cluster","text":"kind create cluster\n# OR\n#./hack/create-dev-gke-cluster.yaml\n\n# When CRDs are changed reapply using kubectl:\nkubectl apply -f ./charts/kubeai/charts/crds/crds\n\n# Model with special address annotations:\nkubectl apply -f ./hack/dev-model.yaml\n\n# For developing in-cluster features:\nhelm upgrade --install kubeai ./charts/kubeai \\\n --set openwebui.enabled=true \\\n --set image.tag=latest \\\n --set image.pullPolicy=Always \\\n --set image.repository=us-central1-docker.pkg.dev/substratus-dev/default/kubeai \\\n --set replicaCount=1 # 0 if running out-of-cluster (using \"go run\")\n\n# -f ./helm-values.yaml \\\n\n# Run in development mode.\nCONFIG_PATH=./hack/dev-config.yaml POD_NAMESPACE=default go run ./cmd/main.go --allow-pod-address-override\n\n# In another terminal:\nwhile true; do kubectl port-forward service/dev-model 7000:7000; done\n
"},{"location":"development/#running","title":"Running","text":""},{"location":"development/#completions-api","title":"Completions API","text":"# If you are running kubeai in-cluster:\n# kubectl port-forward svc/kubeai 8000:80\n\ncurl http://localhost:8000/openai/v1/completions -H \"Content-Type: application/json\" -d '{\"prompt\": \"Hi\", \"model\": \"dev\"}' -v\n
"},{"location":"development/#messaging-integration","title":"Messaging Integration","text":"gcloud pubsub topics publish test-kubeai-requests \\ \n --message='{\"path\":\"/v1/completions\", \"metadata\":{\"a\":\"b\"}, \"body\": {\"model\": \"dev\", \"prompt\": \"hi\"}}'\n\ngcloud pubsub subscriptions pull test-kubeai-responses-sub --auto-ack\n
"},{"location":"model-management/","title":"Model Management","text":"KubeAI uses Model Custom Resources to configure what ML models are available in the system.
Example:
apiVersion: kubeai.org/v1\nkind: Model\nmetadata:\n name: llama-3.1-8b-instruct-fp8-l4\nspec:\n features: [\"TextGeneration\"]\n owner: neuralmagic\n url: hf://neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8\n engine: VLLM\n args:\n - --max-model-len=16384\n - --max-num-batched-token=16384\n - --gpu-memory-utilization=0.9\n minReplicas: 0\n maxReplicas: 3\n resourceProfile: L4:1\n
"},{"location":"model-management/#listing-models","title":"Listing Models","text":"You can view all installed models through the Kubernetes API using kubectl get models (use the -o yaml flag for more details).
You can also list all models via the OpenAI-compatible /v1/models endpoint:
curl http://your-deployed-kubeai-endpoint/openai/v1/models\n
"},{"location":"model-management/#installing-a-predefined-model-using-helm","title":"Installing a predefined Model using Helm","text":"When you are defining your Helm values, you can install a predefined Model by setting enabled: true:
models:\n catalog:\n llama-3.1-8b-instruct-fp8-l4:\n enabled: true\n
You can also optionally override settings for a given model:
models:\n catalog:\n llama-3.1-8b-instruct-fp8-l4:\n enabled: true\n env:\n MY_CUSTOM_ENV_VAR: \"some-value\"\n
"},{"location":"model-management/#adding-custom-models-with-helm","title":"Adding Custom Models with Helm","text":"If you prefer to add a custom model via the same Helm chart you use for installed KubeAI, you can add your custom model entry into the .models.catalog array of your existing Helm values file:
# ...\nmodels:\n catalog:\n my-custom-model-name:\n enabled: true\n features: [\"TextEmbedding\"]\n owner: me\n url: \"hf://me/my-custom-model\"\n resourceProfile: CPU:1\n
They you can re-run helm upgrade with the same flags you used to install KubeAI.
"},{"location":"model-management/#adding-custom-models-directly","title":"Adding Custom Models Directly","text":"You can add your own model by defining a Model yaml file and applying it using kubectl apply -f model.yaml.
If you have a running cluster with KubeAI installed you can inspect the schema for a Model using kubectl explain:
kubectl explain models\nkubectl explain models.spec\nkubectl explain models.spec.engine\n
"},{"location":"model-management/#model-management-ui","title":"Model Management UI","text":"We are considering adding a UI for managing models in a running KubeAI instance. Give the GitHub Issue a thumbs up if you would be interested in this feature.
"},{"location":"installation/gke/","title":"Install KubeAI on GKE","text":"TIP: Make sure you have enough quota in your GCP project. Open the cloud console quotas page: https://console.cloud.google.com/iam-admin/quotas. Make sure your project is selected in the top left.
There are 3 critical quotas you will need to verify for this guide. The minimum value here is assuming that you have nothing else running in your project.
Quota Location Min Value Preemptible NVIDIA L4 GPUs <your-region> 2 GPUs (all regions) - 2 CPUs (all regions) - 24 See the following screenshot examples of how these quotas appear in the console:
"},{"location":"installation/gke/#gke-autopilot","title":"GKE Autopilot","text":"Create an Autopilot cluster (replace us-central1 with a region that you have quota).
gcloud container clusters create-auto cluster-1 \\\n --location=us-central1\n
Define the installation values for GKE.
cat <<EOF > helm-values.yaml\nmodels:\n catalog:\n llama-3.1-8b-instruct-fp8-l4:\n enabled: true\n\nresourceProfiles:\n L4:\n nodeSelector:\n cloud.google.com/gke-accelerator: \"nvidia-l4\"\n cloud.google.com/gke-spot: \"true\"\nEOF\n
Make sure you have a HuggingFace Hub token set in your environment (HUGGING_FACE_HUB_TOKEN).
Install KubeAI with Helm.
helm repo add kubeai https://substratusai.github.io/kubeai/\nhelm repo update\n\nhelm upgrade --install kubeai kubeai/kubeai \\\n -f ./helm-values.yaml \\\n --set secrets.huggingface.token=$HUGGING_FACE_HUB_TOKEN \\\n --wait\n
"},{"location":"tutorials/langtrace/","title":"Deploying KubeAI with Langtrace","text":"Langtrace is an open source tool that helps you with tracing and monitoring your AI calls. It includes a self-hosted UI that for example shows you the estimated costs of your LLM calls.
KubeAI is used for deploying LLMs with an OpenAI compatible endpoint.
In this tutorial you will learn how to deploy KubeAI and Langtrace end-to-end. Both KubeAI and Langtrace are installed in your Kubernetes cluster. No cloud services or external dependencies are required.
If you don't have a K8s cluster yet, you can create one using kind or minikube.
kind create cluster # OR: minikube start\n
Install Langtrace:
helm repo add langtrace https://Scale3-Labs.github.io/langtrace-helm-chart\nhelm repo update\nhelm install langtrace langtrace/langtrace\n
Install KubeAI:
helm repo add kubeai https://substratusai.github.io/kubeai/\nhelm repo update\ncat <<EOF > helm-values.yaml\nmodels:\n catalog:\n gemma2-2b-cpu:\n enabled: true\n minReplicas: 1\nEOF\n\nhelm upgrade --install kubeai kubeai/kubeai \\\n --wait --timeout 10m \\\n -f ./helm-values.yaml\n
Create a local Python environment and install dependencies:
python3 -m venv .venv\nsource .venv/bin/activate\npip install langtrace-python-sdk openai\n
Expose the KubeAI service to your local port:
kubectl port-forward service/kubeai 8000:80\n
Expose the Langtrace service to your local port:
kubectl port-forward service/langtrace-app 3000:3000\n
A Langtrace API key is required to use the Langtrace SDK. So lets get one by visiting your self hosted Langtace UI.
Open your browser to http://localhost:3000, create a project and get the API keys for your langtrace project.
In the Python script below, replace langtrace_api_key with your API key.
Create file named langtrace-example.py with the following content:
# Replace this with your langtrace API key by visiting http://localhost:3000\nlangtrace_api_key=\"f7e003de19b9a628258531c17c264002e985604ca9fa561debcc85c41f357b09\"\n\nfrom langtrace_python_sdk import langtrace\nfrom langtrace_python_sdk.utils.with_root_span import with_langtrace_root_span\n# Paste this code after your langtrace init function\n\nfrom openai import OpenAI\n\nlangtrace.init(\n api_key=api_key,\n api_host=\"http://localhost:3000/api/trace\",\n)\n\nbase_url = \"http://localhost:8000/openai/v1\"\nmodel = \"gemma2-2b-cpu\"\n\n@with_langtrace_root_span()\ndef example():\n client = OpenAI(base_url=base_url, api_key=\"ignored-by-kubeai\")\n response = client.chat.completions.create(\n model=model,\n messages=[\n {\n \"role\": \"system\",\n \"content\": \"How many states of matter are there?\"\n }\n ],\n )\n print(response.choices[0].message.content)\n\nexample()\n
Run the Python script:
python3 langtrace-example.py\n
Now you should see the trace in your Langtrace UI. Take a look by visiting http://localhost:3000.
"}]}
\ No newline at end of file