Skip to content

Latest commit

 

History

History
82 lines (57 loc) · 2.65 KB

installation.md

File metadata and controls

82 lines (57 loc) · 2.65 KB

Installation Guide

Prerequisites

Requirements:

  • Kubernetes version >= 1.26. LWS requires Kubernetes version v1.26 or higher. If you are using a lower Kubernetes version and most of your workloads rely on single-node inference, we may consider replacing LWS with a Deployment-based approach. This fallback plan would involve using Kubernetes Deployments to manage single-node inference workloads efficiently. See #32 for more details and updates.
  • Helm 3, see installation.
  • Prometheus, see installation.

Note: llmaz helm chart will by default install

  • Envoy Gateway and Envoy AI Gateway as the frontier in the llmaz-system, if you already installed these two components or want to deploy in other namespaces , append --set envoy-gateway.enabled=false --set envoy-ai-gateway.enabled=false to the command below.
  • Open WebUI as the default chatbot, if you want to disable it, append --set open-webui.enabled=false to the command below.

Install a released version

Install

helm repo add inftyai https://inftyai.github.io/llmaz
helm repo update
helm install llmaz inftyai/llmaz --namespace llmaz-system --create-namespace --version 0.0.9

Uninstall

helm uninstall llmaz --namespace llmaz-system
kubectl delete ns llmaz-system

If you want to delete the CRDs as well, run

kubectl delete crd \
    openmodels.llmaz.io \
    backendruntimes.inference.llmaz.io \
    playgrounds.inference.llmaz.io \
    services.inference.llmaz.io

Install from source

Change configurations

If you want to change the default configurations, please change the values in values.global.yaml.

Do you change the values in values.yaml because it's auto-generated and will be overwritten.

Install

git clone https://github.com/inftyai/llmaz.git && cd llmaz
kubectl create ns llmaz-system && kubens llmaz-system
make helm-install

Uninstall

helm uninstall llmaz --namespace llmaz-system
kubectl delete ns llmaz-system

If you want to delete the CRDs as well, run

kubectl delete crd \
    openmodels.llmaz.io \
    backendruntimes.inference.llmaz.io \
    playgrounds.inference.llmaz.io \
    services.inference.llmaz.io

Upgrade

Once you changed your code, run the command to upgrade the controller:

IMG=<image-registry>:<tag> make helm-upgrade