Skip to content

Latest commit

 

History

History
258 lines (198 loc) · 11.6 KB

embedded-config.mdx

File metadata and controls

258 lines (198 loc) · 11.6 KB

Embedded Cluster Config

This topic is a reference for the Replicated Embedded Cluster Config custom resource. For more information about Embedded Cluster, see Using Embedded Cluster.

Overview

To install your application with Embedded Cluster, an Embedded Cluster Config must be included in the release. Embedded Cluster installation artifacts are available only for releases that include an Embedded Cluster Config.

The Embedded Cluster Config lets you define several aspects of the Kubernetes cluster that will be created.

Limitation

The Embedded Cluster Config does not support the use of Go template functions, including KOTS template functions.

For additional property-specific limitations, see the sections below.

Example

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  version: 2.1.3+k8s-1.30
  roles:
    controller:	
      name: app	
      labels:	
        app: "true"
    custom:
    - name: gpu
      labels:
        gpu: "true"
    - name: database
      labels:
        database: "true"
  domains:
    proxyRegistryDomain: proxy.yourcompany.com
    replicatedAppDomain: updates.yourcompany.com
  extensions:
    helm:
      repositories:
        - name: ingress-nginx
          url: https://kubernetes.github.io/ingress-nginx
      charts:
        - name: ingress-nginx
          chartname: ingress-nginx/ingress-nginx
          namespace: ingress-nginx
          version: "4.8.3"
          values: |
            controller:
              service:
                type: NodePort
                nodePorts:
                  http: "80"
                  https: "443"
              # Known issue: Only use image tags for multi-architecture images.
              # Set digest to empty string to ensure the air gap builder uses
              # single-architecture images.
              image:
                digest: ""
                digestChroot: ""
              admissionWebhooks:
                patch:
                  image:
                    digest: ""

version

You must specify which version of Embedded Cluster to install. Each version of Embedded Cluster includes particular versions of components like KOTS (Admin Console) and OpenEBS.

For a full list of versions, see the Embedded Cluster releases page in GitHub. It's recommended to keep this version as up to date as possible because Embedded Cluster is changing rapidly.

roles

You can optionally customize node roles in the Embedded Cluster Config using the roles key.

A common use case for customizing node roles is to assign workloads to specific nodes. For example, if your application has graphics processing unit (GPU) workloads, you could create a custom role that will add a gpu=true label to any node that is assigned the role. This allows you to then schedule GPU workloads on nodes labled gpu=true.

When the roles key is configured, users select one or more roles to assign to a node when it is joined to the cluster. For more information, see Managing Multi-Node Clusters with Embedded Cluster.

If the roles key is not configured, all nodes joined to the cluster are assigned the controller role. The controller role designates nodes that run the Kubernetes control plane. Controller nodes can also run other workloads, such as application or Replicated KOTS workloads.

Example

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  roles:
    controller:
      # Optionally change the name for the default controller role
      name: app
      labels:
        app: "true" # Label applied to "app" nodes
    # Custom roles
    custom:
    - name: db
      labels:
        db: "true" # Label applied to "db" nodes
    - name: gpu
      labels:
        gpu: "true" # Label applied to "gpu" nodes

roles.controller

In the roles.controller key, you can set the following fields to customize the default controller role:

  • name: Set the name that is assigned to controller nodes. By default, controller nodes are named “controller”. :::note If you plan to create any custom roles, Replicated recommends that you change the default name for the controller role to a term that is easy to understand, such as "app". This is because, when you add custom roles, both the name of the controller role and the names of any custom roles are displayed to the user when they join a node. :::
  • labels: Kubernetes labels that Embedded Cluster will apply to any node in the cluster that is assigned the given role.

roles.custom

In the roles.custom key, you can add custom roles. Each custom role includes the following fields:

  • name: (Required) A name for the custom role.
  • labels: Kubernetes labels that Embedded Cluster will apply to any node in the cluster that is assigned the given role.

domains

Configure the domains key so that Embedded Cluster uses your custom domains for the Replicated proxy registry and Replicated app service.

When domains.proxyRegistryDomain and domains.replicatedAppDomain are set, Embedded Cluster uses the custom domains specified when making requests to the given service. Embedded Cluster also passes the values to KOTS to ensure that KOTS uses the same domains for these services.

The custom domains that you specify in the domains.proxyRegistryDomain and domains.replicatedAppDomain fields must be added to the Vendor Portal before they can be used by Embedded Cluster. For more information, see Add a Custom Domain in the Vendor Portal in Using Custom Domains.

If domains.proxyRegistryDomain and domains.replicatedAppDomain are not set, Embedded Cluster uses the default Replicated domains. For more information about aliasing Replicated endpoints with custom domains, see About Custom Domains.

Example

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  domains:
    # Your proxy registry custom domain
    proxyRegistryDomain: proxy.yourcompany.com
    # Your app service custom domain
    replicatedAppDomain: updates.yourcompany.com

extensions

If you need to install Helm charts before your application and as part of the Embedded Cluster itself, you can do this with Helm extensions. One situation where this is useful is if you want to ship an ingress controller, because Embedded Cluster does not yet include one.

Helm extensions are updated when new versions of your application are deployed from the Admin Console. So, for example, you can change the values for a Helm extension from one release to another, and those changes will be applied to the cluster when the new release is deployed.

The format for specifying Helm extensions uses the same k0s Helm extensions format from the k0s configuration. For more information about these fields, see the k0s documentation.

Example

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  extensions:
    helm:
      repositories:
        - name: ingress-nginx
          url: https://kubernetes.github.io/ingress-nginx
      charts:
        - name: ingress-nginx
          chartname: ingress-nginx/ingress-nginx
          namespace: ingress-nginx
          version: "4.8.3"
          values: |
            controller:
              service:
                type: NodePort
                nodePorts:
                  http: "80"
                  https: "443"
              # Known issue: Only use image tags for multi-architecture images.
              # Set digest to empty string to ensure the air gap builder uses
              # single-architecture images.
              image:
                digest: ""
                digestChroot: ""
              admissionWebhooks:
                patch:
                  image:
                    digest: ""

Limitation

If a Helm extension is removed from the Embedded Cluster Config, the associated Helm chart is not removed from the cluster.

Requirements

  • The version field is required. Failing to specify a chart version will cause problems for upgrades.

  • If you need to install multiple charts in a particular order, set the order field to a value greater than or equal to 10. Numbers below 10 are reserved for use by Embedded Cluster to deploy things like a storage provider and the Admin Console. If an order is not provided, Helm extensions are installed with order 10.

unsupportedOverrides

:::important This feature should be used with caution by advanced users who understand the risks and ramifications of changing the default configuration. :::

Unsupported overrides allow you to override Embedded Cluster's default configuration, including the k0s config and the Helm values for extensions like KOTS and OpenEBS. This should be used with caution because changes here are untested and can disrupt or break Embedded Clusters. Any issues that are caused by unsupported overrides are not supported.

While they should be used with caution, unsupported overrides are useful if you need to make changes that are not otherwise exposed by Embedded Cluster.

Override the k0s Config

By default, Embedded Cluster uses a k0s config that is tested and known to work for Embedded Clusters. In some circumstances, you might want to change the k0s config.

For more information on the k0s config, see Configuration options in the k0s documentation.

For example, you can do the following to enable WireGuard-based encryption. Note that other configuration might be necessary. See spec.network.calico in the k0s documentation for more details.

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  unsupportedOverrides:
    k0s: |
      config:
        spec:
          network:
            calico:
              wireguard: true

Limtiations

  • The spec.api and spec.storage keys in the k0s config cannot be changed after installation. Whereas most keys in the k0s config apply to the whole cluster, these two keys are set for each node. Embedded Cluster cannot update these keys on each individual node during updates, so they cannot be changed after installation.

  • Overrides overwrite the corresponding fields in the k0s configuration. They are not merged into Embedded Cluster’s default configuration. When using overrides to override a list, for example, ensure that you include other elements in the list that Embedded Cluster includes by default.

Override the Helm Values for Built-In Extensions

Embedded Cluster deploys built-in extensions like KOTS and OpenEBS to provide capabilities like storage and application management. These extensions are deployed with Helm, and the Helm values for each can be modified if necessary.

To modify these values, you can use the unsupportedOverrides.builtInExtensions key of the Embedded Cluster Config. Each chart you want to modify is an item in the array. The name key identifies the Helm chart that you want to modify, and the values key is a string where you specify your modified Helm values. Your modified values are merged into the values used by Embedded Cluster.

The following are the built-in extensions available for modification:

  • openebs
  • admin-console
  • velero
  • embedded-cluster-operator

Example

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  unsupportedOverrides:
    builtInExtensions:
    - name: openebs
      values: |
        key: value