Skip to main content
Version: v1.5

Installation

Koordinator requires Kubernetes version >= 1.18.

Koordinator may collect metrics from kubelet read-only port (disabled by default, where the secure port is used). You can get more info form here.

For the best experience, koordinator recommends linux kernel 4.19 or higher.

Install with helms

Koordinator can be simply installed by helm v3.5+, which is a simple command-line tool, and you can get it from here.

# Firstly add koordinator charts repository if you haven't do this.
$ helm repo add koordinator-sh https://koordinator-sh.github.io/charts/

# [Optional]
$ helm repo update

# Install the latest version.
$ helm install koordinator koordinator-sh/koordinator --version 1.5.0

Upgrade with helm

# Firstly add koordinator charts repository if you haven't do this.
$ helm repo add koordinator-sh https://koordinator-sh.github.io/charts/

# [Optional]
$ helm repo update

# Upgrade the latest version.
$ helm upgrade koordinator koordinator-sh/koordinator --version 1.5.0 [--force]

Note that:

  1. Before upgrade, you must firstly read the Change Log to make sure that you have understand the breaking changes in the new version.
  2. If you want to drop the chart parameters you configured for the old release or set some new parameters, it is recommended to add --reset-values flag in helm upgrade command. Otherwise, you should use --reuse-values flag to reuse the last release's values.

Optional: download charts manually

If you have problem with connecting to https://koordinator-sh.github.io/charts/ in production, you might need to download the chart from here manually and install or upgrade with it.

$ helm install/upgrade koordinator /PATH/TO/CHART

Optional: Enable NRI Mode Resource Management

Prerequisite

  • Containerd >= 1.7.0 and enable NRI. Please make sure NRI is enabled in containerd. If not, please refer to Enable NRI in Containerd
  • Koordinator >= 1.3

Configurations

NRI mode resource management is Enabled by default. You can use it without any modification on the koordlet config. You can also disable it to set enable-nri-runtime-hook=false in koordlet start args. It doesn't matter if all prerequisites are not meet. You can use all other features as expected.

For more details, please refer to NRI Mode Resource Management.

Options

Note that installing this chart directly means it will use the default template values for Koordinator.

You may have to set your specific configurations if it is deployed into a production cluster, or you want to configure feature-gates.

Optional: chart parameters

The following table lists the configurable parameters of the chart and their default values.

ParameterDescriptionDefault
featureGatesFeature gates for Koordinator, empty string means all by default
installation.namespacenamespace for Koordinator installationkoordinator-system
installation.createNamespaceWhether to create the installation.namespacetrue
imageRepositoryHostImage repository hostghcr.io
manager.log.levelLog level that koord-manager printed4
manager.replicasReplicas of koord-manager deployment2
manager.image.repositoryRepository for koord-manager imagekoordinatorsh/koord-manager
manager.image.tagTag for koord-manager imagev1.5.0
manager.resources.limits.cpuCPU resource limit of koord-manager container1000m
manager.resources.limits.memoryMemory resource limit of koord-manager container1Gi
manager.resources.requests.cpuCPU resource request of koord-manager container500m
manager.resources.requests.memoryMemory resource request of koord-manager container256Mi
manager.metrics.portPort of metrics served8080
manager.webhook.portPort of webhook served9443
manager.nodeAffinityNode affinity policy for koord-manager pod{}
manager.nodeSelectorNode labels for koord-manager pod{}
manager.tolerationsTolerations for koord-manager pod[]
manager.resyncPeriodResync period of informer koord-manager, defaults no resync0
manager.hostNetworkWhether koord-manager pod should run with hostnetworkfalse
scheduler.log.levelLog level that koord-scheduler printed4
scheduler.replicasReplicas of koord-scheduler deployment2
scheduler.image.repositoryRepository for koord-scheduler imagekoordinatorsh/koord-scheduler
scheduler.image.tagTag for koord-scheduler imagev1.5.0
scheduler.resources.limits.cpuCPU resource limit of koord-scheduler container1000m
scheduler.resources.limits.memoryMemory resource limit of koord-scheduler container1Gi
scheduler.resources.requests.cpuCPU resource request of koord-scheduler container500m
scheduler.resources.requests.memoryMemory resource request of koord-scheduler container256Mi
scheduler.portPort of metrics served10251
scheduler.nodeAffinityNode affinity policy for koord-scheduler pod{}
scheduler.nodeSelectorNode labels for koord-scheduler pod{}
scheduler.tolerationsTolerations for koord-scheduler pod[]
scheduler.hostNetworkWhether koord-scheduler pod should run with hostnetworkfalse
koordlet.log.levelLog level that koordlet printed4
koordlet.image.repositoryRepository for koordlet imagekoordinatorsh/koordlet
koordlet.image.tagTag for koordlet imagev1.5.0
koordlet.resources.limits.cpuCPU resource limit of koordlet container500m
koordlet.resources.limits.memoryMemory resource limit of koordlet container256Mi
koordlet.resources.requests.cpuCPU resource request of koordlet container0
koordlet.resources.requests.memoryMemory resource request of koordlet container0
koordlet.enableServiceMonitorWhether to enable ServiceMonitor for koordletfalse
webhookConfiguration.failurePolicy.podsThe failurePolicy for pods in mutating webhook configurationIgnore
webhookConfiguration.timeoutSecondsThe timeoutSeconds for all webhook configuration30
crds.managedKoordinator will not install CRDs with chart if this is falsetrue
imagePullSecretsThe list of image pull secrets for koordinator imagefalse

Specify each parameter using the --set key=value[,key=value] argument to helm install or helm upgrade.

Optional: feature-gate

Feature-gate controls some influential features in Koordinator:

NameDescriptionDefaultEffect (if closed)
PodMutatingWebhookWhether to open a mutating webhook for Pod createtrueDon't inject koordinator.sh/qosClass, koordinator.sh/priority and don't replace koordinator extend resources ad so on
PodValidatingWebhookWhether to open a validating webhook for Pod create/updatetrueIt is possible to create some Pods that do not conform to the Koordinator specification, causing some unpredictable problems

If you want to configure the feature-gate, just set the parameter when install or upgrade. Such as:

$ helm install koordinator https://... --set featureGates="PodMutatingWebhook=true\,PodValidatingWebhook=true"

If you want to enable all feature-gates, set the parameter as featureGates=AllAlpha=true.

Optional: install or upgrade specific CRDs

If you want to skip specific CRDs during the installation or the upgrade, you can set the parameter crds.<crdPluralName> to false and install or upgrade them manually.

# skip install specific CRD
$ helm install koordinator https://... --set crds.managed=true,crds.noderesourcetopologies=false
# only upgrade specific CRDs
$ helm upgrade koordinator https://... --set crds.managed=false,crds.recommendations=true,crds.clustercolocationprofiles=true,crds.elasticquotaprofiles=true,crds.elasticquotas=true,crds.devices=true,crds.podgroups=true

Optional: the local image for China

If you are in China and have problem to pull image from official DockerHub, you can use the registry hosted on Alibaba Cloud:

$ helm install koordinator https://... --set imageRepositoryHost=registry.cn-beijing.aliyuncs.com

Best Practices

Installation parameters for AWS EKS

When using a custom CNI (such as Weave or Calico) on EKS, the webhook cannot be reached by default. This happens because the control plane cannot be configured to run on a custom CNI on EKS, so the CNIs differ between control plane and worker nodes.

To address this, the webhook can be run in the host network, so it can be reached, by setting --set manager.hostNetwork=true when use helm install or upgrade.

Installation parameters for Alibaba Cloud ACK

To install or upgrade Koordinator on Alibaba Cloud ACK, you need to skip some CRDs because the ACK cluster should already prepare them, and they cannot be takeover by the Helm.

e.g. You may get the error below:

$ helm install koordinator koordinator-sh/koordinator --version 1.5.0
Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: CustomResourceDefinition "reservations.scheduling.koordinator.sh" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "koordinator"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"

To resolve the conflict error, you can install or upgrade with the conflict CRDs disabled. Please refer to Optional: install or upgrade specific CRDs.

  1. (Optional) Check the Koordinator-related CRDs already deployed by ACK.
# for the latest CRDs, please refer to https://github.com/koordinator-sh/koordinator/blob/main/charts/koordinator/crds/crds.yaml
$ kubectl get crd | grep "nodemetrics\|noderesourcetopologies\|elasticquotas\|podgroup\|reservations"
elasticquotas.scheduling.sigs.k8s.io 1970-01-01T00:00:00Z
nodemetrics.slo.koordinator.sh 1970-01-01T00:00:00Z
noderesourcetopologies.topology.node.k8s.io 1970-01-01T00:00:00Z
podgroups.scheduling.sigs.k8s.io 1970-01-01T00:00:00Z
reservations.scheduling.koordinator.sh 1970-01-01T00:00:00Z
  1. Install or upgrade Koordinator without the deployed CRDs.
# install without the deployed CRDs
$ helm install koordinator https://... --set crds.managed=true,crds.nodemetrics=false,crds.noderesourcetopologies=false,crds.elasticquotas=false,crds.podgroups=false,crds.reservations=false

# upgrade without the deployed CRDs
$ helm upgrade koordinator https://... --set crds.managed=true,crds.nodemetrics=false,crds.noderesourcetopologies=false,crds.elasticquotas=false,crds.podgroups=false,crds.reservations=false

Uninstall

Note that this will lead to all resources created by Koordinator, including webhook configurations, services, namespace, CRDs and CR instances managed by Koordinator controller, to be deleted!

Please do this ONLY when you fully understand the consequence.

To uninstall koordinator if it is installed with helm charts:

$ helm uninstall koordinator
release "koordinator" uninstalled