KoordRuntimeProxy acts as a proxy between kubelet and containerd(dockerd under dockershim scenario), which is designed to intercept CRI request, and apply some resource management policies, such as setting different cgroup parameters by pod priorities under hybrid workload orchestration scenario, applying new isolation policies for latest Linux kernel, CPU architecture, and etc.
There are two components involved, KoordRuntimeProxy and RuntimePlugins.
- Enhance resource management for QoS based Scheduling.
- Provide interface for new isolation features which are not supported by CRI.
KoordRuntimeProxy is in charge of intercepting request during pod's lifecycle, such as RunPodSandbox, CreateContainer etc., and then calling RuntimePlugins to do resource isolation policies before transferring request to backend containerd(dockerd) and after transferring response to kubelet. KoordRuntimeProxy provides an isolation-policy-execution framework which allows customized plugins registered to do specified isolation policies, these plugins are called RuntimePlugins. KoordRuntimeProxy itself does NOT do any isolation policies.
RuntimePlugins register events(RunPodSandbox etc.) to KoordRuntimeProxy and would receive notifications when events happen. RuntimePlugins should complete resource isolation policies basing on the notification message, and then response KoordRuntimeProxy, KoordRuntimeProxy would decide to transfer request to backend containerd or discard request according to plugins' response.
If no RuntimePlugins registered, KoordRuntimeProxy would become a transparent proxy between kubelet and containerd.
There are 4 main components for KoordRuntimeProxy.
As a proxy between kubelet and containerd, KoordRuntimeProxy acts as a CRI server for kubelet(http server under dockershim scenario). It should intercept all request from kubelet, and generate protocols for talking with plugins before and after talking with backend containerd(dockerd)
PluginsManager is in charge of parsing registered plugin info from
RuntimeDispatcher is designed to manage communications with plugins.
As a proxy, KoordRuntimeProxy had better be designed as stateless, but sometimes it does NOT work. Take StartContainer hook for example, there exists only containerID in CRI StartContainerRequest, which is not enough for plugins to adapt policies since plugins may not store pod/container info(such as meta, priority) locally. So KoordRuntimeProxy should store pod/container info during RunPodSandbox/CreateContainer Stage. When StartContainer request comes, KoordRuntimeProxy can get pod/container info by containerID, and then call plugins with pod/container info.
With store, there would be pod/container info everytime KoordRuntimeProxy calls plugins, so there is no need for plugins to store pod/container info exceptionally, plugins can be designed as stateless.
Considering performance, store locates in memory and does not generate external io to disk.
How to Register Plugins
All the plugin config files should be put to
.json suffix. You can register the plugin implemented by koordlet with RuntimeProxy:
- touch /etc/runtime/hookserver.d/koordlet.json
- Copy the following content into /etc/runtime/hookserver.d/koordlet.json
There are 3 fields involved:
- remote-endpoint: endpoint KoordRuntimeProxy talking with plugin, generated by plugin.
- failure-policy: policy when calling plugin fail, Fail or Ignore, default to Ignore.
- runtime-hooks: currently 7 hook points:
hook points with prefix 'Pre' means calling plugins before transferring request to contianerd(dockerd). hook points with prefix 'Post' means calling plugins after receiving response from containerd(dockerd). plugin provider can set any hook combinations to "runtime-hooks".
Protocols between KoordRuntimeProxy and Plugins
Examples for Runtime Plugins
Installing from sources
git clone https://github.com/koordinator-sh/koordinator.git
cd koordinator; make build-koord-runtime-proxy
Installing from packages
Download latest released package from: https://github.com/koordinator-sh/koordinator/releases
Under containerd scenario, to make koord-runtime-proxy a proxy between kubelet and containerd, kubelet parameters should be altered as shown below:
kubelet <other options> --container-runtime=remote --container-runtime-endpoint=unix:///var/run/koord-runtimeproxy/runtimeproxy.sock
Under docker scenario, to make koord-runtime-proxy a proxy between kubelet and dockerd, kubelet parameters should be altered as shown below:
kubelet <other options> --docker-endpoint=unix:///var/run/koord-runtimeproxy/runtimeproxy.sock
Firstly, please make sure your runtime backend is containerd or dockerd.
Under containerd scenario, koord-runtime-proxy can be setup with command:
koord-runtime-proxy --remote-runtime-service-endpoint=<runtime sockfile path>
--remote-image-service-endpoint=<image sockfile path>
If containerd listening CRI request on default /var/run/koord-runtimeproxy/runtimeproxy.sock, koord-runtime-proxy can be setup by:
Under docker scenario, koord-runtime-proxy should be setup with the additional parameter
koord-runtime-proxy --backend-runtime-mode=Docker --remote-runtime-service-endpoint=<runtime sockfile path>