Container Launch Priority

FEATURE STATE: Kruise v1.0.0

Container Launch Priority provides a way to help users control the sequence of containers start in a Pod.

Usually the sequences of containers start and stop are controlled by Kubelet. Kubernetes used to have a KEP, which plans to add a type field for container to identify the priority of start and stop. However, it has been refused because of sig-node thought it may bring a huge change to code.

Note that the feature works for Pod, no matter what kind of owner it belongs to, which means Deployment, CloneSet or any other Workloads are all supported.

Note: If your kubernetes version >= 1.28, it is recommended that you use k8s native Sidecar Container to solve the above problem.

Usage

Start by containers ordinal

It only requires you to add an annotation in Pod:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    apps.kruise.io/container-launch-priority: Ordered
spec:
  containers:
  - name: sidecar
    # ...
  - name: main
    # ...

Then Kruise will ensure the former container (sidecar) to be started before the later one (main).

Start by configurable sequence

You should set the priority env KRUISE_CONTAINER_PRIORITY in container:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: main
    # ...
  - name: sidecar
    env:
    - name: KRUISE_CONTAINER_PRIORITY
      value: "1"
    # ...

1. The range of the value is [-2147483647, 2147483647]. Defaults to 0 if no such env exists.
2. The container with higher priority will be guaranteed to start before the others with lower priority.
3. The containers with same priority have no limit to their start sequence.

Advanced Example: Multiple Containers with Mixed Priorities

Here is an example of a Pod with three containers, each assigned a different launch priority:

apiVersion: v1
kind: Pod
metadata:
  name: multi-priority-pod
  annotations:
    apps.kruise.io/container-launch-priority: Ordered
spec:
  containers:
    - name: init
      image: busybox
      command: ["sh", "-c", "echo Init"]
      env:
        - name: KRUISE_CONTAINER_PRIORITY
          value: "10"
    - name: main
      image: nginx
      env:
        - name: KRUISE_CONTAINER_PRIORITY
          value: "5"
    - name: sidecar
      image: busybox
      command: ["sh", "-c", "echo Sidecar"]
      env:
        - name: KRUISE_CONTAINER_PRIORITY
          value: "1"

Expected behavior:

• The init container (priority 10) will start first.
• The main container (priority 5) will start after init is running and ready.
• The sidecar container (priority 1) will start last, after both init and main are running and ready.

This allows you to control complex startup dependencies between containers in a Pod.

Requirement

ContainerLaunchPriority requires PodWebhook feature-gate to be enabled, which is the default state.

Implementation Details

Kruise webhook will admit for all pod creation. When webhook finds a pod has apps.kruise.io/container-launch-priority annotation or KRUISE_CONTAINER_PRIORITY in env, it will inject KRUISE_CONTAINER_BARRIER env into containers.

The value of KRUISE_CONTAINER_BARRIER is from a ConfigMap named {pod-name}-barrier, and the key is related to the priority of each container.

For example:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: main
    # ...
    env:
    - name: KRUISE_CONTAINER_BARRIER
      valueFrom:
        configMapKeyRef:
          name: {pod-name}-barrier
          key: "p_0"
  - name: sidecar
    env:
    - name: KRUISE_CONTAINER_PRIORITY
      value: "1"
    - name: KRUISE_CONTAINER_BARRIER
      valueFrom:
        configMapKeyRef:
          name: {pod-name}-barrier
          key: "p_1"
    # ...

Kruise controller will create an empty ConfigMap for this pod, then add the keys into ConfigMap according to the priorities and containerStatuses of pod.

As the example before, controller will firstly add p_1 key into ConfigMap, waiting for sidecar container running and ready, and finally add p_0 into ConfigMap to let Kubelet start main container.

Besides, you may see CreateContainerConfigError state when you use kubectl get during pod is starting with priority. It is because Kubelet can't find some keys at that moment, and will be fine after all container in Pod started.