OpenKruise v1.0, Reaching New Peaks of application automation

We’re pleased to announce the release of Kubernetes 1.0, which is a CNCF Sandbox level project.

OpenKruise is an extended component suite for Kubernetes, which mainly focuses on application automations, such as deployment, upgrade, ops and availability protection. Mostly features provided by OpenKruise are built primarily based on CRD extensions. They can work in pure Kubernetes clusters without any other dependences.

Overall, OpenKruise currently provides features in these areas:

• Application workloads: Enhanced strategies of deploy and upgrade for stateless/stateful/daemon applications, such as in-place update, canary/flowing upgrade.
• Sidecar container management: supports to define sidecar container alone, which means it can inject sidecar containers, upgrade them with no effect on application containers and even hot upgrade.
• Enhanced operations: such as restart containers in-place, pre-download images on specific nodes, keep containers launch priority in a Pod, distribute one resource to multiple namespaces.
• Application availability protection: protect availability for applications that deployed in Kubernetes.

What's new?

1. InPlace Update for environments

Author: @FillZpp

OpenKruise has supported InPlace Update since very early version, mostly for workloads like CloneSet and Advanced StatefulSet. Comparing to recreate Pods during upgrade, in-place update only has to modify the fields in existing Pods.

As the picture shows above, we only modify the image field in Pod during in-place update. So that:

• Avoid additional cost of scheduling, allocating IP, allocating and mounting volumes.
• Faster image pulling, because of we can re-use most of image layers pulled by the old image and only to pull several new layers.
• When a container is in-place updating, the other containers in Pod will not be affected and remain running.

However, OpenKruise only supports to in-place update image field in Pod and has to recreate Pods if other fields need to update. All the way through, more and more users hope OpenKruise could support in-place update more fields such as env -- which is hard to implement, for it is limited by kube-apiserver.

After our unremitting efforts, OpenKruise finally support in-place update environments via Downward API since version v1.0. Take the CloneSet YAML below as an example, user has to set the configuration in annotation and write a env from it. After that, he just needs to modify the annotation value when changing the configuration. Kruise will restart all containers with env from the annotation in such Pod to enable the new configuration.

apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
  ...
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        app-config: "... the real env value ..."
    spec:
      containers:
      - name: app
        env:
        - name: APP_CONFIG
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['app-config']
  updateStrategy:
    type: InPlaceIfPossible

At the same time, we have removed the limit of imageID for in-place update, which means you can update a new image with the same imageID to the old image.

For more details please read documentation.

2. Distribute resources over multiple namespaces

Author: @veophi

For the scenario, where the namespace-scoped resources such as Secret and ConfigMap need to be distributed or synchronized to different namespaces, the native k8s currently only supports manual distribution and synchronization by users one-by-one, which is very inconvenient.

Typical examples:

• When users want to use the imagePullSecrets capability of SidecarSet, they must repeatedly create corresponding Secrets in relevant namespaces, and ensure the correctness and consistency of these Secret configurations;
• When users want to configure some common environment variables, they probably need to distribute ConfigMaps to multiple namespaces, and the subsequent modifications of these ConfigMaps might require synchronization among these namespaces.

Therefore, in the face of these scenarios that require the resource distribution and continuously synchronization across namespaces, we provide a tool, namely ResourceDistribution, to do this automatically.

Currently, ResourceDistribution supports the two kind resources --- Secret & ConfigMap.

apiVersion: apps.kruise.io/v1alpha1
kind: ResourceDistribution
metadata:
  name: sample
spec:
  resource:
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: game-demo
    data:
      ...
  targets:
    namespaceLabelSelector:
      ...
    # or includedNamespaces, excludedNamespaces

So you can see ResourceDistribution is a kind of cluster-scoped CRD, which is mainly composed of two fields: resource and targets.

• resource is a complete and correct resource structure in YAML style.
• targets indicates the target namespaces that the resource should be distributed into.

For more details please read documentation.

3. Container launch priority

Author: @Concurrensee

Containers in a same Pod in it might have dependence, which means the application in one container runs depending on another container. For example:

1. Container A has to start first. Container B can start only if A is already running.
2. Container B has to exit first. Container A can stop only if B has already exited.

Currently, the sequences of containers start and stop are controlled by Kubelet. Kubernetes used to have a KEP, which plans to add a type field for container to identify the priority of start and stop. However, it has been refused because of sig-node thought it may bring a huge change to code.

So OpenKruise provides a feature named Container Launch Priority, which helps user control the sequence of containers start in a Pod.

1. User only has to put the annotation apps.kruise.io/container-launch-priority: Ordered in a Pod, then Kruise will ensure all containers in this Pod should be started by the sequence of pod.spec.containers list.
2. If you want to customize the launch sequence, you can add KRUISE_CONTAINER_PRIORITY environment in container. The range of the value is [-2147483647, 2147483647]. The container with higher priority will be guaranteed to start before the others with lower priority.

For more details please read documentation.

4. kubectl-kruise commandline tool

Author: @hantmac

OpenKruise used to provide SDK like kruise-api and client-java for some programming languages, which can be imported into users' projects. On the other hand, some users also need to operate the workload resources with commandline in test environment.

However, the rollout, set image commands in original kubectl can only work for built-in workloads, such as Deployment and StatefulSet.

So, OpenKruise now provide a commandline tool named kubectl-kruise, which is a standard plugin of kubectl and can work for OpenKruise workload types.

# rollout undo cloneset
$ kubectl kruise rollout undo cloneset/nginx

#  rollout status advanced statefulset
$ kubectl kruise rollout status statefulsets.apps.kruise.io/sts-demo

# set image of a cloneset
$ kubectl kruise set image cloneset/nginx busybox=busybox nginx=nginx:1.9.1

For more details please read documentation.

5. Other changes

CloneSet:

• Add maxUnavailable field in scaleStrategy to support rate limiting of scaling up.
• Mark revision stable when all pods updated to it, won't wait all pods to be ready.

WorkloadSpread:

• Manage the pods that have created before WorkloadSpread.
• Optimize the update and retry logic for webhook injection.

Advanced DaemonSet:

• Support in-place update Daemon Pod.
• Support progressive annotation to control if pods creation should be limited by partition.

SidecarSet:

• Fix SidecarSet filter active pods.
• Add SourceContainerNameFrom and EnvNames fields in transferenv to make the container name flexible and the list shorter.

PodUnavailableBudget:

• Add no pub-protection annotation to skip validation for the specific Pod.
• PodUnavailableBudget controller watches workload replicas changed.

NodeImage:

• Add --nodeimage-creation-delay flag to delay NodeImage creation after Node ready.

UnitedDeployment:

• Fix pod NodeSelectorTerms length 0 when UnitedDeployment NodeSelectorTerms is nil.

Other optimization:

• kruise-daemon list and watch pods using protobuf.
• Export cache resync args and defaults to be 0 in chart value.
• Fix http checker reloading after webhook certs updated.
• Generate CRDs with original controller-tools and markers.

Get Involved

Welcome to get involved with OpenKruise by joining us in Github/Slack/DingTalk/WeChat. Have something you’d like to broadcast to our community? Share your voice at our Bi-weekly community meeting (Chinese), or through the channels below:

• Join the community on Slack (English).
• Join the community on DingTalk: Search GroupID 23330762 (Chinese).
• Join the community on WeChat: Search User openkruise and let the robot invite you (Chinese).