Cloud-Native Devops Best Practices(2) - GitOps + OpenKruise CloneSet

What is GitOps？

GitOps is an approach to continuous delivery. Its core idea is to store the declarative infrastructure and applications of an application in a Git repository.

With Git at the core of the delivery pipeline, every developer can submit Pull Requests and use Git to accelerate and simplify application deployment and maintenance tasks for Kubernetes. By using a simple tool like Git, developers can more efficiently focus on creating new features rather than operations-related tasks (e.g., application installation, configuration, migration, etc.).

GitOps + OpenKruise CloneSet Practice

Requirements

• Install Kubernetes Cluster, Since v1.0.0 (alpha/beta), OpenKruise requires Kubernetes version >= 1.16.
• Install Tekton, Reference Official Documents, Tekton is a Google open source Kubernetes native framework for creating continuous integration and continuous deployment/delivery (CI/CD) systems.
• Install Argo-cd, Reference Official Documents, Argo-cd is a declarative GitOps continuous delivery tool for Kubernetes.

Install OpenKruise（Enable: TemplateNoDefaults）

Openkruise installed by default will inject the default value of pod / PVC template, which will conflict with the sync judgment logic of Argo CD. Therefore, when installing openkruise, you need to open gates TemplateNoDefaults, as follows:

# Firstly add openkruise charts repository if you haven't do this.
$ helm repo add openkruise https://openkruise.github.io/charts/

# [Optional]
$ helm repo update

# Install the latest version.
$ helm install kruise openkruise/kruise --set featureGates="TemplateNoDefaults=true"

# Those that have been installed need to be upgraded
$ helm upgrade kruise openkruise/kruise --set featureGates="TemplateNoDefaults=true"

CloneSet Deploy Stateless Application

CloneSet is the ability provided by OpenKruise to efficiently manage stateless applications, it is similar to the official workload: Deployment, but offers many enhancements such as InPlace Update, Batch Release, etc. Please refer to the documentation CloneSet. This article provides a helloworld http service demo, It contains Helm Charts, and the cloneSet configuration is shown below:

apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
  name: helloworld-server
  labels:
    app: helloworld-server
spec:
  updateStrategy:
    # CloneSet will try to in-place update Pod instead of recreating them if possible
    type: InPlaceIfPossible
    # Batch release, currently updating only one Pod
    partition: 1
  replicas: 2
  selector:
    matchLabels:
      app: helloworld-server
  template:
    metadata:
      labels:
        app: helloworld-server
    spec:
      containers:
      - name: helloworld
        image: "openkruise/kruise:hello_world-d92ae174b"

Argo-cd CloneSet Health Check

Configure CloneSet Argo-cd Custom CRD Health Checks, With this configuration argo-cd is able to perform a healthy check of the CloneSet, such as whether the CloneSet is published and whether the Pods are ready, as follows：

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.apps.kruise.io_CloneSet: |
    hs = {}
    -- if paused
    if obj.spec.updateStrategy.paused then
      hs.status = "Suspended"
      hs.message = "CloneSet is Suspended"
      return hs
    end

    -- check cloneSet status
    if obj.status ~= nil then
      if obj.status.observedGeneration < obj.metadata.generation then
        hs.status = "Progressing"
        hs.message = "Waiting for rollout to finish: observed cloneSet generation less then desired generation"
        return hs
      end

      if obj.status.updatedReplicas < obj.spec.replicas then
        hs.status = "Progressing"
        hs.message = "Waiting for rollout to finish: replicas hasn't finished updating..."
        return hs
      end

      if obj.status.updatedReadyReplicas < obj.status.updatedReplicas then
        hs.status = "Progressing"
        hs.message = "Waiting for rollout to finish: replicas hasn't finished updating..."
        return hs
      end

      hs.status = "Healthy"
      return hs
    end

    -- if status == nil
    hs.status = "Progressing"
    hs.message = "Waiting for cloneSet"
    return hs

OpenKruise internal CRD resources in addition to CloneSet, others such as: Advanced StatefulSet, SidecarSet, etc. can be similar to the above way to achieve Custom Resource Health.

Tekton Pipeline + Argo-cd

Argo-CD together with Tekton Pipeline is a popular DevOps practice and integrate well with CI process. Such practice requires storing the Argo-cd admin secret in K8S Secret CRD (method of obtaining secret), which in turn can be used in Tekton Pipeline, as follows:

apiVersion: v1
data:
  # argo-cd admin secret
  username: xxxxx
  password: xxxxx
  server: xxxxx
kind: Secret
metadata:
  name: argosecret
---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  labels:
    app: helloworld
  name: helloworld-argocd
spec:
  params:
  - name: gitrepositoryurl
    type: string
  - name: branch
    type: string
  - name: short_sha
    type: string
  - name: docker_repo
    type: string
  - name: app_name
    type: string
  - name: app_ns
    type: string
  - name: k8s_server
    type: string
  steps:
  - name: argocd-deploy
    image: argoproj/argocd:latest
    command:
    - sh
    args:
    - '-ce'
    - >
      set -e

      echo "upgrade app $(params.app_name)"; username=`cat /var/secret/username`; password=`cat /var/secret/password`; server=`cat /var/secret/server`;

      argocd login ${server} --insecure --username ${username} --password ${password}

      argocd app create $(params.app_name) --upsert --repo $(params.gitrepositoryurl) --path $(params.app_name)/charts --dest-namespace $(params.app_ns) --dest-server $(params.k8s_server) --revision $(params.branch) --helm-set image.repository=$(params.docker_repo) --helm-set image.tag=$(params.branch)-$(params.short_sha) --helm-set installation.namespace=$(params.app_ns)

      argocd app list; argocd app sync $(params.app_name)

      argocd app wait $(params.app_name) --health
    volumeMounts:
    - name: argocd-secret
      mountPath: "/var/secret"
  volumes:
  - name: argocd-secret
    secret:
      secretName: argosecret
---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: helloworld-pipeline
spec:
  params:
  - name: gitrepositoryurl
    type: string
  - name: branch
    type: string
  - name: short_sha
    type: string
  - name: docker_repo
    type: string
  - name: app_name
    type: string
  - name: app_ns
    type: string
  - name: k8s_server
    type: string
  # Here you can connect with CI process to realize CI/CD Pipeline
  tasks:
  - name: helloworld-argocd
    taskRef:
      name: helloworld-argocd
    params:
    - name: gitrepositoryurl
      value: $(params.gitrepositoryurl)
    - name: short_sha
      value: $(params.short_sha)
    - name: branch
      value: $(params.branch)
    - name: docker_repo
      value: $(params.docker_repo)
    - name: app_name
      value: $(params.app_name)
    - name: app_ns
      value: $(params.app_ns)
    - name: k8s_server
      value: $(params.k8s_server)

Run Tekton Pipeline

Configure PipelineRun CRD, and kubectl apply -f in k8s cluster to run Pipeline, as follows:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: helloworld-pipeline-run-1
spec:
  pipelineRef:
    name: helloworld-pipeline
  params:
  - name: gitrepositoryurl
    value: https://github.com/zmberg/samples.git
  - name: branch
    value: hello_world
  - name: short_sha
    value: d92ae174b
  - name: docker_repo
    value: zhaomingshan/kruise
  - name: app_name
    value: helloworld
  - name: app_ns
    value: helloworld
  - name: k8s_server
    value: https://kubernetes.default.svc

The results can be viewed via the argo-cd cli, as follows:

Summary

OpenKruise is more of a Kubernetes level extended capability, such as in-place upgrade, preImageDownload, etc. So many community users using OpenKruise in production environments have some additional costs, need to integrate or self-research container PaaS. The main purpose of this article is to combine some of the best Paas solutions in the community with OpenKruise, so that as many people as possible get to enjoy the dividends of cloud-native at a lower cost.

Argo-cd is a very good product for the community, and it integrate well with OpenKruise's CRD resources, we hope that this article can intrigue more ideas from the community about "How to use OpenKruise easily". We will try to integrate OpenKruise with other CI/CD pipelines later to better implement DevOps practice.