Skip to main content

Installation

Overviewโ€‹

Kruise Agents is installed through two Helm charts, which deploy the Sandbox Controller and the Sandbox Manager โ€” together with the Sandbox Gateway bundled in the Manager chart โ€” forming a complete Sandbox runtime environment.

  • Sandbox Controller (agents-sandbox-controller chart) is the control plane that manages all Sandbox CRD resources. The chart contains:
    • Eight agents.kruise.io CRDs: Sandbox, SandboxSet, SandboxClaim, SandboxTemplate, SandboxUpdateOps, Checkpoint, Commit, and PoolAutoscaler.
    • The sandbox-controller Deployment that reconciles these resources: sandbox lifecycle management, warm pool maintenance, claiming, in-place updates, Checkpoint/Commit, and pool autoscaling.
    • Mutating and validating webhooks, RBAC, the ServiceAccount, and the metrics Service.
    • The sandbox-injection-config ConfigMap, defining the in-pod injection templates for the agent-runtime sidecar and the per-sandbox traffic-proxy.
    • Optional TLS resources when enableTLS=true: the shared root CA, the runtime client and server certificates, and the trust-manager CA bundle.
  • Sandbox Manager (agents-sandbox-manager chart) is the sandbox data plane component and also provides the E2B API adaptation service. The chart contains:
    • The sandbox-manager Deployment, along with the chart's Service, Secret, and Ingress resources.
    • The Sandbox Gateway Deployment: an Envoy + Golang Filter data plane responsible for traffic routing, load balancing, and circuit-breaking protection, scaling independently from the Manager.
    • An optional ServiceMonitor for Manager and Gateway metrics when prometheus.enabled=true.
    • Optional TLS resources when enableTLS=true: the ingress server certificate, the runtime client certificates, and the managerโ†”gateway peer certificates.
    • An optional embedded Agentio stack for sandbox ingress/egress control (agentio.enabled, disabled by default): the agentiod control plane, the EPE traffic extension, and the egress gateway.
    • The TrafficPolicy, GlobalTrafficPolicy, SecurityProfile, and GlobalSecurityProfile CRDs describing sandbox ingress/egress policies, installed with the chart.

Version Compatibilityโ€‹

Sandbox Component VersionKubernetes VersionE2B Version
0.6.0>= 1.28>= 2.8.0

Note:

  • The agent-runtime sidecar injection requires Kubernetes >= 1.29 (native sidecar containers), see Agent Runtime Injection.
  • enableTLS=true requires cert-manager and, for the CA bundle, trust-manager to be installed in the cluster.

Prerequisitesโ€‹

  1. Kubernetes cluster version >= 1.28 (>= 1.29 if you use agent-runtime sidecar injection)
  2. Helm v3.5+ installed
  3. Namespace created manually (see installation steps below)
  4. cert-manager and trust-manager installed in the cluster if you enable TLS (enableTLS=true)

Install via Helmโ€‹

1. Add OpenKruise Charts Repositoryโ€‹

## Add openkruise charts repository
helm repo add openkruise https://openkruise.github.io/charts/

## Update repository (if openkruise charts repository was previously installed)
helm repo update

2. Install Sandbox Controllerโ€‹

Manually Create Namespace

kubectl create ns sandbox-system

Installation Order: Sandbox Controller must be installed before Sandbox Manager, as it provides the CRD resources required by Sandbox Manager.

helm install agents-sandbox-controller openkruise/agents-sandbox-controller \
-n sandbox-system \
--version 0.6.0

โš ๏ธ Server-side apply is not supported: The sandbox-controller manages the template annotation on the mutating and validating webhook configurations at runtime under its own field manager (manager). Applying the chart with Kubernetes server-side apply competes for that same annotation and fails with a field-ownership conflict such as Apply failed with 1 conflict: conflict with "manager". Do not pass --server-side to helm install / helm upgrade (Helm 3 uses client-side apply by default). If you use Helm 4 or another tool that applies with SSA by default, turn SSA off.

3. Install Sandbox Managerโ€‹

Required Parameters: The following parameters must be explicitly specified during installation:

  • e2b.domain: E2B protocol domain (the Ingress is created for api.<domain>, *.<domain>, and <domain>)
  • e2b.adminApiKey: E2B admin API Key for authentication
  • ingress.className: Ingress controller class name (e.g., nginx, alb, etc., depending on your cluster's Ingress implementation)
helm install agents-sandbox-manager openkruise/agents-sandbox-manager \
-n sandbox-system \
--version 0.6.0 \
--set e2b.domain=<your-domain> \
--set e2b.adminApiKey=<your-api-key> \
--set ingress.className=<your-ingress-class>

Note: The Sandbox Manager chart also deploys Sandbox Gateway, no additional installation required. The Ingress routes control-plane traffic (api.<domain>) to the Sandbox Manager Service (port 8080) and data-plane traffic (<domain> and *.<domain>) to the Sandbox Gateway Service (port 7788).


Using China Mirror Registryโ€‹

Due to network restrictions, users in China may not be able to pull images directly from Docker Hub. It is recommended to use China mirrors provided by Alibaba Cloud Container Registry.

China Mirror Addressesโ€‹

ComponentImage AddressVersion
Sandbox Controlleropenkruise-registry.cn-shanghai.cr.aliyuncs.com/openkruise/agent-sandbox-controllerv0.6.0
Sandbox Manageropenkruise-registry.cn-shanghai.cr.aliyuncs.com/openkruise/sandbox-managerv0.6.0
Sandbox Gatewayopenkruise-registry.cn-shanghai.cr.aliyuncs.com/openkruise/sandbox-gatewayv0.6.0

Install with China Mirrorsโ€‹

All images in both charts resolve through the chart-wide image.registry, so a single parameter switches every image (Controller, Manager, Gateway, plus auxiliary images such as agent-runtime and the Commit job) to the China mirror.

Install Sandbox Controller (using China mirror)

helm install agents-sandbox-controller openkruise/agents-sandbox-controller \
-n sandbox-system \
--version 0.6.0 \
--set image.registry=openkruise-registry.cn-shanghai.cr.aliyuncs.com

Install Sandbox Manager (using China mirror)

helm install agents-sandbox-manager openkruise/agents-sandbox-manager \
-n sandbox-system \
--version 0.6.0 \
--set e2b.domain=<your-domain> \
--set e2b.adminApiKey=<your-api-key> \
--set ingress.className=<your-ingress-class> \
--set image.registry=openkruise-registry.cn-shanghai.cr.aliyuncs.com

Note:

  • Agentio images resolve their registry through agentio.global.registry first and fall back to image.registry when both are unset.
  • An image whose repository already starts with a host (for example registry.cn-beijing.aliyuncs.com/acs/busybox) bypasses the chart-wide registry, which is handy when the mirror does not carry an auxiliary image.
  • To enable the Gateway init container with a mirror-hosted busybox, add the following:
    --set gateway.initContainer.enabled=true \
    --set gateway.initContainer.image.repository=registry.cn-beijing.aliyuncs.com/acs/busybox \
    --set gateway.initContainer.image.tag=1.36.1

Upgrade via Helmโ€‹

Upgrade Sandbox Controllerโ€‹

helm upgrade agents-sandbox-controller openkruise/agents-sandbox-controller \
-n sandbox-system \
--version 0.6.0 \
--server-side=false

Upgrade Sandbox Managerโ€‹

helm upgrade agents-sandbox-manager openkruise/agents-sandbox-manager \
-n sandbox-system \
--version 0.6.0

Note:

  1. Upgrade order: Upgrade Sandbox Controller first, then Sandbox Manager to ensure CRD compatibility.
  2. Before upgrading, you must read the Change Log to ensure you understand the incompatible changes in the new version.
  3. If you want to reset parameters used in previous versions or configure new parameters, it is recommended to add --reset-values to the helm upgrade command.
  4. You must pass --server-side=false. The sandbox-controller manages the webhook template annotation at runtime under its own field manager (manager), so a server-side apply competes for the same annotation and fails with a field-ownership conflict (Apply failed with 1 conflict: conflict with "manager" ...). Helm 3 applies client-side by default, but Helm 4 and some CI tooling default to server-side apply, so set the flag explicitly to force client-side apply.

Manually Update CRDs (Required)โ€‹

Helm upgrade will not automatically update CRD definitions under the crds/ directory, so you must manually apply the new CRDs before running helm upgrade, otherwise new features will not work properly. The 0.6.0 release adds the Commit and PoolAutoscaler CRDs and updates the schemas of existing CRDs.

# Extract CRDs from chart package and apply (online installation example)
helm pull openkruise/agents-sandbox-controller --version 0.6.0 --untar
kubectl apply -f agents-sandbox-controller/crds/
rm -rf agents-sandbox-controller

Key CRD changes in 0.6.0 include:

  • New Commit CRD (commits.agents.kruise.io): Commit sandbox filesystem state to a container image through a job
  • New PoolAutoscaler CRD (poolautoscalers.agents.kruise.io): Capacity-based and cron-driven SandboxSet pool autoscaling
  • SandboxSet enhancements: PauseStrategy (Stop / Snapshot / CloudDisk) configuration

Note: The traffic and security CRDs shipped by the Sandbox Manager chart (trafficpolicies.agents.kruise.io, globaltrafficpolicies.agents.kruise.io, securityprofiles.agents.kruise.io, globalsecurityprofiles.agents.kruise.io) are rendered as templates rather than crds/ files, so they are updated automatically during helm upgrade and need no manual step.


Manual Chart Downloadโ€‹

If you cannot connect to https://openkruise.github.io/charts/ in your production environment, you can manually download the chart package from GitHub Releases and then install or upgrade it to your cluster.

helm install/upgrade agents-sandbox-controller /PATH/TO/CONTROLLER/CHART -n sandbox-system
helm install/upgrade agents-sandbox-manager /PATH/TO/MANAGER/CHART -n sandbox-system

Optionsโ€‹

Sandbox Controller Installation Parametersโ€‹

The following tables list the configurable parameters of the Sandbox Controller chart and their default values.

Common Parametersโ€‹

ParameterDescriptionDefault
replicaCountNumber of sandbox-controller replicas2
image.registryRegistry prepended to every image in this chartdocker.io
image.repositorysandbox-controller image repositoryopenkruise/agent-sandbox-controller
image.tagsandbox-controller image tagv0.6.0
image.pullPolicyController image pull policyIfNotPresent
imagePullSecretsImage pull secrets list[]
namespace.nameNamespace name for deploymentsandbox-system
resources.limits.cpuController CPU resource limit2
resources.limits.memoryController memory resource limit4Gi
resources.requests.cpuController CPU resource request2
resources.requests.memoryController memory resource request4Gi
webhook.portWebhook service port9443
metrics.portMetrics service port (HTTPS with authn/authz delegation to kube-apiserver)8443
healthProbe.portHealth probe port8081
controller.featureGatesComma-separated --feature-gates key=value pairs (e.g. Foo=true,Bar=false); empty sets no flag""

Note: To pass multiple feature gates with --set, escape every comma inside the value as \, and wrap the whole argument in single quotes so the shell preserves the backslashes:

--set 'controller.featureGates=Commit=true\,KruiseIntegration=true'

--set splits its argument on unescaped commas, so without the escape Helm reads KruiseIntegration=true as a separate top-level assignment and the gate is silently dropped from controller.featureGates.

Advanced Parametersโ€‹

All remaining parameters are optional. Sensible defaults apply and most installs do not need to change them.

ParameterDescriptionDefault
controller.workers.sandboxWorkersConcurrent workers for the Sandbox reconciler200
controller.workers.sandboxsetWorkersConcurrent workers for the SandboxSet reconciler10
controller.workers.sandboxclaimWorkersConcurrent workers for the SandboxClaim reconciler200
controller.workers.sandboxupdateopsWorkersConcurrent workers for the SandboxUpdateOps reconciler5
controller.workers.poolautoscalerWorkersConcurrent workers for the PoolAutoscaler reconciler3
controller.workers.commitWorkersConcurrent workers for the Commit reconciler5
controller.clientQPSKubernetes API client QPS rate limit30000
controller.clientBurstKubernetes API client burst limit60000
metrics.rbac.createCreate a ClusterRole granting get on the /metrics nonResourceURL; bind it to your Prometheus / ARMS scraper ServiceAccounttrue
nameOverrideOverride Chart name""
fullnameOverrideOverride full name""
serviceAccount.createWhether to create ServiceAccounttrue
serviceAccount.automountWhether to automount ServiceAccount Tokentrue
serviceAccount.annotationsServiceAccount annotations{}
serviceAccount.nameServiceAccount name to use""
rbac.createWhether to create RBAC resourcestrue
podAnnotationsPod annotations{}
podLabelsPod labels{}
podSecurityContextPod security context{runAsNonRoot: true, seccompProfile: {type: RuntimeDefault}}
securityContextContainer security context{allowPrivilegeEscalation: false, capabilities: {drop: [ALL], add: [NET_BIND_SERVICE]}, readOnlyRootFilesystem: true}
nodeSelectorNode selector for Pod scheduling{}
tolerationsTolerations for Pod scheduling[]
affinityAffinity for Pod scheduling{}
agentRuntime.image.repositoryInjected agent-runtime sidecar image repositoryopenkruise/agent-runtime
agentRuntime.image.tagInjected agent-runtime sidecar image tagv0.3.0
agentRuntime.image.pullPolicyInjected agent-runtime sidecar image pull policyIfNotPresent
commitJob.image.repositoryImage repository of the Commit job podsopenkruise/commit-job
commitJob.image.tagImage tag of the Commit job podsv0.6.0
enableTLSMaster switch for cert-manager / trust-manager based TLS provisioning; when false nothing under templates/tls/ renders and the controller keeps plaintext runtime behaviorfalse
tls.createCACreate the shared root CA (selfSigned Issuer โ†’ CA Certificate โ†’ CA Issuer). The controller chart owns the CA; the sandbox-manager chart sets this to false and references the Issuer by nametrue
tls.selfSignedIssuerNameSelf-signed bootstrap Issuer namesandbox-selfsigned-issuer
tls.caCertificateNameCA Certificate resource namesandbox-ca
tls.caSecretNameSecret holding the CA key pair (tls.crt/tls.key); also the trust-manager Bundle source and the CA Issuer's signing materialsandbox-ca-key-pair
tls.signingIssuerNameCA Issuer name used to sign leaf certificatessandbox-signing-issuer
tls.caCommonNameCA certificate common namesandbox-ca
tls.caOrganizationCA certificate organizationopenkruise
tls.caDurationCA certificate lifetime87600h (10 years)
tls.certDurationLeaf certificate lifetime2160h (90 days)
tls.certRenewBeforeLeaf certificate renewal window360h (15 days)
tls.runtime.enabledIssue the runtime client/server certificates and activate the client-side TLS paths (--runtime-client-cert-dir plus the agent-runtime sidecar TLS env). Fails against an agent-runtime that does not serve TLS; requires enableTLSfalse
tls.runtimeClientCertNameController โ†’ agent-runtime client Certificate resource namesandbox-controller-runtime-client
tls.runtimeClientCertSecretNameController client certificate Secret namesandbox-controller-runtime-client-cert
tls.runtimeClientCommonNameController client certificate common namesystem:sandbox-controller-manager
tls.runtimeClientCertDirMount point for --runtime-client-cert-dir; the volume remaps tls.crt/tls.key/ca.crt to client.crt/client.key/ca.crt/etc/agent-runtime-client/certs
tls.agentRuntimeServerCertNameagent-runtime server Certificate resource namesandbox-agent-runtime-server
tls.agentRuntimeServerCertSecretNameagent-runtime server certificate Secret namesandbox-agent-runtime-server-certs
tls.agentRuntimeServerSANSAN the gateway/controller/manager use to reach the runtimeagentruntime.sandbox.agents.kruise.io
tls.bundle.enabledCreate the trust-manager Bundle distributing the CA as a ca.crt ConfigMaptrue
tls.bundle.nametrust-manager Bundle namesandbox-ca-bundle
tls.bundle.configMapKeyKey holding the CA in the distributed ConfigMapca.crt
tls.bundle.namespaceSelectorNamespace selector limiting where the CA ConfigMap is written; empty selects every namespace{}
agentio.trafficProxy.controlPlaneNamespaceNamespace containing the Agentio control planesandbox-system
agentio.trafficProxy.controlPlaneServiceAgentio control-plane Service nameagentiod
agentio.trafficProxy.xdsAddressExplicit XDS address; generated from service and namespace when empty""
agentio.trafficProxy.caAddressExplicit CA address; generated from service and namespace when empty""
agentio.trafficProxy.caCertConfigMapCA ConfigMap mounted in injected workload namespacesagentio-ca-root-cert
agentio.trafficProxy.clusterIdCluster identifier reported to the control planeKubernetes
agentio.trafficProxy.clusterDomainKubernetes service DNS domaincluster.local
agentio.trafficProxy.tokenAudienceProjected workload token audienceagentio-ca
agentio.trafficProxy.includeInboundPortsInbound ports captured by the traffic proxy*
agentio.trafficProxy.includeOutboundIPRangesOutbound IP ranges captured by the traffic proxy*
agentio.trafficProxy.includeOutboundPortsOutbound ports captured by the traffic proxy""
agentio.trafficProxy.excludeInboundPortsInbound ports excluded from capture""
agentio.trafficProxy.excludeOutboundIPRangesOutbound IP ranges excluded from capture""
agentio.trafficProxy.excludeOutboundPortsOutbound ports excluded from capture""
agentio.trafficProxy.enableFirewallRulesEnable traffic-proxy firewall rulestrue
agentio.trafficProxy.firewallBackendFirewall backend selectionauto
agentio.trafficProxy.imagePullPolicyTraffic-proxy image pull policyIfNotPresent
agentio.trafficProxy.healthProbeRewriteRewrite health probes for injected traffic proxiestrue
agentio.trafficProxy.dnsCaptureEnable DNS capturetrue
agentio.trafficProxy.image.registryInjected ztunnel image registry; empty inherits image.registry""
agentio.trafficProxy.image.repositoryInjected ztunnel image repositoryopenkruise/ztunnel
agentio.trafficProxy.image.tagInjected ztunnel image tag0.2.0
agentio.trafficProxy.image.digestOptional digest override; takes precedence over tag""
agentio.trafficProxy.initImage.registryInjected iptables init image registry; empty inherits image.registry""
agentio.trafficProxy.initImage.repositoryInjected iptables init image repositoryopenkruise/proxy-init
agentio.trafficProxy.initImage.tagInjected iptables init image tag0.2.0
agentio.trafficProxy.initImage.digestOptional digest override; takes precedence over tag""
agentio.trafficProxy.resourcesInjected ztunnel resourcesrequests: 100m/64Mi, limits: 200m/128Mi
agentio.trafficProxy.initResourcesInjected iptables init resourcesrequests: 100m/128Mi, limits: 1/1Gi

Sandbox Manager Installation Parametersโ€‹

The following tables list the configurable parameters of the Sandbox Manager chart and their default values.

Common Parametersโ€‹

ParameterDescriptionDefault
replicaCountNumber of sandbox-manager replicas2
image.registryRegistry prepended to every image in this chartdocker.io
imagePullSecretsImage pull secrets list{}
controller.repositorysandbox-manager controller image repositoryopenkruise/sandbox-manager
controller.tagsandbox-manager controller image tagv0.6.0
controller.pullPolicyController container image pull policyIfNotPresent
controller.resources.cpuController container CPU resource2
controller.resources.memoryController container memory resource4Gi
e2b.domainE2B protocol domain (required)"your.domain.com"
e2b.enableAuthWhether to enable E2B authenticationtrue
e2b.adminApiKeyE2B admin API key (required)""
ingress.classNameIngress class name (required)""
ingress.annotationsIngress annotations{}
prometheus.enabledCreate a ServiceMonitor for manager and gateway metricsfalse
gateway.replicaCountNumber of sandbox-gateway replicas2
gateway.image.repositorysandbox-gateway image repositoryopenkruise/sandbox-gateway
gateway.image.tagsandbox-gateway image tagv0.6.0
gateway.image.pullPolicysandbox-gateway image pull policyIfNotPresent
gateway.resources.cpusandbox-gateway container CPU resources2
gateway.resources.memorysandbox-gateway container memory resources4Gi

Advanced Parametersโ€‹

All remaining parameters are optional. Sensible defaults apply and most installs do not need to change them.

ParameterDescriptionDefault
controller.logLevelController log level5
controller.infraSandbox manager infrastructure typesandbox-cr
controller.hostNetworkWhether controller uses host networkfalse
controller.maxClaimWorkersMaximum claim worker threads100
controller.maxCreateQPSMaximum QPS for creating sandboxes200
controller.extProcMaxConcurrencyMaximum concurrency for external processors10000
controller.enableShortSandboxIdEnable short, human-friendly sandbox identifierstrue
controller.shortSandboxIdPrefixOptional prefix for short sandbox ids; only applied when non-empty""
e2b.extraDomainsExtra domains added to the Ingress host list alongside e2b.domain[]
e2b.maxTimeoutE2B maximum timeout (seconds)2592000
e2b.keyStorage.modeWhere E2B API keys are stored: secret (the e2b-key-store Secret) or mysqlsecret
e2b.keyStorage.mysql.dsnMySQL DSN; required (non-empty) when keyStorage.mode=mysql, validated at startup""
e2b.keyStorage.mysql.hashPepperMySQL key-hash pepper; required (non-empty) when keyStorage.mode=mysql, validated at startup""
quota.enabledEnable Redis-backed quota accountingfalse
quota.redis.addrRedis address for quota; only passed to the manager when enabled""
quota.redis.dbRedis database number for quota0
quota.redis.usernameRedis username (rendered into the manager Secret)""
quota.redis.passwordRedis password (rendered into the manager Secret)""
service.typesandbox-manager service typeClusterIP
ingress.dataplaneServiceDataplane backend service name for Ingresssandbox-gateway
ingress.certSecretNameIngress TLS certificate Secret namesandbox-manager-tls
nameOverrideOverride Chart name""
fullnameOverrideOverride full name""
serviceAccount.automountWhether to automount ServiceAccount Tokentrue
serviceAccount.annotationsServiceAccount annotations{}
serviceAccount.nameServiceAccount name to use""
podAnnotationsPod annotations{}
podLabelsPod labels{}
podSecurityContextPod security context{fsGroup: 2000, seccompProfile: {type: RuntimeDefault}}
securityContextContainer security context{capabilities: {drop: [ALL], add: [NET_BIND_SERVICE]}, readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, runAsNonRoot: true, runAsUser: 65532}
nodeSelectorNode selector for Pod scheduling{}
tolerationsTolerations for Pod scheduling[]
affinityAffinity for Pod schedulingPreferred Pod anti-affinity (preferredDuringSchedulingIgnoredDuringExecution), spread across hostnames
gracefulShutdown.preStopSleepSecondsSleep in the preStop hook before termination; also raises terminationGracePeriodSeconds when > 00
gateway.imagePullSecretsGateway image pull secrets list[]
gateway.nameOverrideOverride gateway name""
gateway.serviceAccount.annotationsGateway ServiceAccount annotations{}
gateway.serviceAccount.nameGateway ServiceAccount name to use""
gateway.podAnnotationsGateway pod annotations{}
gateway.podSecurityContextGateway pod security context{}
gateway.securityContextGateway container security context{}
gateway.initContainer.enabledWhether to enable gateway initContainerfalse
gateway.initContainer.image.repositoryinitContainer image repositorybusybox
gateway.initContainer.image.taginitContainer image tag1.36.1
gateway.initContainer.image.pullPolicyinitContainer image pull policyIfNotPresent
gateway.initContainer.securityContextinitContainer security context{capabilities: {drop: [ALL], add: [SYS_ADMIN]}}
gateway.service.typesandbox-gateway service typeClusterIP
gateway.service.portsandbox-gateway service port7788
gateway.service.targetPortsandbox-gateway service target port7788
gateway.service.annotationsGateway service annotations{}
gateway.service.labelsGateway service labels{}
gateway.nodeSelectorNode selector for gateway pod scheduling{}
gateway.tolerationsTolerations for gateway pod scheduling[]
gateway.affinityAffinity for gateway pod scheduling{}
gateway.livenessProbeGateway liveness probe (TCP on port 7788)initialDelay: 10s, period: 10s, timeout: 5s, failureThreshold: 3
gateway.readinessProbeGateway readiness probe (TCP on port 7788)initialDelay: 5s, period: 5s, timeout: 3s, failureThreshold: 3
gateway.podAntiAffinityGateway pod anti-affinitysoft, weight: 100, topologyKey: kubernetes.io/hostname
gateway.envoy.admin.addressEnvoy admin interface address127.0.0.1
gateway.envoy.admin.portEnvoy admin interface port9901
gateway.envoy.prometheus.enabledExpose Envoy Prometheus metrics externally (proxies admin /stats/prometheus)false
gateway.envoy.prometheus.addressEnvoy Prometheus listener address0.0.0.0
gateway.envoy.prometheus.portEnvoy Prometheus listener port9902
gateway.envoy.prometheus.pathEnvoy Prometheus metrics path/metrics
gateway.envoy.listener.addressEnvoy listener address0.0.0.0
gateway.envoy.listener.portEnvoy listener port7788
gateway.envoy.logLevelEnvoy log levelwarn
gateway.envoy.concurrencyEnvoy worker thread concurrency; empty falls back to gateway.resources.cpu""
gateway.envoy.drainTimeSecondsEnvoy drain time on shutdown30
gateway.envoy.streamIdleTimeoutEnvoy stream idle timeout600s
gateway.envoy.connectTimeoutEnvoy upstream connect timeout5s
gateway.envoy.useDownstreamProtocolConfigApply the downstream protocol config to upstream connectionsfalse
gateway.envoy.perConnectionBufferLimitBytesPer-connection buffer limit1048576
gateway.envoy.circuitBreakersEnvoy circuit-breaker thresholdsenabled: true, maxConnections: 80000, maxPendingRequests: 32768, maxRequests: 80000, maxRetries: 5
gateway.envoy.tcpKeepaliveEnvoy upstream TCP keepaliveenabled: false, keepaliveProbes: 3, keepaliveTime: 60, keepaliveInterval: 10
gateway.envoy.golangFilterGateway golang filter librarylibraryId/pluginName: sandbox-gateway, libraryPath: /etc/envoy/sandbox-gateway.so
gateway.envoy.pluginConfig.hostHeaderNameHeader carrying the upstream hostHost
gateway.envoy.pluginConfig.sandboxHeaderNameHeader carrying the sandbox ide2b-sandbox-id
gateway.envoy.pluginConfig.sandboxPortHeaderHeader carrying the sandbox porte2b-sandbox-port
gateway.envoy.pluginConfig.defaultPortPort used when no sandbox port header is present"49983"
gateway.envoy.pluginConfig.enableAuthEnforce traffic access-token auth in the gateway plugintrue
gateway.envoy.pluginConfig.trafficAccessTokenHeaderHeader carrying the traffic access tokene2b-traffic-access-token
gateway.envoy.pluginConfig.enableJwtAuthEnable OIDC/JWT verification; requires tls.oidc.discoveryUrl to be setfalse
gateway.envoy.pluginConfig.wakeTimeoutSecondsTimeout waiting for a sleeping sandbox to wake60
enableTLSMaster switch for cert-manager/trust-manager TLS. Issues the ingress server certificate, the manager/gateway runtime client certificates, the peer certificates, and turns on runtime mTLS in the gateway envoy config. The shared root CA is owned by the sandbox-controller chart, which must be installed in the same namespacefalse
tls.signingIssuerNameCA Issuer created by the sandbox-controller chart (same namespace); not created heresandbox-signing-issuer
tls.signingIssuerKindKind of the referenced issuerIssuer
tls.caCommonNameIngress certificate common name; also the subject organization on issued certificatessandbox-ca
tls.caOrganizationSubject organization on issued certificatesopenkruise
tls.certDurationLeaf certificate lifetime2160h
tls.certRenewBeforeLeaf certificate renewal window360h
tls.ingressCertNameIngress server Certificate resource; its Secret name must match ingress.certSecretName. Provisioned by enableTLS alone and safe against a plaintext runtimesandbox-manager-ingress-cert
tls.runtime.enabledIssue the manager/gateway runtime client certificates and activate --runtime-client-cert-secret plus the gateway enable-runtime-mtls/transport_socket. Fails against an agent-runtime that does not serve TLS; requires enableTLS; set together with the controller chart's tls.runtime.enabledfalse
tls.managerRuntimeClientCertNameManager runtime client Certificate namesandbox-manager-runtime-client
tls.managerRuntimeClientCertSecretNameManager runtime client certificate Secret namesandbox-manager-runtime-client-cert
tls.managerRuntimeClientCommonNameManager runtime client certificate common namesystem:sandbox-manager
tls.gatewayRuntimeClientCertNameGateway runtime client Certificate namesandbox-gateway-runtime-client
tls.gatewayRuntimeClientCertSecretNameGateway runtime client certificate Secret namesandbox-gateway-runtime-client-cert
tls.gatewayRuntimeClientCommonNameGateway runtime client certificate common namesystem:sandbox-gateway
tls.gatewayRuntimeMtlsDirDirectory where the gateway runtime mTLS Secret is mounted; referenced by the envoy transport_socket/var/run/sandbox-gateway/runtime-mtls
tls.agentRuntimeServerSANSAN the agent-runtime server certificate carries; used as envoy SNI and validated via auto_sni_san_validation. Must match the controller chartagentruntime.sandbox.agents.kruise.io
tls.peer.enabledPeer mTLS + memberlist gossip encryption for the managerโ†”gateway control-plane cluster. Independent of the agent-runtime data path; requires enableTLS and images including openkruise/agents#967 (older sandbox-manager images crash on the unknown --peer-* flags)false
tls.peer.serverCertName / serverCertSecretNamePeer TLS server certificate shared by manager and gateway (server auth only; SAN fixed to tls.agentRuntimeServerSAN)sandbox-peer-server / sandbox-peer-server-cert
tls.peer.managerClientCertName / managerClientCertSecretName / managerClientCommonNameManager peer client certificate (client auth only)sandbox-peer-manager-client / sandbox-peer-manager-client-cert / system:sandbox-manager
tls.peer.gatewayClientCertName / gatewayClientCertSecretName / gatewayClientCommonNameGateway peer client certificate (client auth only)sandbox-peer-gateway-client / sandbox-peer-gateway-client-cert / system:sandbox-gateway
tls.peer.keySecretNameExisting Secret (data key key, exactly 32 bytes) for memberlist gossip encryption; empty disables. cert-manager cannot create it; provide it out-of-band, referenced by both manager and gateway""
tls.peer.allowedClientCNsOptional comma-separated inbound allow-list matched against the client certificate CN or DNS SANs; empty accepts any client trusted by the peer server CA""
tls.oidc.discoveryUrlAbsolute HTTPS discovery URL of the token issuer; required when JWT auth is enabled, the verifier fails to initialize while empty""
tls.oidc.caConfigMapName / caConfigMapKeyConfigMap holding the CA verifying the issuer's TLS certificate; defaults point at the trust-manager Bundle created by the controller chartsandbox-ca-bundle / ca.crt
tls.oidc.clockSkewOptional token clock-skew override (e.g. 1m); empty uses the default""
tls.epe.enabledIssue the traffic-extension (EPE) credential-provider client certificate; requires agentio.epe.mode=managed and agentio.epe.credentialProvider.mtls.source=filesfalse
tls.epe.certificateName / commonNameCertificate resource / common name; empty defaults to <epe-fullname>-mtls-client-cert / system:<epe-fullname>""
tls.epe.issuerName / issuerKind / issuerGroupIssuer signing the EPE client certificate; the namespaced default Issuer only works when the agentio namespace also holds the CA Issuersandbox-signing-issuer / Issuer / cert-manager.io
tls.epe.duration / renewBeforeEPE certificate lifetime / renewal window2160h / 360h
agentio.enabledDeploy the embedded Agentio control planefalse
agentio.global.registryDefault registry for Agentio images; empty inherits image.registry""
agentio.global.tagDefault tag for Agentio images, overridable per component0.2.0
agentio.global.imagePullPolicyDefault pull policy for Agentio imagesIfNotPresent
agentio.global.imagePullSecretsDefault pull secrets for Agentio images[]
agentio.global.namespaceAgentio control-plane namespacesandbox-system
agentio.global.createNamespaceCreate the Agentio control-plane namespacetrue
agentio.global.trustDomainAgentio workload identity trust domaincluster.local
agentio.global.clusterDomainKubernetes service DNS domaincluster.local
agentio.global.clusterIdAgentio cluster identifierKubernetes
agentio.global.caCertConfigMapConfigMap distributing the Agentio mesh root CAagentio-ca-certs
agentio.agentiod.ca.trustBundleConfigMapNameCA trust bundle distributed to traffic proxiesagentio-ca-root-cert
agentio.agentiod.replicaCountAgentio control-plane replicas1
agentio.agentiod.image.registryAgentio control-plane image registry; empty inherits agentio.global.registry""
agentio.agentiod.image.repositoryAgentio control-plane image repositoryopenkruise/agentiod
agentio.agentiod.image.tagAgentio control-plane image tag0.2.0
agentio.agentiod.image.digestOptional digest override; takes precedence over tag""
agentio.agentiod.resourcesControl-plane resource requests500m CPU, 512Mi
agentio.epe.modeEPE deployment mode: disabled, managed, or externalmanaged
agentio.epe.image.registryEPE image registry; empty inherits agentio.global.registry""
agentio.epe.image.repositoryEPE image repositoryopenkruise/agentio-epe
agentio.epe.image.tagEPE image tag0.2.0
agentio.epe.image.digestOptional digest override; takes precedence over tag""
agentio.egressGateway.modeGateway mode: disabled, static, or gatewayAPIstatic
agentio.egressGateway.image.registryEgress gateway proxy image registry; empty inherits agentio.global.registry""
agentio.egressGateway.image.repositoryEgress gateway proxy image repositoryopenkruise/proxyv2
agentio.egressGateway.image.tagEgress gateway proxy image tag0.2.0
agentio.egressGateway.image.digestOptional digest override; takes precedence over tag""
agentio.agentiod.config.valuesRaw overrides for the Agentio configuration{}

These parameters can be set via --set key=value[,key=value] in the helm install or helm upgrade commands.

Gateway Default Probe Configuration Reference:

# Liveness probe
gateway.livenessProbe:
tcpSocket:
port: 7788
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3

# Readiness probe
gateway.readinessProbe:
tcpSocket:
port: 7788
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3

Gateway Default Circuit Breaker Configuration Reference:

gateway.envoy.circuitBreakers:
enabled: true
maxConnections: 80000
maxPendingRequests: 32768
maxRequests: 80000
maxRetries: 5

Image Registryโ€‹

Images render as <registry>/<repository>:<tag> or, with a digest override, <registry>/<repository>@<digest>. The registry defaults to the chart-wide image.registry. Agentio images resolve their registry through a longer chain: their own image.registry, then agentio.global.registry, then image.registry. Agentio images use the fixed 0.2.0 tag by default; an optional digest overrides the tag.

The registry prefix is dropped when the first path segment of a repository already names a host (it contains a . or a :), so setting controller.repository to myreg.io/openkruise/sandbox-manager or image.repository to myreg.io/openkruise/agent-sandbox-controller keeps working without also clearing image.registry.

TLS (cert-manager / trust-manager)โ€‹

All certificate issuance is gated by enableTLS (default false), which requires cert-manager (and trust-manager, for the CA Bundle) in the cluster. The shared root CA (Issuer and CA Certificate) is owned by the Sandbox Controller chart and must be installed in the same namespace; the Sandbox Manager chart only references the tls.signingIssuerName Issuer by name.

The switches are independent and can be adopted incrementally:

  • enableTLS=true provisions the ingress server certificate (tls.ingressCertName, wired into ingress.certSecretName). External client โ†’ ingress HTTPS is independent of the agent-runtime, so this is safe and effective today, even with a plaintext runtime.
  • tls.runtime.enabled=true additionally issues the manager, gateway, and controller runtime certificates and activates --runtime-client-cert-secret plus the gateway enable-runtime-mtls/transport_socket. Those client paths fail against an agent-runtime that does not yet serve TLS โ€” enable only once the agent-runtime image supports runtime TLS, and set it together with tls.runtime.enabled in the controller chart for the full runtime path.
  • tls.peer.enabled=true secures the managerโ†”gateway route-sync/gossip channel (peer mTLS plus optional memberlist gossip encryption via tls.peer.keySecretName). It is independent of the agent-runtime data path and safe to enable on its own, but requires manager/gateway images that include openkruise/agents#967: an older sandbox-manager crashes on the unknown --peer-* flags. Verify the image before enabling.
  • tls.epe.enabled=true issues the EPE credential-provider client certificate. Requires agentio.epe.mode=managed with agentio.epe.credentialProvider.mtls.source=files; the certificate is written into the same files Secret EPE mounts with keys remapped to client.crt/client.key/ca.crt. Mesh workload certificates (egress gateway, traffic-proxy/ztunnel mTLS) are not issued here โ€” agentiod/pilot issues them on demand as SPIFFE SVIDs.
  • gateway.envoy.pluginConfig.enableJwtAuth=true (with enableTLS) turns on OIDC/JWT verification at the gateway. Requires tls.oidc.discoveryUrl; the verifier reads its CA from the tls.oidc.caConfigMapName ConfigMap through the API server, so that ConfigMap must exist in this namespace (the defaults point at the trust-manager Bundle created by the controller chart).

Agent Runtime Injectionโ€‹

The sandbox-injection-config ConfigMap ships an agent-runtime entry. It is applied only to sandboxes that explicitly opt in by declaring the runtime in Sandbox.spec.runtimes:

apiVersion: agents.kruise.io/v1alpha1
kind: Sandbox
metadata:
name: demo
spec:
runtimes:
- name: agent-runtime
# ... pod template

When the runtime is declared, the controller injects a native sidecar container named agent-runtime (built from agentRuntime.image.*, default openkruise/agent-runtime:v0.3.0), ENVD_DIR/GODEBUG/POD_UID environment variables, the envd-volume mount at /mnt/envd, and a postStart hook into the first business container.

Requirements:

  • Kubernetes >= 1.29. The agent-runtime container is injected as a native sidecar (an init container with restartPolicy: Always), which requires the SidecarContainers feature to be enabled by default. On older clusters the injected pod will be rejected or the sidecar will not restart as expected.
  • The first business container image must contain bash. The injected postStart hook runs bash /mnt/envd/envd-run.sh inside that container, so images without a bash binary (for example plain distroless or busybox based images) will fail to start.

The Controller chart intentionally ships only the traffic-proxy and agent-runtime injection entries. The TLS / helper runtime and the CSI runtime are not included; deploy those components separately if your environment needs them.


Third-Party Dependenciesโ€‹

Neither chart requires any of the components below to run. Each component only backs a specific feature: install it and turn on the corresponding switch when you need that feature.

cert-manager / trust-managerโ€‹

  • Dependent feature: TLS termination. With enableTLS=true, cert-manager issues the ingress server certificate plus the manager/gateway/controller runtime and peer certificates, and trust-manager distributes the shared CA as a ca.crt ConfigMap.
  • How to enable: install cert-manager (and trust-manager for the CA Bundle) in the cluster, then set enableTLS=true on both charts โ€” the Sandbox Controller chart owns the root CA Issuer and Certificate, the Sandbox Manager chart only references the Issuer by name. See TLS (cert-manager / trust-manager) for the individual switches.

OpenKruiseโ€‹

  • Dependent feature: probe-driven auto pause/resume on real nodes. With the KruiseIntegration feature gate enabled, the controller creates OpenKruise PodProbeMarker resources and kruise-daemon executes the probes defined in Sandbox.spec.probes inside the sandbox pod. Without the gate, real-node probe conditions stay Unknown and auto-pause decisions fail closed โ€” probes on virtual-kubelet nodes still run through the kruise.io/podprobe annotation.
  • How to enable: install OpenKruise in the cluster, then install or upgrade the Sandbox Controller with the gate enabled: --set 'controller.featureGates=KruiseIntegration=true'.

Redisโ€‹

  • Dependent feature: quota accounting. With quota.enabled=true, the manager keeps quota counters in Redis.
  • How to enable: deploy a Redis instance, then install or upgrade the Sandbox Manager with --set quota.enabled=true --set quota.redis.addr=<host:port>, plus quota.redis.db, quota.redis.username, and quota.redis.password as needed.

MySQLโ€‹

  • Dependent feature: E2B API key storage. With e2b.keyStorage.mode=mysql, API keys are stored in MySQL instead of the default e2b-key-store Secret.
  • How to enable: prepare the database and the DSN, then install or upgrade the Sandbox Manager with --set e2b.keyStorage.mode=mysql --set e2b.keyStorage.mysql.dsn=<dsn> --set e2b.keyStorage.mysql.hashPepper=<pepper>. Both the DSN and the pepper are required and validated at manager startup.

Prometheusโ€‹

  • Dependent feature: metrics collection.
  • How to enable:
    • Manager and gateway: --set prometheus.enabled=true creates a ServiceMonitor for both; requires the Prometheus Operator CRDs in the cluster.
    • Gateway Envoy: --set gateway.envoy.prometheus.enabled=true exposes the Envoy admin /stats/prometheus on port 9902.
    • Controller: the chart grants get on the /metrics nonResourceURL through metrics.rbac.create (default true); bind the generated ClusterRole to your Prometheus scraper ServiceAccount.

Ingress Controllerโ€‹

  • Dependent feature: external access. ingress.className is required to expose the manager API (api.<domain>) and the gateway (*.<domain>) through an Ingress.
  • How to enable: install an Ingress controller (for example ALB or nginx), then install or upgrade the Sandbox Manager with --set ingress.className=<alb|nginx>. See Expose Service via Ingress for a full example.

Best Practicesโ€‹

Custom Resource Configurationโ€‹

Based on your cluster scale, it is recommended to adjust the following resource parameters:

Sandbox Controller resource adjustment

helm install agents-sandbox-controller openkruise/agents-sandbox-controller \
-n sandbox-system \
--version 0.6.0 \
--set resources.limits.cpu=4 \
--set resources.limits.memory=8Gi \
--set resources.requests.cpu=2 \
--set resources.requests.memory=4Gi

Sandbox Manager + Gateway resource adjustment

helm install agents-sandbox-manager openkruise/agents-sandbox-manager \
-n sandbox-system \
--version 0.6.0 \
--set e2b.domain=<your-domain> \
--set e2b.adminApiKey=<your-api-key> \
--set ingress.className=<your-ingress-class> \
--set controller.resources.cpu=4 \
--set controller.resources.memory=8Gi \
--set gateway.resources.cpu=4 \
--set gateway.resources.memory=8Gi

Configure E2B Domain and Authenticationโ€‹

helm install agents-sandbox-manager openkruise/agents-sandbox-manager \
-n sandbox-system \
--version 0.6.0 \
--set e2b.domain=sandbox.example.com \
--set e2b.enableAuth=true \
--set e2b.adminApiKey=your-secure-api-key \
--set ingress.className=<your-ingress-class>

Expose Service via Ingressโ€‹

helm install agents-sandbox-manager openkruise/agents-sandbox-manager \
-n sandbox-system \
--version 0.6.0 \
--set e2b.domain=<your-domain> \
--set e2b.adminApiKey=<your-api-key> \
--set ingress.className=nginx \
--set ingress.annotations."cert-manager\.io/cluster-issuer"=letsencrypt-prod \
--set ingress.certSecretName=sandbox-manager-tls

Configure Gateway High Availabilityโ€‹

helm install agents-sandbox-manager openkruise/agents-sandbox-manager \
-n sandbox-system \
--version 0.6.0 \
--set e2b.domain=<your-domain> \
--set e2b.adminApiKey=<your-api-key> \
--set ingress.className=<your-ingress-class> \
--set gateway.replicaCount=3 \
--set gateway.podAntiAffinity.type=hard \
--set gateway.resources.cpu=4 \
--set gateway.resources.memory=8Gi

Enable Gateway Init Containerโ€‹

If special initialization operations are needed (such as sysctl tuning, etc.), you can enable the init container:

helm install agents-sandbox-manager openkruise/agents-sandbox-manager \
-n sandbox-system \
--version 0.6.0 \
--set e2b.domain=<your-domain> \
--set e2b.adminApiKey=<your-api-key> \
--set ingress.className=<your-ingress-class> \
--set gateway.initContainer.enabled=true \
--set gateway.initContainer.image.repository=busybox \
--set gateway.initContainer.image.tag=1.36.1

Uninstallโ€‹

Note:

  • helm uninstall will delete Deployments, Services, Webhook Configurations, and other chart-managed resources, but will not delete CRDs. This is standard Helm behavior โ€” CRDs are located in the crds/ directory, and Helm only creates them during initial installation, leaving them untouched during uninstall and upgrade.
  • CRDs not being deleted means that already created Sandbox, SandboxSet, and other CR resources along with their associated Pods will be retained.
  • The Namespace will not be automatically deleted either. For a complete cleanup, refer to the "Complete Cleanup" section below.

Uninstall order: Uninstall Sandbox Manager first, then Sandbox Controller.

Uninstall Sandbox Managerโ€‹

helm uninstall agents-sandbox-manager -n sandbox-system

Uninstall Sandbox Controllerโ€‹

helm uninstall agents-sandbox-controller -n sandbox-system

Complete Cleanup (Optional)โ€‹

To completely clean up all resources, including CRDs and Namespace:

# Delete all Sandbox-related CRDs (will cascade delete all Sandbox CRs and corresponding Pods)
kubectl get crd | grep agents.kruise.io | awk '{print $1}' | xargs kubectl delete crd

# Delete the Namespace
kubectl delete ns sandbox-system

โš ๏ธ Warning: Deleting CRDs will irreversibly destroy all Sandbox instances and their associated Pods. Make sure data is backed up before proceeding.


Version Update Notesโ€‹

Major Changes in 0.6.0 Compared to 0.3.0โ€‹

CategoryChanges
New CRDsNew Commit (commits.agents.kruise.io) and PoolAutoscaler (poolautoscalers.agents.kruise.io) CRDs from the Controller chart; new TrafficPolicy, GlobalTrafficPolicy, SecurityProfile, and GlobalSecurityProfile CRDs from the Manager chart for sandbox ingress/egress control. Controller CRDs must be manually updated during upgrade; the Manager chart's CRDs are updated automatically by helm upgrade
Chart ArchitectureThe Manager Pod no longer runs an Envoy sidecar; the data plane is served solely by the Sandbox Gateway Deployment (Service port 7788), and the Manager Service listens on port 8080. All images resolve through a chart-wide image.registry. The Controller chart no longer supports server-side apply
Checkpoint, Pause & CommitNew CheckpointControl lifecycle with PersistentContents filesystem checkpoints and selectable checkpoint labels; pause waits for active checkpoints; new Commit CRD with registry authentication and a nerdctl-based commit/push job
Cost OptimizationSandbox recycle (return-to-pool) to avoid cold starts; probe-driven auto-pause (AutoPausePolicy) and wake-on-traffic resume (OnIngressTraffic); PauseStrategy (Stop / Snapshot / CloudDisk) on SandboxSet; PoolAutoscaler for capacity-based and cron-driven scaling
Security & Egress ControlTrafficPolicy / SecurityProfile driven egress control (protocol and scheme matching, inline E2B L7 rules), MCP tool access control, header manipulation, and hardened CRD admission validation
Identity & TokensFeatureGate-controlled Security Identity Provider issuing and propagating tokens across the sandbox lifecycle; tokens issued at claim time and re-issued after resume; proactive rotation via a SecurityTokenRefresh reconciler
Gateway & TransportGateway JWT verification with optional runtime mTLS; CA bundle injection framework; the CSI mount and /init handshake move to TLS-capable transports; traffic access token header aligned with the E2B SDK
Upgrade & In-Place UpdatePaused sandboxes can be upgraded via SandboxUpdateOps (two-phase flow) and the CheckpointRestore strategy; sandbox memory can be resized during claims; in-place resource resize checks relaxed
E2B CompatibilityClaude Code support; pod-IP metadata; E2B >= v2.25.0 API key encoding; named cloned sandboxes; dynamically resolved sandbox domains; Volume API and Network API (Volume management endpoints temporarily disabled); dimension-aware API key quota
Controller & SandboxSetSandboxSet auto-creates SandboxTemplate; a legacy revision hash prevents sandbox recreation on upgrade; maxUnavailable scoped to a startup-failure budget; scale-down candidates sorted by priority
Identifiers & CLIShort, stable sandbox IDs reduce identifier length while preserving uniqueness across lifecycle operations; new okactl CLI for sandbox operations; multi-arch image publishing
ObservabilityNew metrics for abnormal runtime containers; events and conditions for Pod creation failures and sandbox lifecycle; lifecycle tracing for controller and manager

For detailed changes, please refer to the Change Log.