Skip to main content

Gameservers Scale

The horizontal scaling feature of OpenKruiseGame​

Sequence of scale-down​

OpenKruiseGame allows you to set the states of game servers. You can manually set the value of opsState or DeletionPriority for a game server. You can also use the service quality feature to automatically set the value of opsState or DeletionPriority for a game server. During scale-in, a proper GameServerSet workload is selected for scale-down based on the states of game servers. The scale-down rules are as follows:

  1. Scale in game servers based on the opsState values. Scale in the game servers for which the opsState values are WaitToBeDeleted, None, Allocated, and Maintaining in sequence.

  2. If two or more game servers have the same opsState value, game servers are scaled-down based on the values of DeletionPriority. The game server with the largest DeletionPriority value is deleted first.

  3. If two or multiple game servers have the same opsState value and DeletionPriority value, the game server whose name contains the largest sequence number in the end is deleted first.

Examples​

Deploy a game server with five replicas:

cat <<EOF | kubectl apply -f -
apiVersion: game.kruise.io/v1alpha1
kind: GameServerSet
metadata:
name: minecraft
namespace: default
spec:
replicas: 5
updateStrategy:
rollingUpdate:
podUpdatePolicy: InPlaceIfPossible
gameServerTemplate:
spec:
containers:
- image: registry.cn-hangzhou.aliyuncs.com/acs/minecraft-demo:1.12.2
name: minecraft
EOF

Five game servers are generated:

kubectl get gs
NAME STATE OPSSTATE DP UP
minecraft-0 Ready None 0 0
minecraft-1 Ready None 0 0
minecraft-2 Ready None 0 0
minecraft-3 Ready None 0 0
minecraft-4 Ready None 0 0

Set DeletionPriority to 10 for minecraft-2:

kubectl edit gs minecraft-2

...
spec:
DeletionPriority: 10 # Change the value of DeletionPriority from the initial value 0 to 10.
opsState: None
updatePriority: 0
...

Manually perform scale-down to reduce the number of the game servers to 4:

kubectl scale gss minecraft --replicas=4
gameserverset.game.kruise.io/minecraft scale

The number of the game servers is changed to 4. The following example shows that minecraft-2 is deleted because it has the largest DeletionPriority value.

kubectl get gs
NAME STATE OPSSTATE DP UP
minecraft-0 Ready None 0 0
minecraft-1 Ready None 0 0
minecraft-2 Deleting None 10 0
minecraft-3 Ready None 0 0
minecraft-4 Ready None 0 0

# After a while
...

kubectl get gs
NAME STATE OPSSTATE DP UP
minecraft-0 Ready None 0 0
minecraft-1 Ready None 0 0
minecraft-3 Ready None 0 0
minecraft-4 Ready None 0 0

Set opsState to WaitToBeDeleted for minecraft-3:

kubectl edit gs minecraft-3

...
spec:
deletionPriority: 0
opsState: WaitToBeDeleted # Change the value of opsState from the initial value None to WaitToBeDeleted.
updatePriority: 0
...

Manually perform scale-down to reduce the number of the game servers to 3:

kubectl scale gss minecraft --replicas=3
gameserverset.game.kruise.io/minecraft scaled

The number of replicas for the game server is changed to 3. You can see that minecraft-3 is deleted because its opsState value is WaitToBeDeleted.

kubectl get gs
NAME STATE OPSSTATE DP UP
minecraft-0 Ready None 0 0
minecraft-1 Ready None 0 0
minecraft-3 Deleting WaitToBeDeleted 0 0
minecraft-4 Ready None 0 0

# After a while
...

kubectl get gs
NAME STATE OPSSTATE DP UP
minecraft-0 Ready None 0 0
minecraft-1 Ready None 0 0
minecraft-4 Ready None 0 0

Manually perform scale-up and change the number of replicas for the game server back to 5:

kubectl scale gss minecraft --replicas=5
gameserverset.game.kruise.io/minecraft scaled

The number of replicas for the game server is changed back to 5. You can see that minecraft-2 and minecraft-3 are added for the game server.

kubectl get gs
NAME STATE OPSSTATE DP UP
minecraft-0 Ready None 0 0
minecraft-1 Ready None 0 0
minecraft-2 Ready None 0 0
minecraft-3 Ready None 0 0
minecraft-4 Ready None 0 0

GameServer ID Reserve​

GameServerSet provides the Spec.ReserveGameServerIds field. Through this field, the user specifies the ID to delete the corresponding game service; or to avoid the generation of the game service corresponding to this serial number when creating a new game service.

For example, there exist 5 game suits under gss with IDs 0, 1, 2, 3 and 4. At this time, set ReserveGameServerIds and fill in 3 and 4, without changing the number of copies, gss will delete 3 and 4 and generate game suits of 5 and 6 at the same time, as follows:

kubectl edit gss minecraft
...
spec:
reserveGameServerIds:
- 3
- 4
...

# After a while
kubectl get gs
NAME STATE OPSSTATE DP UP AGE
minecraft-0 Ready None 0 0 79s
minecraft-1 Ready None 0 0 79s
minecraft-2 Ready None 0 0 79s
minecraft-3 Deleting None 0 0 78s
minecraft-4 Deleting None 0 0 78s
minecraft-5 Ready None 0 0 23s
minecraft-6 Ready None 0 0 23s

If you fill in the ReserveGameServerIds field by adding 5 and 6 while decreasing the number of copies to 3, gss will delete the game service for 5 and 6, as follows:

kubectl edit gss minecraft
...
spec:
replicas: 3
reserveGameServerIds:
- 3
- 4
- 5
- 6
...

# After a while
kubectl get gs
NAME STATE OPSSTATE DP UP AGE
minecraft-0 Ready None 0 0 10m
minecraft-1 Ready None 0 0 10m
minecraft-2 Ready None 0 0 10m
minecraft-5 Deleting None 0 0 9m55s
minecraft-6 Deleting None 0 0 9m55s

When downsizing, OKG will give priority to the game suits that are Reserved, and then follow the downsizing order mentioned above

Kill GameServer​

OKG provides Kill mode to delete specified game server. Users only need to mark the OpsState of the game server they want to delete as "Kill". Different from Game Server ID Reserved, in this mode, users do not need to manually adjust the replicas, OKG will automatically reduce replicas according to the number of GameServers whose OpsState is Kill; in addition, the deleted game server ID will not appear in the ReserveGameServerIds field(With General scaleDownStrategyType), which means that the game server with the corresponding serial number may be regenerated during scaling-up.

For example:

# There are 3 GameServers
kubectl get gs
NAME STATE OPSSTATE DP UP AGE
minecraft-0 Ready None 0 0 70s
minecraft-1 Ready None 0 0 70s
minecraft-2 Ready None 0 0 70s

# If you want to delete game server 1, just mark its OpsState as "Kill"
kubectl edit gs minecraft-1
...
spec:
opsState: Kill
...

# minecraft-1 will be deleted, and the replicas of gss becomes 2
kubectl get gs
NAME STATE OPSSTATE DP UP AGE
minecraft-0 Ready None 0 0 78s
minecraft-1 Deleting Kill 0 0 78s
minecraft-2 Ready None 0 0 78s

kubectl get gs
NAME STATE OPSSTATE DP UP AGE
minecraft-0 Ready None 0 0 82s
minecraft-2 Ready None 0 0 82s

Strategy of scale-down​

OKG provides two kinds of scale-down strategies: 1) General; 2) ReserveIds. You can set the corresponding strategy in GameServerSet.Spec.ScaleStrategy.

General​

When the user does not configure the ScaleDownStrategyType field, General is the default configuration. The scale-down behavior is as described above.

ReserveIds​

The user sets the ScaleDownStrategyType to ReserveIds, when the game service is scale-down, the deleted game service tail serial numbers will be recorded in reserveGameServerIds, and these tail serial numbers will not be used again when scale-up; if you want to use these tail serial numbers again, you only need to take them out from reserveGameServerIds and adjust the number of copies at the same time.

Examples​

For example, there are 5 game services under gss, with IDs 0, 1, 2, 3, 4. Set GameServerSet.Spec.ScaleStrategy.ScaleDownStrategyType to ReserveIds and reduce the number of copies to 3.

kubectl edit gss minecraft
...
spec:
replicas: 3
scaleStrategy:
scaleDownStrategyType: ReserveIds
...

# After a while
kubectl get gs
NAME STATE OPSSTATE DP UP AGE
minecraft-0 Ready None 0 0 10m
minecraft-1 Ready None 0 0 10m
minecraft-2 Ready None 0 0 10m
minecraft-3 Deleting None 0 0 9m55s
minecraft-4 Deleting None 0 0 9m55s
...

kubectl get gss minecraft -o yaml
spec:
replicas: 3
reserveGameServerIds:
- 3
- 4
scaleStrategy:
scaleDownStrategyType: ReserveIds

As you can see, the game services with serial numbers 3 and 4 are backfilled into the reserveGameServerIds field. At this point, if you wish to specify the expansion of game service number 4, remove 4 from the reserveGameServerIds and increase the number of copies to 4:

kubectl edit gss minecraft
...
spec:
replicas: 4
reserveGameServerIds:
- 3
scaleStrategy:
scaleDownStrategyType: ReserveIds
...

# After a while

kubectl get gs
NAME STATE OPSSTATE DP UP AGE
minecraft-0 Ready None 0 0 17m
minecraft-1 Ready None 0 0 17m
minecraft-2 Ready None 0 0 17m
minecraft-4 Ready None 0 0 6s

This function can scale up the game service with the specified serial number.

Horizontal autoscale of gameserver​

Auto Scaling-down​

Compared to stateless service types, game servers have higher requirements for automatic scaling, especially in terms of scaling down.

The differences between game servers become more and more obvious over time, and the precision requirements for scaling down are extremely high. Coarse-grained scaling mechanisms can easily cause negative effects such as player disconnections, resulting in huge losses for the business.

The horizontal scaling mechanism in native Kubernetes is shown in the following figure:

In the game scenario, its main problems are:

  • At the pod level, it is unable to perceive the game server game status and therefore cannot set deletion priority based on game status.
  • At the workload level, it cannot select scale-down objects based on game status.
  • At the autoscaler level, it cannot accurately calculate the appropriate number of replicas based on the game server game status.

In this way, the automatic scaling mechanism based on native Kubernetes will cause two major problems in the game scenario:

  • The number of scale-down is not accurate. It is easy to delete too many or too few game servers.
  • The scale-down object is not accurate. It is easy to delete game servers with high game load levels.

The automatic scaling mechanism of OKG is shown in the following figure:

  • At the game server level, each game server can report its own status and expose whether it is in the WaitToBeDeleted state through custom service quality or external components.
  • At the workload level, the GameServerSet can determine the scale-down object based on the business status reported by the game server. As described in Game Server Horizontal Scaling, the game server in the WaitToBeDeleted state is the highest priority game server to be deleted during scale-down.
  • At the autoscaler level, accurately calculate the number of game servers in the WaitToBeDeleted state, and use it as the scale-down quantity to avoid accidental deletion.

In this way, OKG's automatic scaler will only delete game servers in the WaitToBeDeleted state during the scale-down window, achieving targeted and precise scale-down.

Usage Example

Prerequisites: Install KEDA in the cluster.

Deploy the ScaledObject object to set the automatic scaling strategy. Refer to the ScaledObject API for the specific field meanings.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: minecraft # Fill in the name of the corresponding GameServerSet
spec:
scaleTargetRef:
name: minecraft # Fill in the name of the corresponding GameServerSet
apiVersion: game.kruise.io/v1alpha1
kind: GameServerSet
pollingInterval: 30
minReplicaCount: 0
advanced:
horizontalPodAutoscalerConfig:
behavior: # Inherit from HPA behavior, refer to https://kubernetes.io/zh-cn/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior
scaleDown:
stabilizationWindowSeconds: 45 # Set the scaling-down stabilization window time to 45 seconds
policies:
- type: Percent
value: 100
periodSeconds: 15
triggers:
- type: external
metricType: AverageValue
metadata:
scalerAddress: kruise-game-external-scaler.kruise-game-system:6000

After deployment, change the opsState of the gs minecraft-0 to WaitToBeDeleted (see Custom Service Quality for automated setting of game server status).

kubectl edit gs minecraft-0

...
spec:
deletionPriority: 0
opsState: WaitToBeDeleted # Set to None initially, and change it to WaitToBeDeleted
updatePriority: 0
...

After the scale-down window period, the game server minecraft-0 is automatically deleted.

kubectl get gs
NAME STATE OPSSTATE DP UP
minecraft-0 Deleting WaitToBeDeleted 0 0
minecraft-1 Ready None 0 0
minecraft-2 Ready None 0 0

# After a while


kubectl get gs
NAME STATE OPSSTATE DP UP
minecraft-1 Ready None 0 0
minecraft-2 Ready None 0 0

Auto Scaling-up​

In addition to setting the automatic scaling policy, you can also set the automatic scaling policy.

Scaling with resource metrics or custom metrics​

Native Kubernetes supports auto scaling-up using CPU utilization, and its complete yaml is as follows:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: minecraft # Fill in the name of the corresponding GameServerSet
spec:
scaleTargetRef:
name: minecraft # Fill in the name of the corresponding GameServerSet
apiVersion: game.kruise.io/v1alpha1
kind: GameServerSet
pollingInterval: 30
minReplicaCount: 0
advanced:
horizontalPodAutoscalerConfig:
behavior: # Inherit from HPA behavior, refer to https://kubernetes.io/zh-cn/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior
scaleDown:
stabilizationWindowSeconds: 45 # Set the scaling-down stabilization window time to 45 seconds
policies:
- type: Percent
value: 100
periodSeconds: 15
triggers:
- type: external
metricType: AverageValue
metadata:
scalerAddress: kruise-game-external-scaler.kruise-game-system:6000
- type: cpu
metricType: Utilization # Allowed types are 'Utilization' or 'AverageValue'
metadata:
value: "50"

Pressure testing of the gameserver, you can see that the gameserver began to scale-up

kubectl get gss
NAME DESIRED CURRENT UPDATED READY MAINTAINING WAITTOBEDELETED AGE
minecraft 5 5 5 0 0 0 7s

# After a while

kubectl get gss
NAME DESIRED CURRENT UPDATED READY MAINTAINING WAITTOBEDELETED AGE
minecraft 20 20 20 20 0 0 137s

Set the minimum number of game servers whose opsState is None​

OKG supports setting the minimum number of game servers. When the current number of game servers whose opsState is None is less than the set value, OKG will automatically expand new game servers so that the number of game servers whose opsState is None meets the set minimum number.

The configuration method is as follows. In this example, the minimum number of game servers with opsState set to None is 3:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: minecraft # Fill in the name of the corresponding GameServerSet
spec:
scaleTargetRef:
name: minecraft # Fill in the name of the corresponding GameServerSet
apiVersion: game.kruise.io/v1alpha1
kind: GameServerSet
pollingInterval: 30
minReplicaCount: 0
advanced:
horizontalPodAutoscalerConfig:
behavior: # Inherit from HPA behavior, refer to https://kubernetes.io/zh-cn/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior
scaleDown:
stabilizationWindowSeconds: 45 # Set the scaling-down stabilization window time to 45 seconds
policies:
- type: Percent
value: 100
periodSeconds: 15
triggers:
- type: external
metricType: AverageValue
metadata:
minAvailable: "3" # 设置opsState为None的游戏服的最小个数
scalerAddress: kruise-game-external-scaler.kruise-game-system:6000

First apply a GameServerSet with 1 replicas, after the KEDA detection cycle, immediately scale up two new game servers. At this time, the number of game servers whose opsState is None is not less than the minAvailable value, and scale-up process is completed.

kubectl get gs
NAME STATE OPSSTATE DP UP AGE
minecraft-0 Ready None 0 0 7s

# After a while

kubectl get gs
NAME STATE OPSSTATE DP UP AGE
minecraft-0 Ready None 0 0 20s
minecraft-1 Ready None 0 0 5s
minecraft-2 Ready None 0 0 5s

Other Settings​

Kubernetes has a certain tolerance for automatic scaling behavior, which is determined by the kube-controller-manager parameter --horizontal-pod-autoscaler-tolerance, and the default is 0.1, which means that the difference between the desired replicas and the current replicas is 10% No expansion or contraction will be triggered when the value is within. If more accurate auto-scaling is achieved, this parameter can be lowered. For example, when it is set to 0.0, OKG will scale down all GameServers with WaitToBeDeleted opsState.