Spanner Autoscaler is a Kubernetes Operator to scale Google Cloud Spanner automatically based on Cloud Spanner Instance CPU utilization like Horizontal Pod Autoscaler.

Cloud Spanner is scalable. When CPU utilization becomes high, we can reduce it by increasing compute capacity.

Spanner Autoscaler is created to reconcile Cloud Spanner compute capacity like Horizontal Pod Autoscaler by configuring a compute capacity range and targetCPUUtilization.

When CPU Utilization(High Priority) is above (or below) targetCPUUtilization, Spanner Autoscaler tries to bring it back to the threshold by calculating desired compute capacity and then increasing (or decreasing) compute capacity.

The pricing of Cloud Spanner states that any compute capacity which is provisioned will be billed for a minimum of one hour, so Spanner Autoscaler maintains the increased compute capacity for about an hour. Spanner Autoscaler has --scale-down-interval flag (default: 55min) for achieving this.

While scaling down, removing large amounts of compute capacity at once (like 10000 PU -> 1000 PU) can cause a latency increase. Therefore, Spanner Autoscaler decreases the compute capacity in steps to avoid such large disruptions. This step size can be provided with the scaledownStepSize parameter (default: 2000 PU).

If there are some batch jobs or any other compute intensive tasks which are run periodically on the Cloud Spanner, it is now possible to bump up the scaling range only for a specified duration. For example, the following SpannerAutoscaleSchedule will add an extra compute capacity of 600 Processing Units to the spanner instance every day at 2 o'clock, just for 3 hours:

apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaleSchedule
metadata:
  name: spannerautoscaleschedule-sample
  namespace: your-namespace
spec:
  targetResource: spannerautoscaler-sample
  additionalProcessingUnits: 600
  schedule:
    cron: "0 2 * * *"
    duration: 3h

Spanner Autoscaler can be installed using KPT by following 2 steps:

1. Deploy the operator through kpt

   $ kpt pkg get https://github.com/mercari/spanner-autoscaler/config spanner-autoscaler-pkg
   $ kpt live init spanner-autoscaler-pkg/kpt
   $ kpt live install-resource-group
   
   ## Append '--dry-run' to the below line to just
   ## check the resources which will be created
   $ kustomize build spanner-autoscaler-pkg/kpt | kpt live apply -
   
   ## To uninstall, use the following
   $ kustomize build spanner-autoscaler-pkg/kpt | kpt live destroy -

   ℹ️ TIP: Instead of kpt, you can also use kubectl directly to install the resources (use ?ref=master for latest version) as follows:

   $ kustomize build "https://github.com/mercari/spanner-autoscaler.git/config/default?ref=v0.4.1" | kubectl apply -f -

   These resources can then be adopted by kpt by using the --inventory-policy=adopt flag while using kpt live apply command. More info.

2. Create a Custom Resource for managing a spanner instance

   $ kubectl apply -f spanner-autoscaler-pkg/samples

   Examples of CustomResources can be found below.
   For authentication using a GCP service account JSON key, follow these steps to create a k8s secret with credentials.

• SpannerAutoscaler CRD reference
• SpannerAutoscaleSchedule CRD reference

apiVersion: spanner.mercari.com/v1beta1
kind: SpannerAutoscaler
metadata:
  name: spannerautoscaler-sample
  namespace: your-namespace
spec:
  targetInstance:
    projectId: your-gcp-project-id
    instanceId: your-spanner-instance-id
  scaleConfig:
    processingUnits:
      min: 1000
      max: 4000
    scaledownStepSize: 1000
    targetCPUUtilization:
      highPriority: 60

  apiVersion: spanner.mercari.com/v1beta1
  kind: SpannerAutoscaler
  metadata:
    name: spannerautoscaler-sample
    namespace: your-namespace
  spec:
    targetInstance:
      projectId: your-gcp-project-id
      instanceId: your-spanner-instance-id
    authentication:
      iamKeySecret:
        namespace: your-namespace
        name: spanner-autoscaler-gcp-sa
        key: service-account
    scaleConfig:
      processingUnits:
        min: 1000
        max: 4000
      scaledownStepSize: 1000
      targetCPUUtilization:
        highPriority: 60

  apiVersion: spanner.mercari.com/v1beta1
  kind: SpannerAutoscaler
  metadata:
    name: spannerautoscaler-sample
    namespace: your-namespace
  spec:
    targetInstance:
      projectId: your-gcp-project-id
      instanceId: your-spanner-instance-id
    authentication:
      impersonateConfig:
        targetServiceAccount: GSA_SPANNER@TENANT_PROJECT.iam.gserviceaccount.com
    scaleConfig:
      processingUnits:
        min: 1000
        max: 4000
      scaledownStepSize: 1000
      targetCPUUtilization:
        highPriority: 60

On your GCP project, you will need to enable spanner.googleapis.com and monitoring.googleapis.com APIs.

You will need to create at least one GCP service account, which will be used by the spanner-autoscaler controller to authenticate with GCP for modifying compute capacity of a Spanner instance. This service account should have the following roles:

• roles/spanner.admin (on the Spanner instances)
• roles/monitoring.viewer (on the project)

For fine grained access control, you should create one GCP service account per Spanner instance. This way, you will be able to specify a different service account in each of SpannerAutoscaler CRD resources you create later.

Generate a JSON key for the GCP service account (created above) and put it in a Kubernetes Secret:

$ kubectl create secret generic spanner-autoscaler-gcp-sa --from-file=service-account=./service-account-key.json -n your-namespace

ℹ️ By default, spanner-autoscaler will have read access to secrets named spanner-autoscaler-gcp-sa in any namespace. If you wish to use a different name for your secret, then you need to explicitly create a Role and a RoleBinding (example) in your namespace. This will provide spanner-autoscaler with read access to any secret of your choice.

You can then refer to this secret in your SpannerAutoscaler CRD resource with serviceAccountSecretRef field [example].

Following are some other advanced methods which can also be used for GCP authentication:

See docs/development.md and CONTRIBUTING.md respectively.

The older version 0.3.0 (with apiVersion: spanner.mercari.com/v1alpha1) is now deprecated in favor of 0.4.0 (with apiVersion: spanner.mercari.com/v1beta1).

Version 0.4.0 is backward compatible with 0.3.0, but there is a restructuring of the SpannerAutoscaler resource definition and names of many fields have changed. Thus it is recommended to go through the SpannerAutoscaler CRD reference and replace v1alpha1 resources with v1beta1 spec definition.

Spanner Autoscaler is released under the Apache License 2.0.

⚠️ NOTE:

1. This project is currently in active development phase and there might be some backward incompatible changes in future versions.
2. Spanner Autoscaler watches High Priority CPU utilization only. It doesn't watch Low Priority CPU utilization and Rolling average 24 hour utilization.
3. It doesn't check the storage size and the number of databases as well. You must take care of these metrics by yourself.

ℹ️ More information and background of spanner-autoscaler is available on this blog!