在 GKE 上部署 Elasticsearch 矢量数据库

设置您的环境

如需使用 Cloud Shell 设置您的环境，请按照以下步骤操作：

1. 为您的项目、区域和 Kubernetes 集群资源前缀设置环境变量：

   export PROJECT_ID=PROJECT_ID
   export KUBERNETES_CLUSTER_PREFIX=elasticsearch
   export REGION=us-central1
   

   • 将 PROJECT_ID 替换为您的 Google Cloud 项目 ID。

   本教程使用 us-central1 区域创建部署资源。

2. 检查 Helm 的版本：

   helm version
   

   更新版本（如果低于 3.13）：

   curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
   

3. 从 GitHub 克隆示例代码库：

   git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples
   

4. 转到 elasticsearch 目录以开始创建部署资源：

   cd kubernetes-engine-samples/databases/elasticsearch
   

创建集群基础架构

在本部分中，您将运行 Terraform 脚本来创建高可用性专用区域级 GKE 集群，以部署 Elasticsearch 数据库。

您可以选择使用 Standard 集群或 Autopilot 集群部署 Elasticsearch。每种类型都有自己的优势和不同的价格模式。

export GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token)
terraform -chdir=terraform/gke-autopilot init
terraform -chdir=terraform/gke-autopilot apply \
-var project_id=${PROJECT_ID} \
-var region=${REGION} \
-var cluster_prefix=${KUBERNETES_CLUSTER_PREFIX}

...
Apply complete! Resources: 9 added, 0 changed, 0 destroyed.

Outputs:

kubectl_connection_command = "gcloud container clusters get-credentials elasticsearch-cluster --region us-central1"

export GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token)
terraform -chdir=terraform/gke-standard init
terraform -chdir=terraform/gke-standard apply \
-var project_id=${PROJECT_ID} \
-var region=${REGION} \
-var cluster_prefix=${KUBERNETES_CLUSTER_PREFIX}

...
Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:

kubectl_connection_command = "gcloud container clusters get-credentials elasticsearch-cluster --region us-central1"

连接到集群

配置 kubectl 以提取凭据并与新的 GKE 集群通信：

gcloud container clusters get-credentials \
    ${KUBERNETES_CLUSTER_PREFIX}-cluster --region ${REGION}

部署 Elasticsearch 数据库和 StatefulHA Operator

在本部分中，您将使用 ECK Operator Helm 图表将 Elasticsearch 数据库（在集群模式下）和 StatefulHA Operator 部署到 GKE 集群。

Deployment 会创建具有以下配置的 GKE 集群：

• Elasticsearch 节点的三个副本。
• 用于更改虚拟内存设置的 DaemonSet，以实现最佳 Elasticsearch 性能。
• NodeAffinity 和 PodAntiAffinity 配置，以确保在 Kubernetes 节点之间正确分布，从而优化节点池的使用，并最大限度地提高不同可用区的可用性。
• 用于管理故障切换过程并确保高可用性的有状态 HA Operator。
• 对于身份验证，数据库会使用身份验证凭据、密码和证书创建 Kubernetes Secret。

如需使用 Helm 图表部署 Elasticsearch 数据库，请按照以下步骤操作：

1. 启用 StatefulHA 插件：

   Autopilot

   GKE 会在创建集群时自动启用 StatefulHA 插件。

   标准

   运行以下命令：

   gcloud container clusters update ${KUBERNETES_CLUSTER_PREFIX}-cluster \
       --project=${PROJECT_ID} \
       --region=${REGION} \
       --update-addons=StatefulHA=ENABLED
   

   完成此命令并使集群显示就绪状态可能需要 15 分钟。

2. 创建 Elastic Cloud on Kubernetes (ECK) 自定义资源定义 (CRD)：

   kubectl apply -f https://download.elastic.co/downloads/eck/2.11.1/crds.yaml
   

3. 部署 ECK Operator：

   kubectl apply -f https://download.elastic.co/downloads/eck/2.11.1/operator.yaml
   

4. 为数据库创建命名空间 elastic：

   kubectl create ns elastic
   

5. 安装 HighAvailabilityApplication (HAA) 资源，以定义 Elasticsearch 的故障切换规则。

   kubectl apply -n elastic -f manifests/01-regional-pd/ha-app.yaml
   

   ha-app.yaml 清单描述了 HighAvailabilityApplication 资源：

   kind: HighAvailabilityApplication
   apiVersion: ha.gke.io/v1
   metadata:
     name: elasticsearch-ha-es-main
     namespace: elastic
   spec:
     resourceSelection:
       resourceKind: StatefulSet
     policy:
       storageSettings:
         requireRegionalStorage: false
       failoverSettings:
         forceDeleteStrategy: AfterNodeUnreachable
         afterNodeUnreachable:
           afterNodeUnreachableSeconds: 20 # 60 seconds total

6. 应用此清单以创建区域级永久性 SSD 磁盘 StorageClass：

   kubectl apply -n elastic -f manifests/01-regional-pd/regional-pd.yaml
   

   regional-pd.yaml 清单描述了永久性 SSD 磁盘 StorageClass：

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   allowVolumeExpansion: true
   metadata:
     name: ha-regional
   parameters:
     replication-type: regional-pd
     type: pd-ssd
     availability-class: regional-hard-failover
   provisioner: pd.csi.storage.gke.io
   reclaimPolicy: Retain
   volumeBindingMode: WaitForFirstConsumer

7. 部署 DaemonSet 资源以在每个节点中设置虚拟内存：

   kubectl apply -n elastic -f manifests/02-elasticsearch/mmap-count.yaml
   

   mmap-count.yaml 清单描述了 DaemonSet：

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: max-map-count-setter
     labels:
       k8s-app: max-map-count-setter
   spec:
     selector:
       matchLabels:
         name: max-map-count-setter
     template:
       metadata:
         labels:
           name: max-map-count-setter
       spec:
         initContainers:
           - name: max-map-count-setter
             image: docker.io/bash:5.2.21
             resources:
               limits:
                 cpu: 100m
                 memory: 32Mi
             securityContext:
               privileged: true
               runAsUser: 0
             command: ['/usr/local/bin/bash', '-e', '-c', 'echo 262144 > /proc/sys/vm/max_map_count']
         containers:
           - name: sleep
             image: docker.io/bash:5.2.21
             command: ['sleep', 'infinity']

8. 应用此清单以部署 Elasticsearch 集群：

   kubectl apply -n elastic -f manifests/02-elasticsearch/elasticsearch.yaml
   

   elasticsearch.yaml 清单描述了 Deployment：

   apiVersion: elasticsearch.k8s.elastic.co/v1
   kind: Elasticsearch
   metadata:
     name: elasticsearch-ha
   spec:
     version: 8.11.4
     nodeSets:
     - name: main
       count: 3
       volumeClaimTemplates:
       - metadata:
           name: elasticsearch-data 
         spec:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 10Gi
           storageClassName: ha-regional
       config:
       podTemplate:
         metadata:
           labels:
             app.stateful/component: elasticsearch
         spec:
           initContainers:
           - name: max-map-count-check
             command: ['sh', '-c', "while true; do mmc=$(cat /proc/sys/vm/max_map_count); if [ ${mmc} -eq 262144 ]; then exit 0; fi; sleep 1; done"]
           containers:
           - name: metrics
             image: quay.io/prometheuscommunity/elasticsearch-exporter:v1.7.0
             command:
               - /bin/elasticsearch_exporter
               - --es.ssl-skip-verify
               - --es.uri=https://$(ES_USER):$(ES_PASSWORD)@localhost:9200
             securityContext:
               runAsNonRoot: true
               runAsGroup: 10000
               runAsUser: 10000
             resources:
               requests:
                 memory: "128Mi"
                 cpu: "25m"
               limits:
                 memory: "128Mi"
                 cpu: "100m"
             ports:
             - containerPort: 9114
             env:
             - name: ES_USER
               value: "elastic"
             - name: ES_PASSWORD
               valueFrom:
                 secretKeyRef:
                   name: elasticsearch-ha-es-elastic-user
                   key: elastic
           - name: elasticsearch
             resources:
               limits:
                 memory: 4Gi
                 cpu: 1
           affinity:
             nodeAffinity:
               preferredDuringSchedulingIgnoredDuringExecution:
                 - weight: 1
                   preference:
                     matchExpressions:
                     - key: app.stateful/component
                       operator: In
                       values:
                       - elasticsearch
             podAntiAffinity:
               preferredDuringSchedulingIgnoredDuringExecution:
               - weight: 1
                 podAffinityTerm:
                   labelSelector:
                     matchLabels:
                       app.stateful/component: elasticsearch
                   topologyKey: topology.kubernetes.io/zone

   等待几分钟，让 Elasticsearch 集群完全启动。

9. 检查部署状态：

   kubectl get elasticsearch -n elastic --watch
   

   如果 elasticsearch 数据库成功部署，则输出类似于以下内容：

   NAME               HEALTH   NODES   VERSION   PHASE   AGE
   elasticsearch-ha   green    3       8.11.4    Ready   2m30s
   

   等待 HEALTH 显示为 green。如果需要，请按 Ctrl C 退出该命令。

10. 如需检查故障切换规则是否已应用，请描述资源并确认 Status: Message: Application is protected。

    kubectl describe highavailabilityapplication elasticsearch-ha-es-main -n elastic
    

    输出类似于以下内容

    Status:
      Conditions:
        Last Transition Time:  2024-02-01T13:27:50Z
        Message:               Application is protected
        Observed Generation:   1
        Reason:                ApplicationProtected
        Status:                True
        Type:                  Protected
    Events:                    <none>
    

11. GKE 启动工作负载后，验证 GKE 是否已创建 Elasticsearch 工作负载：

    kubectl get pod,svc,statefulset,pdb,secret,daemonset -n elastic
    

    输出类似于以下内容：

    NAME                             READY   STATUS    RESTARTS   AGE
    pod/elasticsearch-ha-es-main-0   2/2     Running   0          7m16s
    pod/elasticsearch-ha-es-main-1   2/2     Running   0          7m16s
    pod/elasticsearch-ha-es-main-2   2/2     Running   0          7m16s
    pod/max-map-count-setter-28wt9   1/1     Running   0          7m27s
    pod/max-map-count-setter-cflsw   1/1     Running   0          7m27s
    pod/max-map-count-setter-gzq9k   1/1     Running   0          7m27s
    
    NAME                                        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
    service/elasticsearch-ha-es-http            ClusterIP   10.52.8.28   <none>        9200/TCP   7m18s
    service/elasticsearch-ha-es-internal-http   ClusterIP   10.52.3.48   <none>        9200/TCP   7m18s
    service/elasticsearch-ha-es-main            ClusterIP   None         <none>        9200/TCP   7m16s
    service/elasticsearch-ha-es-transport       ClusterIP   None         <none>        9300/TCP   7m18s
    
    NAME                                        READY   AGE
    statefulset.apps/elasticsearch-ha-es-main   3/3     7m16s
    
    NAME                                                     MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
    poddisruptionbudget.policy/elasticsearch-ha-es-default   2               N/A               1                     7m16s
    
    NAME                                                 TYPE     DATA   AGE
    secret/elasticsearch-ha-es-elastic-user              Opaque   1      7m18s
    secret/elasticsearch-ha-es-file-settings             Opaque   1      7m16s
    secret/elasticsearch-ha-es-http-ca-internal          Opaque   2      7m17s
    secret/elasticsearch-ha-es-http-certs-internal       Opaque   3      7m17s
    secret/elasticsearch-ha-es-http-certs-public         Opaque   2      7m17s
    secret/elasticsearch-ha-es-internal-users            Opaque   4      7m18s
    secret/elasticsearch-ha-es-main-es-config            Opaque   1      7m16s
    secret/elasticsearch-ha-es-main-es-transport-certs   Opaque   7      7m16s
    secret/elasticsearch-ha-es-remote-ca                 Opaque   1      7m16s
    secret/elasticsearch-ha-es-transport-ca-internal     Opaque   2      7m16s
    secret/elasticsearch-ha-es-transport-certs-public    Opaque   1      7m16s
    secret/elasticsearch-ha-es-xpack-file-realm          Opaque   4      7m18s
    
    NAME                                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/max-map-count-setter   6         6         6       6            6           <none>          13m
    

系统会为 Elasticsearch 集群创建以下 GKE 资源：

• 控制三个 Pod 副本的 Elasticsearch StatefulSet。
• 用于配置虚拟内存设置的 DaemonSet。
• 用于连接到 Elasticsearch 的服务。
• 具有超级用户凭据和服务相关证书的 Secret。
• 有状态 HA Operator Pod 和 HighlyAvailableApplication 资源，主动监控 Elasticsearch 应用。

使用 Jupyter 笔记本上传演示数据集并运行搜索查询

在本部分中，您需要将矢量上传到 Elasticsearch 文档，并使用官方 Elasticsearch Python 客户端执行语义搜索查询。Elasticsearch 中的文档由各种字段组成，每个字段与其对应的值配对。为了有效利用 Elasticsearch，我们建议您将数据结构设计成这些文档，然后将它们编入索引以供搜索。

在此示例中，您将使用包含不同类型图书列表的 CSV 文件中的数据集。Elasticsearch 充当搜索引擎，您创建的 Pod 充当查询 Elasticsearch 数据库的客户端。

1. 创建 books-dataset 和 notebook ConfigMap，并运行 Jupyter Pod 以与 Elasticsearch 集群进行交互：

   kubectl create -n elastic configmap books-dataset --from-file=manifests/03-notebook/dataset.csv
   kubectl create -n elastic configmap notebook --from-file=manifests/03-notebook/vector-database.ipynb
   kubectl apply -n elastic -f manifests/03-notebook/jupyter.yaml
   

   • 之前创建名为 elasticsearch-ha-es-elastic-user 的 Secret 会作为名为 PW 的环境变量装载到客户端 Pod 上。
   • books-dataset ConfigMap 包含一个 csv 文件，其中包含用于 Elasticsearch 索引的图书数据。
   • notebook ConfigMap 包含用于从 books-dataset 创建 Elasticsearch 索引的 Jupyter 笔记本。

   jupyter.yaml 清单描述了 notebook Deployment 及其 Service：

   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels: &labels
       app: jupyter-notebook
     name: notebook
   spec:
     ports:
     - port: 8888
     selector: *labels
     type: LoadBalancer
     # type: ClusterIP
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: notebook
     labels: &labels
       app: jupyter-notebook
   spec:
     selector:
       matchLabels: *labels
     template:
       metadata: 
         labels: *labels
       spec:
         containers:
         - name: jupyter
           image: tensorflow/tensorflow:2.15.0-jupyter
           resources:
             requests:
               memory: "4500Mi"
               cpu: "1"
             limits:
               memory: "4500Mi"
               cpu: "1"
           ports:
           - containerPort: 8888
           env:
           - name: PW
             valueFrom:
               secretKeyRef:
                 name: elasticsearch-ha-es-elastic-user
                 key: elastic
           volumeMounts:
           - name: books-dataset
             mountPath: /usr/local/dataset
           - name: notebook
             mountPath: /tf
           - name: elastic-cert
             mountPath: /usr/local/cert
         volumes:
         - name: books-dataset
           configMap:
             name: books-dataset
         - name: notebook
           configMap:
             name: notebook
         - name: elastic-cert
           secret:
             secretName: elasticsearch-ha-es-http-certs-public

2. 等待 GKE 启动 Jupyter Pod：

   kubectl wait pods -l app=jupyter-notebook --for condition=Ready --timeout=300s -n elastic
   

3. 获取包含用于连接到 Jupyter 的访问令牌的网址：

   export EXTERNAL_IP=$(kubectl -n elastic get svc notebook --output jsonpath='{.status.loadBalancer.ingress[0].ip}')
   kubectl logs deploy/notebook -n elastic| grep '^ .*http://127'|sed "s|127.0.0.1|${EXTERNAL_IP}|"
   

4. 打开此网址，然后点击 vector-database.ipynb 文件。

5. 点击运行 > Run all cells。Jupyter 会执行代码并对文本 drama about people and unhappy love 执行搜索查询。

   此查询会对 Elasticsearch 中的 books 索引执行语义搜索，检索最多两个具有与您的查询相关最高匹配得分的结果。

   输出类似于以下内容：

   Title: Romeo and Juliet, Author: William Shakespeare, Paul Werstine (Editor),
   Barbara A. Mowat (Editor), Paavo Emil Cajander (Translator), score: 1.8473973
   In Romeo and Juliet, Shakespeare creates a violent world, in which two young people
   fall in love. It is not simply that their families disapprove; the Montagues and
   the Capulets are engaged in a blood feud.In this death-filled setting, the movement
   from love at first sight to the lovers' final union in death seems almost inevitable.
   And yet, this play set in an extraordinary world has become the quintessential
   story of young love. In part because of its exquisite language, it is easy to
   respond as if it were about all young lovers.
   ---------
   Title: A Midsummer Night's Dream, Author: William Shakespeare, Paul Werstine
   (Editor), Barbara A. Mowat (Editor), Catherine Belsey (Contributor), score: 1.8415744
   Shakespeare's intertwined love polygons begin to get complicated from the
   start--Demetrius and Lysander both want Hermia but she only has eyes for Lysander.
   Bad news is, Hermia's father wants Demetrius for a son-in-law. On the outside
   is Helena, whose unreturned love burns hot for Demetrius. Hermia and Lysander
   plan to flee from the city under cover of darkness but are pursued by an enraged
   Demetrius (who is himself pursued by an enraptured Helena). In the forest, unbeknownst
   to the mortals, Oberon and Titania (King and Queen of the faeries) are having a
   spat over a servant boy. The plot twists up when Oberon's head mischief-maker,
   Puck, runs loose with a flower which causes people to fall in love with the first
   thing they see upon waking. Throw in a group of labourers preparing a play for
   the Duke's wedding (one of whom is given a donkey's head and Titania for a lover
   by Puck) and the complications become fantastically funny.
   ---------
   

查看集群的 Prometheus 指标

GKE 集群配置了 Google Cloud Managed Service for Prometheus，可启用 Prometheus 格式的指标收集。该服务提供了用于监控和提醒的全托管式解决方案，支持收集、存储和分析来自集群及其应用的指标。

下图展示了 Prometheus 如何收集集群的指标：

图中的 GKE 专用集群包含以下组件：

• 公开路径 / 和端口 9114 上指标的 Elasticsearch Pod。这些指标由名为 metrics 的边车容器（包含 elasticsearch_exporter）提供。
• 基于 Prometheus 的收集器，用于处理来自 Elasticsearch Pod 的指标。
• 将指标发送到 Cloud Monitoring 的 PodMonitoring 资源。

集群配置使用 Prometheus 格式的指标导出器定义边车容器：

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-ha
spec:
  ...
  nodeSets:
  - name: main
    ...
    podTemplate:
      spec:
        containers:
        ...
        - name: metrics
          image: quay.io/prometheuscommunity/elasticsearch-exporter:v1.7.0
          command:
          - /bin/elasticsearch_exporter
          - --es.ssl-skip-verify
          - --es.uri=https://$(ES_USER):$(ES_PASSWORD)@localhost:9200
          ...
          env:
          - name: ES_USER
            value: "elastic"
          - name: ES_PASSWORD
            valueFrom:
            secretKeyRef:
              name: elasticsearch-ha-es-elastic-user
              key: elastic

如需导出和查看指标，请按照以下步骤操作：

1. 创建 PodMonitoring 资源，以按 labelSelector 爬取指标：

   kubectl apply -n elastic -f manifests/04-prometheus-metrics/pod-monitoring.yaml
   

   pod-monitoring.yaml 清单描述了 PodMonitoring 资源：

   apiVersion: monitoring.googleapis.com/v1
   kind: PodMonitoring
   metadata:
     name: elasticsearch
   spec:
     selector:
       matchLabels:
         app.stateful/component: elasticsearch
         elasticsearch.k8s.elastic.co/cluster-name: elasticsearch-ha
     endpoints:
     - port: 9114
       interval: 30s
       path: /metrics

   几分钟后，系统会显示内置信息中心“Elasticsearch Prometheus 概览”。

2. 如需查看更多数据相关图表，请导入包含 dashboard.json 中定义的配置的自定义 Cloud Monitoring 信息中心：

   gcloud --project "${PROJECT_ID}" monitoring dashboards create --config-from-file monitoring/dashboard.json
   

3. 命令成功运行后，前往 Cloud Monitoring 信息中心：

   前往信息中心概览

4. 从信息中心列表中，打开 ElasticSearch Overview 信息中心。收集和显示指标可能需要 1-2 分钟时间。

   信息中心会显示关键指标的计数：

   • 索引
   • 文档和分片
   • 待处理的操作
   • 正在运行的节点及其健康状况

备份集群配置

借助 Backup for GKE 功能，您可以安排对整个 GKE 集群配置（包括已部署的工作负载及其数据）的定期备份。

在本教程中，您将为 GKE 集群配置备份方案，以在每天凌晨 3 点对所有工作负载（包括 Secret 和卷）执行备份。为确保高效存储管理，三天前的备份会被自动删除。

1. 为集群启用 Backup for GKE 功能：

   gcloud container clusters update ${KUBERNETES_CLUSTER_PREFIX}-cluster \
       --project=${PROJECT_ID} \
       --region=${REGION} \
       --update-addons=BackupRestore=ENABLED
   

2. 为集群中的所有命名空间创建具有每日时间表的备份方案：

   gcloud beta container backup-restore backup-plans create ${KUBERNETES_CLUSTER_PREFIX}-cluster-backup \
       --project=${PROJECT_ID} \
       --location=${REGION} \
       --cluster="projects/${PROJECT_ID}/\locations/${REGION}/\clusters/${KUBERNETES_CLUSTER_PREFIX}-cluster" \
       --all-namespaces \
       --include-secrets \
       --include-volume-data \
       --cron-schedule="0 3 * * *" \
       --backup-retain-days=3
   

   该命令在运行时使用相关的环境变量。

   集群名称的格式相对于您的项目和区域，如下所示：

   projects/PROJECT_ID/locations/REGION/clusters/CLUSTER_NAME
   

   出现提示时，请输入 y.。输出类似于以下内容：

   Create request issued for: [elasticsearch-cluster-backup]
   Waiting for operation [projects/PROJECT_ID/locations/us-central1/operations/operation-1706528750815-610142ffdc9ac-71be4a05-f61c99fc] to complete...⠹
   

   此操作可能需要几分钟才能成功完成。执行完成后，输出类似于以下内容：

   Created backup plan [elasticsearch-cluster-backup].
   

3. 您可以看到 Backup for GKE 控制台中列出新创建的备份方案 elasticsearch-cluster-backup。

   前往 Backup for GKE

如果您要恢复已保存的备份配置，请参阅恢复备份。