k8s部署es和skywalking-Toy模板网

这篇具有很好参考价值的文章主要介绍了k8s部署es和skywalking。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

使用k8s部署es和skywalking

skywalking介绍

skywalking架构

k8s部署es和skywalking,kubernetes,skywalking,云原生,运维

整个架构，分成上、下、左、右四部分：

上部分
Agent ：负责从应用中，收集链路信息，发送给 SkyWalking OAP 服务器。目前支持 SkyWalking、Zikpin、Jaeger 等提供的 Tracing 数据信息。而我们目前采用的是，SkyWalking Agent 收集 SkyWalking Tracing 数据，传递给服务器
下部分
SkyWalking OAP ：负责接收 Agent 发送的 Tracing 数据信息，然后进行分析(Analysis Core) ，存储到外部存储器( Storage )，最终提供查询( Query )功能
右部分
Storage ：Tracing 数据存储。目前支持 ES、MySQL、Sharding Sphere、TiDB、H2 多种存储器。而我们目前采用的是 ES ，主要考虑是 SkyWalking 开发团队自己的生产环境采用 ES 为主
左部分
SkyWalking UI ：负责提供控台，查看链路等等
部署elasticsearch
skywalking的持久化选择的是elasticsearch，请大家根据自己自己的情况来选择持久化的方式。

部署storageclass

我不太喜欢使用helm一键部署的方式来进行部署服务，helm的方式有点像黑盒。我喜欢使用yaml文件一个服务一个服务来部署。下面是部署的yaml 文件。

storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storage
  namespace: skywalking 
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"
reclaimPolicy: Retain

部署nfs-provisioner

由于在k8s 1.20以上版本弃用了selfLink，我这里就不用nfs-client-provisioner镜像了，使用dyrnq/nfs-subdir-external-provisioner。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: dyrnq/nfs-subdir-external-provisioner:v4.0.2
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 192.168.5.166
            - name: NFS_PATH
              value: /mnt/jf/skywalking
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.5.166
            path: /mnt/jf/skywalking
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: skywalking
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

部署elasticsearch

由于es属于有状态的服务，我这里选择使用statefulset方式来部署
es-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es
  namespace: skywalking
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels: 
        app: elasticsearch
    spec:
      imagePullSecrets:
        - name: harborsecret
      initContainers:
      - name: increase-vm-max-map
        image: busybox:latest
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox:latest
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        image: elasticsearch:7.17.14
        ports:
        - name: rest
          containerPort: 9200
        - name: inter
          containerPort: 9300
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 1000m
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
        - name: cluster.name
          value: k8s-logs
        - name: node.name
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: cluster.initial_master_nodes
          value: "es-0,es-1,es-2"
        - name: discovery.seed_hosts
          value: "elasticsearch"
        - name: ES_JAVA_OPTS
          value: "-Xms512m -Xmx512m"
        - name: network.host
          value: "0.0.0.0"
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: nfs-storage
      resources:
        requests:
          storage: 300Gi 
          
service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: skywalking
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node
---
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-client
  namespace: skywalking
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

部署完es以后，通过访问IP:9200进行访问。测试es是否部署成功。
在浏览器访问出现如下界面，证书部署成功：
k8s部署es和skywalking,kubernetes,skywalking,云原生,运维

部署skywalking

部署RBAC

RBAC.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: skywalking
  name: skywalking-oap
  namespace: skywalking
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: skywalking
  namespace: skywalking  
  labels:
    app: skywalking
rules:
  - apiGroups: [""]
    resources: ["pods", "endpoints", "services", "nodes"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["extensions"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: skywalking
  namespace: skywalking  
  labels:
    app: skywalking
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: skywalking
subjects:
  - kind: ServiceAccount
    name: skywalking-oap
    namespace: skywalking

部署数据初始化Job

apiVersion: batch/v1
kind: Job
metadata:
  name: "skywalking-es-init"
  namespace: skywalking  
  labels:
    app: skywalking-job
spec:
  template:
    metadata:
      name: "skywalking-es-init"
      labels:
        app: skywalking-job
    spec:
      serviceAccountName: skywalking-oap
      restartPolicy: Never
      initContainers:
      - name: wait-for-elasticsearch
        image: busybox:1.30
        imagePullPolicy: IfNotPresent
        command: ['sh', '-c', 'for i in $(seq 1 60); do nc -z -w3 elasticsearch 9200 && exit 0 || sleep 5; done; exit 1']
      containers:
      - name: oap
        image: skywalking.docker.scarf.sh/apache/skywalking-oap-server:8.9.0
        imagePullPolicy: IfNotPresent
        env:
        - name: JAVA_OPTS
          value: "-Xmx2g -Xms2g -Dmode=init"
        - name: SW_STORAGE
          value: elasticsearch
        - name: SW_STORAGE_ES_CLUSTER_NODES
          value: "elasticsearch:9200"
        volumeMounts:
      volumes:

部署OAP

oap.yaml

apiVersion: v1
kind: Service
metadata:
  name: oap-svc
  namespace: skywalking  
  labels:
    app: oap
spec:
  type: ClusterIP
  ports:
  - port: 11800
    name: grpc
  - port: 12800
    name: rest
  selector:
    app: oap
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: oap
  name: oap
  namespace: skywalking  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oap
  template:
    metadata:
      labels:
        app: oap
    spec:
      serviceAccountName: skywalking-oap
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app: "skywalking"
      initContainers:
      - name: wait-for-elasticsearch
        image: busybox:1.30
        imagePullPolicy: IfNotPresent
        command: ['sh', '-c', 'for i in $(seq 1 60); do nc -z -w3 elasticsearch 9200 && exit 0 || sleep 5; done; exit 1']
      containers:
      - name: oap
        image: skywalking.docker.scarf.sh/apache/skywalking-oap-server:8.9.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          tcpSocket:
            port: 12800
          initialDelaySeconds: 15
          periodSeconds: 20
        readinessProbe:
          tcpSocket:
            port: 12800
          initialDelaySeconds: 15
          periodSeconds: 20
        ports:
        - containerPort: 11800
          name: grpc
        - containerPort: 12800
          name: rest
        env:
        - name: JAVA_OPTS
          value: "-Dmode=no-init -Xmx2g -Xms2g"
        - name: SW_CLUSTER
          value: kubernetes
        - name: SW_CLUSTER_K8S_NAMESPACE
          value: "skywalking"
        - name: SW_CLUSTER_K8S_LABEL
          value: "app=skywalking,release=skywalking,component=oap"
        # 记录数据
        - name: SW_CORE_RECORD_DATA_TTL
          value: "2"
        # Metrics数据  
        - name: SW_CORE_METRICS_DATA_TTL
          value: "2"
        - name: SKYWALKING_COLLECTOR_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
        - name: SW_STORAGE
          value: elasticsearch
        - name: SW_STORAGE_ES_CLUSTER_NODES
          value: "elasticsearch:9200"

部署UI

skywalking的UI界面
ui.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: ui
  name: ui-svc
  namespace: skywalking  
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  selector:
    app: ui
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ui
  namespace: skywalking  
  labels:
    app: ui
spec:
  replicas: 1
  selector:
    matchLabels:
        app: ui
  template:
    metadata:
      labels:
        app: ui
    spec:
      affinity:
      containers:
      - name: ui
        image: skywalking.docker.scarf.sh/apache/skywalking-ui:8.9.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: page
        env:
        - name: SW_OAP_ADDRESS
          value: http://elasticsearch:12800  #根据oap的svc一致

查看服务

测试部署项目接入agent

编写entrypoint.sh

在jvm启动参数中添加skywalking相关的启动参数

java -javaagent:/skywalking-agent/skywalking-agent.jar \
-Dskywalking.agent.service_name=${appName} \
-Dskywalking.collector.backend_service=skywalking的service名称:11800

添加java的agent

根据你后端语言选择不同的agent进行下载，下载地址，我这边使用的是java agent

下载java-agnet

可以选择将agent下载到指定的服务器

wget https://archive.apache.org/dist/skywalking/java-agent/8.10.0/apache-skywalking-java-agent-8.10.0.tgz
tar xf apache-skywalking-java-agent-8.10.0.tgz 
ls
apache-skywalking-java-agent-8.10.0.tgz  skywalking-agent

我这直接将java的agent添加到dockerfile中。

# 添加以下指令将SkyWalking Agent文件拷贝到容器内
ADD https://archive.apache.org/dist/skywalking/java-agent/8.10.0/apache-skywalking-java-agent-8.10.0.tgz $PROJECT_ROOT/conf/skywalking-agent.tgz
# 解压Agent文件
RUN tar -xzf $PROJECT_ROOT/conf/skywalking-agent.tgz -C $PROJECT_ROOT/conf/

调用链忽略（取消跟踪）

cd skywalking-agent
cp optional-plugins/apm-trace-ignore-plugin-8.10.0.jar plugins/
vim config/apm-trace-ignore-plugin.config 
#添加下面这段
trace.ignore_path=${SW_AGENT_TRACE_IGNORE_PATH:GET:/actuator/**,Redisson/**,Mysql/**,HikariCP/**,Lettuce/**,/xxl-job/**,UndertowDispatch/**}

将服务部署到k8s集群上

根据自己对应的服务进行编写yaml文件

验证

部署完自己的微服务以后，就可以进行验证了，登录到swkwalking的UI界面，查看是否有自己部署的服务。
k8s部署es和skywalking,kubernetes,skywalking,云原生,运维文章来源地址https://www.toymoban.com/news/detail-768457.html