前言
在前一篇文章中简单地介绍了《5分钟通过docker快速部署并使用apollo配置中心》,用户可以快速地了解到配置中心apollo的基本使用,但是在生产环境中我们往往需要高可用的部署配置中心,这样我们就得有k8s来进行部署。在apollo的官方文档中有关于k8s部署的文章(https://github.com/ctripcorp/apollo/tree/master/scripts/apollo-on-kubernetes),但是在执行的过程中是会遇到一定的坑的。包括实际部署中遇到的,镜像的制作、portal服务的多实例支持、ingress的创建都没有说到,本人在部署的过程中已经全部解决以上问题,希望可以帮到没有部署过的用户。
部署成功页面
下面是部署完成后,访问apollo的登录页面
登录页面
输入用户名密码:apollo/admin,部署环境完成portal页面图
部署完成apollo后页面
k8s的dashboard部署页面,本文部署了dev, fat, pro三个环境。
部署环境完成k8s后dashboard图
部署过程
本文在部署的时候使用了当前最新的apollo版本为:1.7.1,所以下面的所有构建也是基于当前版本的。
一、构建镜像
首先从git上下载源码,可以从github下载:https://github.com/ctripcorp/apollo;也可以从gitee下载:https://gitee.com/nobodyiam/apollo,国内的会快一点。然后进入到目录
- /scripts/apollo-on-kubernetes
去构建镜像。
1、 直接使用编译的的包进行安装,获取 apollo 压缩包
可以直接从官网下载,因为github实在是太慢了。建议直接从我的百度云下载。
A、下载比较慢,直接用我百度云
- 链接:https://pan.baidu.com/s/1eLL2ocYE1uzXcvzO2Y3dNg
- 提取码:nfvm
B、从 https://github.com/ctripcorp/apollo/releases 下载预先打好的 java 包
- (1)进入scripts/apollo-on-kubernetes/
- 执行 wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-portal-1.7.1-github.zip
- (2)进入scripts/apollo-on-kubernetes/
- 执行 wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-adminservice-1.7.1-github.zip
- (3)进入scripts/apollo-on-kubernetes/
- 执行 wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-configservice-1.7.1-github.zip
2、解压压缩包, 获取程序 jar 包
不要忘记重命名,把版本号去掉。
- 解压 apollo-portal-1.7.1-github.zip
- 获取 apollo-portal-1.7.1.jar, 重命名为 apollo-portal.jar, 放到 scripts/apollo-on-kubernetes/apollo-portal-server
- 解压 apollo-adminservice-1.7.1-github.zip
- 获取 apollo-adminservice-1.7.1.jar, 重命名为 apollo-adminservice.jar, 放到 scripts/apollo-on-kubernetes/apollo-admin-server
- 解压 apollo-configservice-1.7.1-github.zip
- 获取 apollo-configservice-1.7.1.jar, 重命名为 apollo-configservice.jar, 放到 scripts/apollo-on-kubernetes/apollo-config-server
3、构建镜像
注意:因为许多地方都要同时改,在构建的时候要确定命名空间,我使用zizai。
要构建如下的镜像:alpine-bash-3.8-image,apollo-config-server,apollo-admin-server和apollo-portal-server,对应的镜像文件,在对应的目录下:
构建镜像要去到对应的Dockerfile同级目录下去执行。
例如,去到scripts/apollo-on-kubernetes/apollo-config-server下执行:
- docker build -t apollo-config-server:v1.7.1 .
注意,总共要构建4个镜像。整体的思路是:先构建镜像,然后打tag,再推到仓库里去。
在对应目录下,总结整体脚本如下:
- alpine-bash-3.8-image的镜像:
- docker build -t alpine-bash:3.8 .
- docker tag alpine-bash:3.8 hub.thinkinpower.net/zizai/alpine-bash:3.8
- docker push hub.thinkinpower.net/zizai/alpine-bash:3.8
- apollo对应的镜像:
- docker build -t apollo-config-server:v1.7.1 .
- docker tag apollo-config-server:v1.7.1 hub.xx.net/zizai/apollo-config-server:v1.7.1
- docker push hub.xx.net/zizai/apollo-config-server:v1.7.1
- docker build -t apollo-admin-server:v1.7.1 .
- docker tag apollo-admin-server:v1.7.1 hub.xx.net/zizai/apollo-admin-server:v1.7.1
- docker push hub.xx.net/zizai/apollo-admin-server:v1.7.1
- docker build -t apollo-portal-server:v1.7.1 .
- docker tag apollo-portal-server:v1.7.1 hub.thinkinpower.net/zizai/apollo-portal-server:v1.7.1
- docker push hub.thinkinpower.net/zizai/apollo-portal-server:v1.7.1
二、部署apollo到kubernetes
1、创建数据库脚本
说明一下:
- 在实际的生产环境使用中,通过分布式存储来实现的磁盘在mysql这种IO密集性应用中,性能问题会显得非常突出。所以在实际应用中,一般不会把mysql这种应用直接放入kubernetes中管理,而是使用专用的服务器来独立部署。而像web这种无状态应用依然会运行在kubernetes当中,这个时候web服务器要连接kubernetes管理之外的数据库,有两种方式:一是直接连接数据库所在物理服务器IP,另一种方式就是借助kubernetes的Endpoints直接将外部服务器映射为kubernetes内部的一个服务。
我们使用外面的mysql作为数据库,不会将mysql部署到k8s里。
执行目录scripts/apollo-on-kubernetes/db下的脚本。Apollo服务端共需要两个数据库:ApolloPortalDB和ApolloConfigDB。每一个配置的config一个数据库脚本,portal一个数据库脚本。数据库脚本见:https://github.com/ctripcorp/apollo/tree/master/scripts/apollo-on-kubernetes/db,在git里已经有。如果apollo 开启了 4 个环境, 即 dev、test-alpha、test-beta、prod, 在MySQL 中导入 scripts/apollo-on-kubernetes/db 下的文件。
2、部署k8s的yaml文件
官网的yaml可以下载修改就可以了,因为我用自己的仓库的镜像,并且多次测试,我主要有如下的修改:
(1)配置文件要删除掉安全提示:
- securityContext:
- privileged: true
(2)添加仓库的密钥:
- imagePullSecrets:
- - name: registry-harbor
(3)下面修改为每次都拉镜像:Always
- imagePullPolicy: Always
(4)添加mysql的配置信息
我只用3个环境,需要修改的文件如图:
因为修改得比较多,我将在下面列出每一个文件。我只拿开发环境apollo-env-dev的作为一个示例,其它的只是对应修改就可以了。在执行的时候,建议大家从下面的(3)、(2)、(1)的顺序执行下面的文件。
(1)、service-apollo-admin-server-dev.yaml
- ---
- # configmap for apollo-admin-server-dev
- kind: ConfigMap
- apiVersion: v1
- metadata:
- namespace: zizai
- name: configmap-apollo-admin-server-dev
- data:
- application-github.properties: |
- spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-dev-env.zizai:3306/DevApolloConfigDB?characterEncoding=utf8
- spring.datasource.username = admin
- spring.datasource.password = mysql-admin
- eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:8080/eureka/
- ---
- kind: Service
- apiVersion: v1
- metadata:
- namespace: zizai
- name: service-apollo-admin-server-dev
- labels:
- app: service-apollo-admin-server-dev
- spec:
- ports:
- - protocol: TCP
- port: 8090
- targetPort: 8090
- selector:
- app: pod-apollo-admin-server-dev
- type: ClusterIP
- sessionAffinity: ClientIP
- ---
- kind: Deployment
- apiVersion: apps/v1
- metadata:
- namespace: zizai
- name: deployment-apollo-admin-server-dev
- labels:
- app: deployment-apollo-admin-server-dev
- spec:
- replicas: 3
- selector:
- matchLabels:
- app: pod-apollo-admin-server-dev
- strategy:
- rollingUpdate:
- maxSurge: 1
- maxUnavailable: 1
- type: RollingUpdate
- template:
- metadata:
- labels:
- app: pod-apollo-admin-server-dev
- spec:
- imagePullSecrets: # dokcer仓库密码,不需要的可以去掉
- - name: registry-harbor
- affinity:
- podAntiAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - weight: 100
- podAffinityTerm:
- labelSelector:
- matchExpressions:
- - key: app
- operator: In
- values:
- - pod-apollo-admin-server-dev
- topologyKey: kubernetes.io/hostname
- volumes:
- - name: volume-configmap-apollo-admin-server-dev
- configMap:
- name: configmap-apollo-admin-server-dev
- items:
- - key: application-github.properties
- path: application-github.properties
- initContainers:
- - image: hub.thinkinpower.net/zizai/alpine-bash:3.8
- imagePullPolicy: Always
- name: check-service-apollo-config-server-dev
- command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-config-server-dev.zizai:8080"]
- containers:
- - image: hub.thinkinpower.net/zizai/apollo-admin-server:v1.7.1
- imagePullPolicy: Always
- name: container-apollo-admin-server-dev
- ports:
- - protocol: TCP
- containerPort: 8090
- volumeMounts:
- - name: volume-configmap-apollo-admin-server-dev
- mountPath: /apollo-admin-server/config/application-github.properties
- subPath: application-github.properties
- env:
- - name: APOLLO_ADMIN_SERVICE_NAME
- value: "service-apollo-admin-server-dev.zizai"
- readinessProbe:
- tcpSocket:
- port: 8090
- initialDelaySeconds: 10
- periodSeconds: 5
- livenessProbe:
- tcpSocket:
- port: 8090
- initialDelaySeconds: 120
- periodSeconds: 10
- dnsPolicy: ClusterFirst
- restartPolicy: Always
(2)、service-apollo-config-server-dev.yaml
- ---
- # configmap for apollo-config-server-dev
- kind: ConfigMap
- apiVersion: v1
- metadata:
- namespace: zizai
- name: configmap-apollo-config-server-dev
- data:
- application-github.properties: |
- spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-dev-env.zizai:3306/DevApolloConfigDB?characterEncoding=utf8
- spring.datasource.username = admin
- spring.datasource.password = mysql-admin
- eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:8080/eureka/
- ---
- kind: Service
- apiVersion: v1
- metadata:
- namespace: zizai
- name: service-apollo-meta-server-dev
- labels:
- app: service-apollo-meta-server-dev
- spec:
- ports:
- - protocol: TCP
- port: 8080
- targetPort: 8080
- selector:
- app: pod-apollo-config-server-dev
- type: ClusterIP
- clusterIP: None
- sessionAffinity: ClientIP
- ---
- kind: Service
- apiVersion: v1
- metadata:
- namespace: zizai
- name: service-apollo-config-server-dev
- labels:
- app: service-apollo-config-server-dev
- spec:
- ports:
- - protocol: TCP
- port: 8080
- targetPort: 8080
- nodePort: 30002
- selector:
- app: pod-apollo-config-server-dev
- type: NodePort
- sessionAffinity: ClientIP
- ---
- kind: StatefulSet
- apiVersion: apps/v1
- metadata:
- namespace: zizai
- name: statefulset-apollo-config-server-dev
- labels:
- app: statefulset-apollo-config-server-dev
- spec:
- serviceName: service-apollo-meta-server-dev
- replicas: 3
- selector:
- matchLabels:
- app: pod-apollo-config-server-dev
- updateStrategy:
- type: RollingUpdate
- template:
- metadata:
- labels:
- app: pod-apollo-config-server-dev
- spec:
- imagePullSecrets: # dokcer仓库密码,不需要的可以去掉
- - name: registry-harbor
- affinity:
- podAntiAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - weight: 100
- podAffinityTerm:
- labelSelector:
- matchExpressions:
- - key: app
- operator: In
- values:
- - pod-apollo-config-server-dev
- topologyKey: kubernetes.io/hostname
- volumes:
- - name: volume-configmap-apollo-config-server-dev
- configMap:
- name: configmap-apollo-config-server-dev
- items:
- - key: application-github.properties
- path: application-github.properties
- containers:
- - image: hub.thinkinpower.net/zizai/apollo-config-server:v1.7.1
- imagePullPolicy: Always
- name: container-apollo-config-server-dev
- ports:
- - protocol: TCP
- containerPort: 8080
- volumeMounts:
- - name: volume-configmap-apollo-config-server-dev
- mountPath: /apollo-config-server/config/application-github.properties
- subPath: application-github.properties
- env:
- - name: APOLLO_CONFIG_SERVICE_NAME
- value: "service-apollo-config-server-dev.zizai"
- readinessProbe:
- tcpSocket:
- port: 8080
- initialDelaySeconds: 10
- periodSeconds: 5
- livenessProbe:
- tcpSocket:
- port: 8080
- initialDelaySeconds: 120
- periodSeconds: 10
- dnsPolicy: ClusterFirst
- restartPolicy: Always
(3)、service-mysql-for-apollo-dev-env.yaml
- ---
- # 为外部 mysql 服务设置 service
- kind: Service
- apiVersion: v1
- metadata:
- namespace: zizai
- name: service-mysql-for-apollo-dev-env
- labels:
- app: service-mysql-for-apollo-dev-env
- spec:
- ports:
- - protocol: TCP
- port: 3306
- targetPort: 3306
- type: ClusterIP
- sessionAffinity: None
- ---
- kind: Endpoints
- apiVersion: v1
- metadata:
- namespace: zizai
- name: service-mysql-for-apollo-dev-env
- subsets:
- - addresses:
- - ip: 10.29.254.48
- ports:
- - protocol: TCP
- port: 3306
3、添加Ingress
官网给的示例是用k8s的NodePort来访问,但是在实际中,我们用会用Ingress来访问Portal。
注意:因为我们在部署portal的时候是多实例的,所以Ingress要添加保持会话,要不页面会登录不了,进入不了portal页面。具体为:
- metadata:
- annotations:
- nginx.ingress.kubernetes.io/affinity: "cookie" # 解决会话保持
- nginx.ingress.kubernetes.io/session-cookie-name: "route"
- nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
- nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
ingress的代码示例为如下:
- apiVersion: extensions/v1beta1
- kind: Ingress
- metadata:
- name: zizai-apollo-portal
- namespace: zizai
- annotations:
- nginx.ingress.kubernetes.io/affinity: "cookie" # 解决会话保持
- nginx.ingress.kubernetes.io/session-cookie-name: "route"
- nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
- nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
- spec:
- rules:
- - host: zizai-apollo-portal.test.thinkinpower.net
- http:
- paths:
- - path: /
- backend:
- serviceName: service-apollo-portal-server
- servicePort: 8070
4、配置nginx
添加nginx访问到Ingress里:
nginx配置文件:zizai-apollo-portal.test.thinkinpower.net.conf
- server {
- listen 80;
- server_name zizai-apollo-portal.test.thinkinpower.net;
- access_log /data/logs/nginx/zizai-apollo-portal.test.thinkinpower.net.access.log main;
- error_log /data/logs/nginx/zizai-apollo-portal.test.thinkinpower.net.error.log;
- root /data/webapps/zizai-apollo-portal.test.thinkinpower.net/test/static;
- index index.html index.htm;
- client_max_body_size 50m;
- location / {
- proxy_set_header Host $http_host;
- proxy_set_header X-Real-IP $remote_addr;
- proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
- proxy_pass http://kubernetes; # 指向集群的
- }
- }
这样就可以根据域名:http://zizai-apollo-portal.test.thinkinpower.net 访问portal了。
(1)创建的部署:
(2)创建的有部署副本:
(3)创建的service:
(4)创建的ingress:
(5)创建的配置字典:
三、简单使用
本文章将只有简单的使用,后面会有文章介绍详细的使用,需要的可以在本文留言。
1、创建项目
2、选择一个环境添加变量timeout
3、如果是在添加环境的过程中,刷新页面会有“添加补缺环境”的提示
添加补缺环境
就这样实现了k8s部署高可能的apollo。有建议的可以在评论区留言。谢谢!