前言
首先感谢大佬提供的 open-capacity-platform 开源项目,希望通过这次部署,加深对Kubernetes的应用。下面放上 ocp 作者的gitee地址:https://gitee.com/owenwangwen/open-capacity-platform
环境准备
- Kubernetes集群
- 安装JDK1.8
- 安装Maven
- 安装Git
- 安装Harbor或者使用Docker hub
在master节点上安装JDK1.8
yum install -y java-1.8.0-openjdk-*
|
在master节点上安装Maven
参考文章:https://1335402049.github.io/2020/04/19/Linux%E4%B8%8BMaven%E5%AE%89%E8%A3%85/
在master节点上安装Git
在 master 节点上克隆ocp代码仓库
git clone https://gitee.com/owenwangwen/open-capacity-platform.git
|
构建docker镜像并推送到仓库
修改 open-capacity-platform/pom.xml
# 大概在55 56行 # 指定仓库地址,如果是docker hub 修改为 unix:///var/run/docker.sock <docker.host>unix:///var/run/docker.sock</docker.host> # 指定镜像前缀,修改成你自己docker hub上的账号 <docker.image.prefix>tangweifeng</docker.image.prefix>
|
构建jar包
修改配置
修改eureka配置
修改 open-capacity-platform/register-center/eureka-server/src/main/resources/application-slave0.yml
# service名称为eureka defaultZone: http://cloud-eureka-0.cloud-eureka:1111/eureka,http://cloud-eureka-1.cloud-eureka:1111/eureka,http://cloud-eureka-2.cloud-eureka:1111/eureka prefer-ip-address: false # instance字段下增加 hostname appname hostname: cloud-eureka appname: eureka-server
|
修改auth-server配置
修改open-capacity-platform/oauth-center/auth-server/src/main/resources/bootstrap.yml
defaultZone: http://cloud-eureka-0.cloud-eureka:1111/eureka,http://cloud-eureka-1.cloud-eureka:1111/eureka,http://cloud-eureka-2.cloud-eureka:1111/eureka
|
修改open-capacity-platform/oauth-center/auth-server/src/main/resources/application.yml
# auth-center-mysql 是后面部署的 headless 名称 url: jdbc:mysql://auth-center-mysql:3306/oauth-center?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false # mysql用户名和密码 username: root password: xxxx
# redis配置 # ocp-redis 是后面部署的 headless 名称 host: ocp-redis
|
修改user-center配置
修改open-capacity-platform/business-center/user-center/src/main/resources/bootstrap.yml
defaultZone: http://cloud-eureka-0.cloud-eureka:1111/eureka,http://cloud-eureka-1.cloud-eureka:1111/eureka,http://cloud-eureka-2.cloud-eureka:1111/eureka
|
修改open-capacity-platform/business-center/user-center/src/main/resources/application.yml
# user-center-mysql 是后面部署的 headless 名称 url: jdbc:mysql://user-center-mysql:3306/user-center?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false # mysql用户名和密码 username: root password: xxxx
# redis配置 # ocp-redis 是后面部署的 headless 名称 host: ocp-redis
|
修改log-center配置
修改open-capacity-platform/monitor-center/log-center/src/main/resources/bootstrap.yml
defaultZone: http://cloud-eureka-0.cloud-eureka:1111/eureka,http://cloud-eureka-1.cloud-eureka:1111/eureka,http://cloud-eureka-2.cloud-eureka:1111/eureka
|
修改open-capacity-platform/monitor-center/log-center/src/main/resources/application.yml
# log-center-mysql 是后面部署的 headless 名称 url: jdbc:mysql://log-center-mysql:3306/log-center?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false # mysql用户名和密码 username: root password: xxxx
# redis配置 # ocp-redis 是后面部署的 headless 名称 host: ocp-redis
|
修改api-gateway配置
修改open-capacity-platform/api-gateway/src/main/resources/bootstrap.yml
defaultZone: http://cloud-eureka-0.cloud-eureka:1111/eureka,http://cloud-eureka-1.cloud-eureka:1111/eureka,http://cloud-eureka-2.cloud-eureka:1111/eureka
|
修改open-capacity-platform/api-gateway/src/main/resources/application.yml
url: jdbc:mysql://auth-center-mysql:3306/oauth-center?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false # mysql用户名和密码 username: root password: xxxx
# redis配置 # ocp-redis 是后面部署的 headless 名称 host: ocp-redis
|
打包
cd open-capacity-platform mvn clean package -DskipTests
|
制作镜像
制作eureka镜像
cd open-capacity-platform/register-center/eureka-server mvn docker:build
|
制作auth-server镜像
cd open-capacity-platform/oauth-center/auth-server mvn docker:build
|
制作user-center镜像
cd open-capacity-platform/business-center/user-center mvn docker:build
|
制作log-center镜像
制作镜像的时候会出现 pull access denied for frolvlad/alpine-oraclejdk8, repository does not exist or may require ‘docker login’ 错误
是因为dockerfile里面的 frolvlad/alpine-oraclejdk8 已经找不到了,可以将 open-capacity-platform/monitor-center/log-center/src/main/docker中的第一行 frolvlad/alpine-oraclejdk8替换成openjdk:8-jdk-alpine
cd open-capacity-platform/monitor-center/log-center mvn docker:build
|
制作api-gateway镜像
cd open-capacity-platform/api-gateway mvn docker:build
|
上传镜像至docker hub
# 登录 docker login # 这边的 需要指定前缀 ,镜像使用 docker images 查看,前缀就是在一开始的pom.xml中定义的<docker.image.prefix> docker push tangweifeng/eureka-server docker push tangweifeng/auth-server docker push tangweifeng/user-center docker push tangweifeng/log-center docker push tangweifeng/api-gateway
|
存储准备
在部署微服务之前,考虑mysql、redis等服务数据的存储,我们准备使用nfs storageclass进行动态pv的创建,可以参考这篇文章https://1335402049.github.io/2020/09/16/Kubernetes%E4%B8%AD%E4%BD%BF%E7%94%A8NFS%E7%9A%84StorageClass/
确认相关资源起来

部署ingress-nginx
参考这篇文章:https://1335402049.github.io/2020/09/23/Kubernetes%E9%83%A8%E7%BD%B2ingress-nginx/
部署eureka
应用 eureka-statefulset.yaml
apiVersion: apps/v1 kind: StatefulSet metadata: name: cloud-eureka spec: replicas: 3 selector: matchLabels: app: eureka serviceName: cloud-eureka template: metadata: labels: app: eureka spec: containers: - name: eureka-server image: tangweifeng/eureka-server imagePullPolicy: IfNotPresent ports: - containerPort: 1111
|
应用 eureka-service.yaml
apiVersion: v1 kind: Service metadata: name: cloud-eureka spec: type: ClusterIP ports: - port: 1111 targetPort: 1111 selector: app: eureka
|
应用 eureka-ingress.yaml
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: eureka annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: eureka.twf.com http: paths: - path: / backend: serviceName: cloud-eureka servicePort: 1111
|
配置修改windows hosts文件,然后访问

部署MySQL
部署auth-center-mysql
应用 auth-center-mysql-secret.yaml
apiVersion: v1 kind: Secret metadata: name: auth-center-mysql data: # 使用 echo -n '<密码>' |base64 获取加密后的密码 password: dGFuZzE2MTE= stringData: username: root
|
应用 auth-center-mysql-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: auth-center-mysql data: mysqld.cnf: |- [mysqld] pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = /var/lib/mysql #log-error = /var/log/mysql/error.log # By default we only accept connections from localhost #bind-address = 127.0.0.1 # Disabling symbolic-links is recommended to prevent assorted security risks
|
应用 auth-center-mysql-statefulset.yaml
apiVersion: apps/v1 kind: StatefulSet metadata: name: auth-center-mysql spec: replicas: 1 selector: matchLabels: app: auth-center-mysql serviceName: auth-center-mysql volumeClaimTemplates: - metadata: name: auth-center-mysql-data spec: storageClassName: managed-nfs-storage accessModes: - ReadWriteMany resources: requests: storage: 500Mi template: metadata: labels: app: auth-center-mysql spec: volumes: - name: auth-center-mysql-conf configMap: name: auth-center-mysql containers: - name: auth-center-mysql image: mysql:5.7.26 imagePullPolicy: IfNotPresent ports: - containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: auth-center-mysql key: password volumeMounts: - name: auth-center-mysql-data mountPath: /var/lib/mysql - name: auth-center-mysql-conf mountPath: /etc/mysql/mysql.conf.d/
|
应用 auth-center-mysql-service.yaml
我们先使用NodePort方式暴露,然后导入数据后,之后你也可以修改为headless
apiVersion: v1 kind: Service metadata: name: auth-center-mysql spec: type: NodePort selector: app: auth-center-mysql ports: - port: 3306 targetPort: 3306
|
在 auth-center-mysql 起来后,使用navicat或者其他工具连接mysql,将 open-capacity-platform/sql/02.oauth-center.sql导入数据库

修改并应用 auth-center-mysql-service.yaml
apiVersion: v1 kind: Service metadata: name: auth-center-mysql spec: clusterIP: None selector: app: auth-center-mysql ports: - port: 3306 targetPort: 3306
|
部署log-center-mysql
应用 log-center-mysql-secret.yaml
apiVersion: v1 kind: Secret metadata: name: log-center-mysql data: # 使用 echo -n '<密码>' |base64 获取加密后的密码 password: dGFuZzE2MTE= stringData: username: root
|
应用 log-center-mysql-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: log-center-mysql data: mysqld.cnf: |- [mysqld] pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = /var/lib/mysql #log-error = /var/log/mysql/error.log # By default we only accept connections from localhost #bind-address = 127.0.0.1 # Disabling symbolic-links is recommended to prevent assorted security risks
|
应用 log-center-mysql-statefulset.yaml
apiVersion: apps/v1 kind: StatefulSet metadata: name: log-center-mysql spec: replicas: 1 selector: matchLabels: app: log-center-mysql serviceName: log-center-mysql volumeClaimTemplates: - metadata: name: log-center-mysql-data spec: storageClassName: managed-nfs-storage accessModes: - ReadWriteMany resources: requests: storage: 500Mi template: metadata: labels: app: log-center-mysql spec: volumes: - name: log-center-mysql-conf configMap: name: log-center-mysql containers: - name: log-center-mysql image: mysql:5.7.26 imagePullPolicy: IfNotPresent ports: - containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: log-center-mysql key: password volumeMounts: - name: log-center-mysql-data mountPath: /var/lib/mysql - name: log-center-mysql-conf mountPath: /etc/mysql/mysql.conf.d/
|
应用 log-center-mysql-service.yaml
我们先使用NodePort方式暴露,然后导入数据后,之后你也可以修改为headless
apiVersion: v1 kind: Service metadata: name: log-center-mysql spec: type: NodePort selector: app: log-center-mysql ports: - port: 3306 targetPort: 3306
|
在 log-center-mysql 起来后,使用navicat或者其他工具连接mysql,将 open-capacity-platform/sql/05.log-center.sql导入数据库

修改并应用 log-center-mysql-service.yaml
apiVersion: v1 kind: Service metadata: name: log-center-mysql spec: clusterIP: None selector: app: log-center-mysql ports: - port: 3306 targetPort: 3306
|
部署user-center-mysql
应用 user-center-mysql-secret.yaml
apiVersion: v1 kind: Secret metadata: name: user-center-mysql data: # 使用 echo -n '<密码>' |base64 获取加密后的密码 password: dGFuZzE2MTE= stringData: username: root
|
应用 user-center-mysql-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: user-center-mysql data: mysqld.cnf: |- [mysqld] pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = /var/lib/mysql #log-error = /var/log/mysql/error.log # By default we only accept connections from localhost #bind-address = 127.0.0.1 # Disabling symbolic-links is recommended to prevent assorted security risks
|
应用 user-center-mysql-statefulset.yaml
apiVersion: apps/v1 kind: StatefulSet metadata: name: user-center-mysql spec: replicas: 1 selector: matchLabels: app: user-center-mysql serviceName: user-center-mysql volumeClaimTemplates: - metadata: name: user-center-mysql-data spec: storageClassName: managed-nfs-storage accessModes: - ReadWriteMany resources: requests: storage: 500Mi template: metadata: labels: app: user-center-mysql spec: volumes: - name: user-center-mysql-conf configMap: name: user-center-mysql containers: - name: user-center-mysql image: mysql:5.7.26 imagePullPolicy: IfNotPresent ports: - containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: user-center-mysql key: password volumeMounts: - name: user-center-mysql-data mountPath: /var/lib/mysql - name: user-center-mysql-conf mountPath: /etc/mysql/mysql.conf.d/
|
应用 user-center-mysql-service.yaml
我们先使用NodePort方式暴露,然后导入数据后,之后你也可以修改为headless
apiVersion: v1 kind: Service metadata: name: user-center-mysql spec: type: NodePort selector: app: user-center-mysql ports: - port: 3306 targetPort: 3306
|
在 user-center-mysql 起来后,使用navicat或者其他工具连接mysql,将 open-capacity-platform/sql/01.user-center.sql导入数据库

修改并应用 user-center-mysql-service.yaml
apiVersion: v1 kind: Service metadata: name: user-center-mysql spec: clusterIP: None selector: app: user-center-mysql ports: - port: 3306 targetPort: 3306
|
部署Redis
应用 redis-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: ocp-redis data: redis.conf: |- ################################## INCLUDES ################################### # include /path/to/local.conf # include /path/to/other.conf
################################## MODULES ##################################### # loadmodule /path/to/my_module.so # loadmodule /path/to/other_module.so
bind 0.0.0.0
protected-mode no
port 6379
tcp-backlog 511
# unixsocket /tmp/redis.sock # unixsocketperm 700
timeout 0
tcp-keepalive 300
################################# GENERAL #####################################
daemonize no
supervised no
pidfile /data/pid/redis_6379.pid
loglevel notice
logfile "/data/logs/redis.log"
# syslog-enabled no
# syslog-ident redis
# syslog-facility local0
databases 16
always-show-logo yes
save 900 1 save 300 10 save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /data
# replicaof <masterip> <masterport>
# masterauth <master-password>
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
# repl-ping-replica-period 10
# repl-timeout 60
repl-disable-tcp-nodelay no
# repl-backlog-size 1mb
# repl-backlog-ttl 3600
replica-priority 100
# min-replicas-to-write 3 # min-replicas-max-lag 10
# replica-announce-ip 5.5.5.5 # replica-announce-port 1234
# requirepass xxxx
# rename-command CONFIG ""
maxclients 2000
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5
# replica-ignore-maxmemory yes
lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no replica-lazy-flush no
appendonly yes
appendfilename "appendonly.aof"
# appendfsync always appendfsync everysec # appendfsync no
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
# cluster-enabled yes
# cluster-config-file nodes-6379.conf
# cluster-node-timeout 15000
# cluster-replica-validity-factor 10
# cluster-migration-barrier 1
# cluster-require-full-coverage yes
# cluster-replica-no-failover no
# cluster-announce-ip 10.1.1.5 # cluster-announce-port 6379 # cluster-announce-bus-port 6380
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
# notify-keyspace-events Elg
# notify-keyspace-events Ex
notify-keyspace-events ""
hash-max-ziplist-entries 512 hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128 zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096 stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0 client-output-buffer-limit replica 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60
# client-query-buffer-limit 1gb
# proto-max-bulk-len 512mb
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
# lfu-log-factor 10 # lfu-decay-time 1
########################### ACTIVE DEFRAGMENTATION ####################### # activedefrag yes # active-defrag-ignore-bytes 100mb # active-defrag-threshold-lower 10 # active-defrag-threshold-upper 100 # active-defrag-cycle-min 5 # active-defrag-cycle-max 75 # active-defrag-max-scan-fields 1000
|
应用 redis-statefulset.yaml
apiVersion: apps/v1 kind: StatefulSet metadata: name: ocp-redis spec: replicas: 1 selector: matchLabels: app: ocp-redis serviceName: ocp-redis volumeClaimTemplates: - metadata: name: ocp-redis-data spec: storageClassName: managed-nfs-storage accessModes: - ReadWriteMany resources: requests: storage: 500Mi template: metadata: labels: app: ocp-redis spec: volumes: - name: ocp-redis-conf configMap: name: ocp-redis containers: - name: ocp-redis image: redis:5.0.5 imagePullPolicy: IfNotPresent ports: - containerPort: 6379 volumeMounts: - name: ocp-redis-data mountPath: /data - name: ocp-redis-conf mountPath: /data/redis.conf command: - redis-server - /data/redis.conf
|
应用 redis-service.yaml
apiVersion: v1 kind: Service metadata: name: ocp-redis spec: clusterIP: None selector: app: ocp-redis ports: - port: 6379 targetPort: 6379
|
部署auth-server
应用 auth-server-deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: auth-server spec: replicas: 2 selector: matchLabels: app: auth-server template: metadata: labels: app: auth-server spec: containers: - name: auth-server image: tangweifeng/auth-server imagePullPolicy: IfNotPresent ports: - containerPort: 8000
|
应用 auth-server-service.yaml
apiVersion: v1 kind: Service metadata: name: auth-server spec: type: ClusterIP selector: app: auth-server ports: - port: 8000 targetPort: 8000
|
部署user-center
应用 user-center-deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: user-center spec: replicas: 2 selector: matchLabels: app: user-center template: metadata: labels: app: user-center spec: containers: - name: user-center image: tangweifeng/user-center imagePullPolicy: IfNotPresent ports: - containerPort: 7000
|
应用 user-center-service.yaml
apiVersion: v1 kind: Service metadata: name: user-center spec: type: ClusterIP selector: app: user-center ports: - port: 7000 targetPort: 7000
|
部署log-center
应用 log-center-deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: log-center spec: replicas: 2 selector: matchLabels: app: log-center template: metadata: labels: app: log-center spec: containers: - name: log-center image: tangweifeng/log-center imagePullPolicy: IfNotPresent ports: - containerPort: 5006
|
应用 log-center-service.yaml
apiVersion: v1 kind: Service metadata: name: log-center spec: type: ClusterIP selector: app: log-center ports: - port: 5006 targetPort: 5006
|
部署api-gateway
应用 api-gateway-deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: api-gateway spec: replicas: 1 selector: matchLabels: app: api-gateway template: metadata: labels: app: api-gateway spec: containers: - name: api-gateway image: tangweifeng/api-gateway imagePullPolicy: IfNotPresent ports: - containerPort: 9200
|
应用 api-gateway-service.yaml
apiVersion: v1 kind: Service metadata: name: api-gateway spec: type: ClusterIP selector: app: api-gateway ports: - port: 9200 targetPort: 9200
|
应用 api-gateway-ingress.yaml
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: api-gateway annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: api-gateway.ocp.com http: paths: - path: / backend: serviceName: api-gateway servicePort: 9200
|
在windows hosts文件里面添加映射规则
部署back-center
在完成上面所有的部署后,我们再来部署一个前端的项目back-center。
修改open-capacity-platform/web-portal/back-center/src/main/view/static/module/config.js
# api-gateway的url,端口通过 kubectl get svc -n ingress-nginx 查看 base_server: 'http://api-gateway.ocp.com:32080/' # eureka_server的url eureka_server: 'http://eureka.twf.com:32080/'
|
在 open-capacity-platform/web-portal/back-center/dockerfile 里编写Dockerfile制作nginx镜像
FROM tangweifeng/nginx RUN rm -rf /usr/share/nginx/html ADD ./src/main/view/static /usr/share/nginx/html EXPOSE 80 CMD ["nginx","-g","daemon off;"]
|
制作镜像
docker build -t tangweifeng/back-center .
|
上传到远程仓库
docker push tangweifeng/back-center
|
应用 back-center-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: back-center data: nginx.conf: |- user nginx; worker_processes 1;
error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid;
events { worker_connections 1024; }
http { include /etc/nginx/mime.types; default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on; #tcp_nopush on;
keepalive_timeout 65;
#gzip on;
server { listen 80; listen [::]:80; server_name localhost;
location / { root /usr/share/nginx/html; index index.html index.htm; }
error_page 500 502 503 504 /50x.html;
location = /50x.html { root /usr/share/nginx/html; } } }
|
应用 back-center-deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: back-center spec: replicas: 2 selector: matchLabels: app: back-center template: metadata: labels: app: back-center spec: volumes: - name: nginx-conf configMap: name: back-center containers: - name: back-center image: tangweifeng/back-center imagePullPolicy: IfNotPresent ports: - containerPort: 80 volumeMounts: - name: nginx-conf mountPath: /etc/nginx/nginx.conf subPath: nginx.conf
|
应用 back-center-service.yaml
apiVersion: v1 kind: Service metadata: name: back-center spec: type: ClusterIP selector: app: back-center ports: - port: 80 targetPort: 80
|
应用 back-center-ingress.yaml
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: back-center annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: ocp.twf.com http: paths: - path: / backend: serviceName: back-center servicePort: 80
|
在windows hosts增加一条主机映射规则
测试
通过 ocp.twf.com:32080 访问

默认用户名和密码都是admin
