Jenkins持续发布OCP微服务到Kubernetes

环境准备

• Kubernetes集群：https://1335402049.github.io/2019/10/26/Kubeadm%E9%83%A8%E7%BD%B2Kubernetes%E9%9B%86%E7%BE%A4/
• GitLab(独立部署)：https://1335402049.github.io/2020/04/13/GitLab%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/
• Jenkins：https://1335402049.github.io/2020/12/30/Kubernetes%E9%83%A8%E7%BD%B2jenkins/
• Harbor：https://1335402049.github.io/2020/08/21/CentOS7%E6%90%AD%E5%BB%BAHarbor%E7%A7%81%E6%9C%89%E4%BB%93%E5%BA%93/

OCP项目准备与配置

ocp项目作者Gitee地址：https://gitee.com/owenwangwen/open-capacity-platform

修改docker仓库配置

修改 open-capacity-platform/pom.xml

# 大概55、56行 将 docker.host 改成 docker.repostory 增加 docker.registry.name 标签
<!-- harbor地址，默认80端口 -->
<docker.repostory>192.168.1.112</docker.repostory>
<!-- harbor项目地址 -->
<docker.registry.name>ocp</docker.registry.name>
<!-- 制作镜像的前缀 -->
<docker.image.prefix>ocp</docker.image.prefix>

修改eureka配置

修改 open-capacity-platform/register-center/eureka-server/src/main/resources/application-slave0.yml

# service名称为cloud-eureka
defaultZone: http://cloud-eureka-0.cloud-eureka:1111/eureka,http://cloud-eureka-1.cloud-eureka:1111/eureka,http://cloud-eureka-2.cloud-eureka:1111/eureka
prefer-ip-address: false
# instance字段下增加 hostname appname
hostname: cloud-eureka
appname: eureka-server

修改auth-server配置

修改 open-capacity-platform/oauth-center/auth-server/src/main/resources/bootstrap.yml

defaultZone: http://cloud-eureka-0.cloud-eureka:1111/eureka,http://cloud-eureka-1.cloud-eureka:1111/eureka,http://cloud-eureka-2.cloud-eureka:1111/eureka

修改 open-capacity-platform/oauth-center/auth-server/src/main/resources/application.yml

# auth-center-mysql 是后面部署的 headless 名称
url: jdbc:mysql://auth-center-mysql:3306/oauth-center?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false 
# mysql用户名和密码
username: root
password: xxxx

# redis配置
# ocp-redis 是后面部署的 headless 名称
host: ocp-redis

修改user-center配置

修改 open-capacity-platform/business-center/user-center/src/main/resources/bootstrap.yml

defaultZone: http://cloud-eureka-0.cloud-eureka:1111/eureka,http://cloud-eureka-1.cloud-eureka:1111/eureka,http://cloud-eureka-2.cloud-eureka:1111/eureka

修改 open-capacity-platform/business-center/user-center/src/main/resources/application.yml

# user-center-mysql 是后面部署的 headless 名称
url: jdbc:mysql://user-center-mysql:3306/user-center?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false 
# mysql用户名和密码
username: root
password: xxxx

# redis配置
# ocp-redis 是后面部署的 headless 名称
host: ocp-redis

修改log-center配置

修改 open-capacity-platform/monitor-center/log-center/src/main/resources/bootstrap.yml

defaultZone: http://cloud-eureka-0.cloud-eureka:1111/eureka,http://cloud-eureka-1.cloud-eureka:1111/eureka,http://cloud-eureka-2.cloud-eureka:1111/eureka

修改 open-capacity-platform/monitor-center/log-center/src/main/resources/application.yml

# log-center-mysql 是后面部署的 headless 名称
url: jdbc:mysql://log-center-mysql:3306/log-center?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false 
# mysql用户名和密码
username: root
password: xxxx

# redis配置
# ocp-redis 是后面部署的 headless 名称
host: ocp-redis

修改api-gateway配置

修改 open-capacity-platform/api-gateway/src/main/resources/bootstrap.yml

defaultZone: http://cloud-eureka-0.cloud-eureka:1111/eureka,http://cloud-eureka-1.cloud-eureka:1111/eureka,http://cloud-eureka-2.cloud-eureka:1111/eureka

修改 open-capacity-platform/api-gateway/src/main/resources/application.yml

url: jdbc:mysql://auth-center-mysql:3306/oauth-center?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false
# mysql用户名和密码
username: root
password: xxxx

# redis配置
# ocp-redis 是后面部署的 headless 名称
host: ocp-redis

修改back-center配置

修改 open-capacity-platform/web-portal/back-center/src/main/view/static/module/config.js

# api-gateway的url,端口通过 kubectl get svc -n ingress-nginx 查看
base_server: 'http://api-gateway.ocp.com:32080/'
# eureka_server的url
eureka_server: 'http://eureka.ocp.com:32080/'

在 open-capacity-platform/web-portal/back-center/dockerfile 里编写Dockerfile制作nginx镜像

FROM tangweifeng/nginx
RUN rm -rf /usr/share/nginx/html
ADD ./src/main/view/static /usr/share/nginx/html
EXPOSE 80
CMD ["nginx","-g","daemon off;"]

编写 dockerfile_auth-center

vim open-capacity-platform/sql/dockerfile_auth-center

FROM mysql:5.7.26
ADD 02.oauth-center.sql /docker-entrypoint-initdb.d/02.oauth-center.sql
EXPOSE 3306

编写 dockerfile_log-center

vim open-capacity-platform/sql/dockerfile_log-center

FROM mysql:5.7.26 
ADD 05.log-center.sql /docker-entrypoint-initdb.d/05.log-center.sql 
EXPOSE 3306

编写 dockerfile_user-center

vim open-capacity-platform/sql/dockerfile_user-center

FROM mysql:5.7.26
ADD 01.user-center.sql /docker-entrypoint-initdb.d/01.user-center.sql
EXPOSE 3306

部署ingress-nginx

参考这篇文章：https://1335402049.github.io/2020/09/23/Kubernetes%E9%83%A8%E7%BD%B2ingress-nginx/

部署nfs storageclass

在部署微服务之前，考虑mysql、redis等服务数据的存储，我们准备使用nfs storageclass进行动态pv的创建，可以参考这篇文章：https://1335402049.github.io/2020/09/16/Kubernetes%E4%B8%AD%E4%BD%BF%E7%94%A8NFS%E7%9A%84StorageClass/

Kubernetes资源清单

eureka

eureka-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cloud-eureka
spec:
  replicas: 3
  selector:
    matchLabels:
      app: eureka
  serviceName: cloud-eureka
  template:
    metadata:
      labels:
        app: eureka
    spec:
      imagePullSecrets:
        - name: harbor-registry
      containers:
        - name: eureka-server
          image: 192.168.1.112/ocp/eureka-server
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 1111

eureka-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: cloud-eureka
spec:
  type: ClusterIP
  ports:
    - port: 1111
      targetPort: 1111
  selector:
    app: eureka

eureka-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: eureka
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: eureka.ocp.com
      http:
        paths:
          - path: /
            backend:
              serviceName: cloud-eureka
              servicePort: 1111

mysql

auth-center-mysql

auth-center-mysql-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: auth-center-mysql
data:
  # 使用 echo -n '<密码>' |base64 获取加密后的密码
  password: dGFuZzE2MTE=
stringData:
  username: root

auth-center-mysql-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: auth-center-mysql
data:
  mysqld.cnf: |-
    [mysqld]
    pid-file    = /var/run/mysqld/mysqld.pid
    socket        = /var/run/mysqld/mysqld.sock
    datadir        = /var/lib/mysql
    #log-error    = /var/log/mysql/error.log
    # By default we only accept connections from localhost
    #bind-address    = 127.0.0.1
    # Disabling symbolic-links is recommended to prevent assorted security risks

auth-center-mysql-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: auth-center-mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: auth-center-mysql
  serviceName: auth-center-mysql
  volumeClaimTemplates:
    - metadata:
        name: auth-center-mysql-data
      spec:
        storageClassName: managed-nfs-storage
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 500Mi
  template:
    metadata:
      labels:
        app: auth-center-mysql
    spec:
      imagePullSecrets:
        - name: harbor-registry
      volumes:
        - name: auth-center-mysql-conf
          configMap:
            name: auth-center-mysql
      containers:
        - name: auth-center-mysql
          image: 192.168.1.112/ocp/auth-center-mysql
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3306
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: auth-center-mysql
                  key: password
          volumeMounts:
            - name: auth-center-mysql-data
              mountPath: /var/lib/mysql
            - name: auth-center-mysql-conf
              mountPath: /etc/mysql/mysql.conf.d/

auth-center-mysql-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: auth-center-mysql
spec:
  clusterIP: None
  selector:
    app: auth-center-mysql
  ports:
    - port: 3306
      targetPort: 3306

log-center-mysql

log-center-mysql-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: log-center-mysql
data:
  # 使用 echo -n '<密码>' |base64 获取加密后的密码
  password: dGFuZzE2MTE=
stringData:
  username: root

log-center-mysql-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: log-center-mysql
data:
  mysqld.cnf: |-
    [mysqld]
    pid-file    = /var/run/mysqld/mysqld.pid
    socket        = /var/run/mysqld/mysqld.sock
    datadir        = /var/lib/mysql
    #log-error    = /var/log/mysql/error.log
    # By default we only accept connections from localhost
    #bind-address    = 127.0.0.1
    # Disabling symbolic-links is recommended to prevent assorted security risks

log-center-mysql-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: log-center-mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: log-center-mysql
  serviceName: log-center-mysql
  volumeClaimTemplates:
    - metadata:
        name: log-center-mysql-data
      spec:
        storageClassName: managed-nfs-storage
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 500Mi
  template:
    metadata:
      labels:
        app: log-center-mysql
    spec:
      imagePullSecrets:
        - name: harbor-registry
      volumes:
        - name: log-center-mysql-conf
          configMap:
            name: log-center-mysql
      containers:
        - name: log-center-mysql
          image: 192.168.1.112/ocp/log-center-mysql
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3306
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: log-center-mysql
                  key: password
          volumeMounts:
            - name: log-center-mysql-data
              mountPath: /var/lib/mysql
            - name: log-center-mysql-conf
              mountPath: /etc/mysql/mysql.conf.d/

log-center-mysql-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: log-center-mysql
spec:
  clusterIP: None
  selector:
    app: log-center-mysql
  ports:
    - port: 3306
      targetPort: 3306

user-center-mysql

user-center-mysql-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: user-center-mysql
data:
  # 使用 echo -n '<密码>' |base64 获取加密后的密码
  password: dGFuZzE2MTE=
stringData:
  username: root

user-center-mysql-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-center-mysql
data:
  mysqld.cnf: |-
    [mysqld]
    pid-file    = /var/run/mysqld/mysqld.pid
    socket        = /var/run/mysqld/mysqld.sock
    datadir        = /var/lib/mysql
    #log-error    = /var/log/mysql/error.log
    # By default we only accept connections from localhost
    #bind-address    = 127.0.0.1
    # Disabling symbolic-links is recommended to prevent assorted security risks

user-center-mysql-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: user-center-mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: user-center-mysql
  serviceName: user-center-mysql
  volumeClaimTemplates:
    - metadata:
        name: user-center-mysql-data
      spec:
        storageClassName: managed-nfs-storage
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 500Mi
  template:
    metadata:
      labels:
        app: user-center-mysql
    spec:
      imagePullSecrets:
        - name: harbor-registry
      volumes:
        - name: user-center-mysql-conf
          configMap:
            name: user-center-mysql
      containers:
        - name: user-center-mysql
          image: 192.168.1.112/ocp/user-center-mysql
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3306
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: user-center-mysql
                  key: password
          volumeMounts:
            - name: user-center-mysql-data
              mountPath: /var/lib/mysql
            - name: user-center-mysql-conf
              mountPath: /etc/mysql/mysql.conf.d/

user-center-mysql-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: user-center-mysql
spec:
  clusterIP: None
  selector:
    app: user-center-mysql
  ports:
    - port: 3306
      targetPort: 3306

redis

redis-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: ocp-redis
data:
  redis.conf: |-
    ################################## INCLUDES ###################################
    # include /path/to/local.conf
    # include /path/to/other.conf

    ################################## MODULES #####################################
    # loadmodule /path/to/my_module.so
    # loadmodule /path/to/other_module.so

    bind 0.0.0.0

    protected-mode no

    port 6379

    tcp-backlog 511

    # unixsocket /tmp/redis.sock
    # unixsocketperm 700

    timeout 0

    tcp-keepalive 300

    ################################# GENERAL #####################################

    daemonize no

    supervised no

    pidfile /data/pid/redis_6379.pid

    loglevel notice

    logfile "/data/logs/redis.log"

    # syslog-enabled no

    # syslog-ident redis

    # syslog-facility local0

    databases 16

    always-show-logo yes

    save 900 1
    save 300 10
    save 60 10000

    stop-writes-on-bgsave-error yes

    rdbcompression yes

    rdbchecksum yes

    dbfilename dump.rdb

    dir /data

    # replicaof <masterip> <masterport>

    # masterauth <master-password>

    replica-serve-stale-data yes

    replica-read-only yes

    repl-diskless-sync no

    repl-diskless-sync-delay 5

    # repl-ping-replica-period 10

    # repl-timeout 60

    repl-disable-tcp-nodelay no

    # repl-backlog-size 1mb

    # repl-backlog-ttl 3600

    replica-priority 100

    # min-replicas-to-write 3
    # min-replicas-max-lag 10

    # replica-announce-ip 5.5.5.5
    # replica-announce-port 1234

    # requirepass xxxx

    # rename-command CONFIG ""

    maxclients 2000

    # maxmemory <bytes>

    # maxmemory-policy noeviction

    # maxmemory-samples 5

    # replica-ignore-maxmemory yes

    lazyfree-lazy-eviction no
    lazyfree-lazy-expire no
    lazyfree-lazy-server-del no
    replica-lazy-flush no

    appendonly yes

    appendfilename "appendonly.aof"

    # appendfsync always
    appendfsync everysec
    # appendfsync no

    no-appendfsync-on-rewrite no

    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb

    aof-load-truncated yes

    aof-use-rdb-preamble yes

    lua-time-limit 5000

    # cluster-enabled yes

    # cluster-config-file nodes-6379.conf

    # cluster-node-timeout 15000

    # cluster-replica-validity-factor 10

    # cluster-migration-barrier 1

    # cluster-require-full-coverage yes

    # cluster-replica-no-failover no

    # cluster-announce-ip 10.1.1.5
    # cluster-announce-port 6379
    # cluster-announce-bus-port 6380

    slowlog-log-slower-than 10000

    slowlog-max-len 128

    latency-monitor-threshold 0

    #  notify-keyspace-events Elg

    #  notify-keyspace-events Ex

    notify-keyspace-events ""

    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64

    list-max-ziplist-size -2

    list-compress-depth 0

    set-max-intset-entries 512

    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64

    hll-sparse-max-bytes 3000

    stream-node-max-bytes 4096
    stream-node-max-entries 100

    activerehashing yes

    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit replica 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60

    # client-query-buffer-limit 1gb

    # proto-max-bulk-len 512mb

    hz 10

    dynamic-hz yes

    aof-rewrite-incremental-fsync yes

    rdb-save-incremental-fsync yes

    # lfu-log-factor 10
    # lfu-decay-time 1

    ########################### ACTIVE DEFRAGMENTATION #######################
    # activedefrag yes
    # active-defrag-ignore-bytes 100mb
    # active-defrag-threshold-lower 10
    # active-defrag-threshold-upper 100
    # active-defrag-cycle-min 5
    # active-defrag-cycle-max 75
    # active-defrag-max-scan-fields 1000

redis-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ocp-redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ocp-redis
  serviceName: ocp-redis
  volumeClaimTemplates:
    - metadata:
        name: ocp-redis-data
      spec:
        storageClassName: managed-nfs-storage
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 500Mi
  template:
    metadata:
      labels:
        app: ocp-redis
    spec:
      volumes:
        - name: ocp-redis-conf
          configMap:
            name: ocp-redis
      containers:
        - name: ocp-redis
          image: redis:5.0.5
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: ocp-redis-data
              mountPath: /data
            - name: ocp-redis-conf
              mountPath: /data/redis.conf
          command:
            - redis-server
            - /data/redis.conf

redis-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: ocp-redis
spec:
  clusterIP: None
  selector:
    app: ocp-redis
  ports:
    - port: 6379
      targetPort: 6379

auth-server

auth-server-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: auth-server
  template:
    metadata:
      labels:
        app: auth-server
    spec:
      imagePullSecrets:
        - name: harbor-registry
      containers:
        - name: auth-server
          image: 192.168.1.112/ocp/auth-server
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8000

auth-server-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: auth-server
spec:
  type: ClusterIP
  selector:
    app: auth-server
  ports:
    - port: 8000
      targetPort: 8000

user-center

user-center-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-center
spec:
  replicas: 2
  selector:
    matchLabels:
      app: user-center
  template:
    metadata:
      labels:
        app: user-center
    spec:
      imagePullSecrets:
        - name: harbor-registry
      containers:
        - name: user-center
          image: 192.168.1.112/ocp/user-center
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 7000

user-center-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: user-center
spec:
  type: ClusterIP
  selector:
    app: user-center
  ports:
    - port: 7000
      targetPort: 7000

log-center

log-center-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: log-center
spec:
  replicas: 2
  selector:
    matchLabels:
      app: log-center
  template:
    metadata:
      labels:
        app: log-center
    spec:
      imagePullSecrets:
        - name: harbor-registry
      containers:
        - name: log-center
          image: 192.168.1.112/ocp/log-center
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5006

log-center-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: log-center
spec:
  type: ClusterIP
  selector:
    app: log-center
  ports:
    - port: 5006
      targetPort: 5006

api-gateway

api-gateway-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      imagePullSecrets:
        - name: harbor-registry
      containers:
        - name: api-gateway
          image: 192.168.1.112/ocp/api-gateway
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9200

api-gateway-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: api-gateway
spec:
  type: ClusterIP
  selector:
    app: api-gateway
  ports:
    - port: 9200
      targetPort: 9200

api-gateway-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: api-gateway
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: api-gateway.ocp.com
      http:
        paths:
          - path: /
            backend:
              serviceName: api-gateway
              servicePort: 9200

back-center

back-center-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: back-center
data:
  nginx.conf: |-
    user  nginx;
    worker_processes  1;

    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;

    events {
        worker_connections  1024;
    }

    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;

        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';

        access_log  /var/log/nginx/access.log  main;

        sendfile        on;
        #tcp_nopush     on;

        keepalive_timeout  65;

        #gzip  on;

        server {
            listen       80;
            listen  [::]:80;
            server_name  localhost;

            location / {
                root   /usr/share/nginx/html;
                index  index.html index.htm;
            }

            error_page   500 502 503 504  /50x.html;

            location = /50x.html {
                root   /usr/share/nginx/html;
            }
        }
    }

back-center-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: back-center
spec:
  replicas: 2
  selector:
    matchLabels:
      app: back-center
  template:
    metadata:
      labels:
        app: back-center
    spec:
      imagePullSecrets:
        - name: harbor-registry
      volumes:
        - name: nginx-conf
          configMap:
            name: back-center
      containers:
        - name: back-center
          image: 192.168.1.112/ocp/back-center
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nginx-conf
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf

back-center-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: back-center
spec:
  type: NodePort
  selector:
    app: back-center
  ports:
    - port: 80
      targetPort: 80

推送项目到GitLab

Jenkins持续发布

• 在jenkins配置kubernetes，参考文章：https://1335402049.github.io/2020/12/30/Jenkins%E9%85%8D%E7%BD%AEKubernetes%E9%9B%86%E7%BE%A4/
• 在jenkins配置GitLab凭据（凭据选择用户名密码类型，填写gitlab登录的用户名密码），以及如何远程执行GitLab上的jenkinsfile，参考文章：https://1335402049.github.io/2020/12/30/Jenkins%E4%BB%8E%E8%BF%9C%E7%A8%8Bgit%E4%BB%93%E5%BA%93%E6%89%A7%E8%A1%8Cjenkinsfile/

Pipeline

目前整个ocp微服务持续集成/持续交付分为7个流水线任务

mysql_redis

pipeline {
    agent {
        label 'node1'
    }
    environment {
       git_url = 'http://192.168.1.112/root/open-capacity-platform.git'
       credentialsId = 'gitlab'
       git_branch = 'master'
    }
    stages {
       stage('pull git code') {
            steps {
                git credentialsId:"${credentialsId}",url:"${git_url}",branch: "${git_branch}"
            }
       }
        stage('make mysql image') {
            steps {
                sh'''
                    cd sql
                    docker build -f dockerfile_auth-center -t 120.78.187.149/ocp/auth-center-mysql .
                    docker build -f dockerfile_user-center -t 120.78.187.149/ocp/user-center-mysql .
                    docker build -f dockerfile_log-center -t 120.78.187.149/ocp/log-center-mysql .
                '''
            }
        }
       stage('upload image') {
                   steps {
                       sh'''
                           docker login 120.78.187.149 -u admin -p tang1611
                           docker push 120.78.187.149/ocp/log-center-mysql
                           docker push 120.78.187.149/ocp/auth-center-mysql
                           docker push 120.78.187.149/ocp/user-center-mysql
                       '''
                   }
       }
        stage('apply mysql resource list'){
            steps {
                sh'''
                    kubectl create secret docker-registry harbor-registry  --docker-server=192.168.1.112 --docker-username=admin --docker-password=harbor123 --docker-email=admin@qq.com -n default
                    kubectl apply -f k8s/auth-center-mysql/auth-center-mysql-secret.yaml -n default
                    kubectl apply -f k8s/auth-center-mysql/auth-center-mysql-configmap.yaml -n default
                    kubectl apply -f k8s/auth-center-mysql/auth-center-mysql-statefulset.yaml -n default
                    kubectl apply -f k8s/auth-center-mysql/auth-center-mysql-service.yaml -n default
                    kubectl apply -f k8s/user-center-mysql/user-center-mysql-secret.yaml -n default
                    kubectl apply -f k8s/user-center-mysql/user-center-mysql-configmap.yaml -n default
                    kubectl apply -f k8s/user-center-mysql/user-center-mysql-statefulset.yaml -n default
                    kubectl apply -f k8s/user-center-mysql/user-center-mysql-service.yaml -n default
                    kubectl apply -f k8s/log-center-mysql/log-center-mysql-secret.yaml -n default
                    kubectl apply -f k8s/log-center-mysql/log-center-mysql-configmap.yaml -n default
                    kubectl apply -f k8s/log-center-mysql/log-center-mysql-statefulset.yaml -n default
                    kubectl apply -f k8s/log-center-mysql/log-center-mysql-service.yaml -n default
                '''
            }
        }
       stage('apply redis resource list'){
            steps {
                sh'''
                    cd k8s/redis
                    kubectl apply -f redis-configmap.yaml -n default
                    kubectl apply -f redis-statefulset.yaml -n default
                    kubectl apply -f redis-service.yaml -n default
                '''
            }
       }
    }
}

eureka

pipeline {
    agent {
        label 'node1'
    }
    environment {
       git_url = 'http://192.168.1.112/root/open-capacity-platform.git'
       credentialsId = 'gitlab'
       git_branch = 'master'
    }
    stages {
       stage('pull git code') {
            steps {
                git credentialsId:"${credentialsId}",url:"${git_url}",branch: "${git_branch}"
            }
       }
       stage('unpack code') {
                   steps {
                       sh "mvn -U -pl register-center/eureka-server/ -am clean package -DskipTests"
                   }
       }
       stage('make eureka image') {
                   steps {
                       sh'''
                          cd register-center/eureka-server && mvn docker:build
                       '''
                   }
       }
       stage('upload image') {
                   steps {
                       sh'''
                           docker login 120.78.187.149 -u admin -p tang1611
                           docker tag ocp/eureka-server 120.78.187.149/ocp/eureka-server
                           docker rmi ocp/eureka-server
                           docker push 120.78.187.149/ocp/eureka-server
                       '''
                   }
       }
        stage('apply eureka resource list'){
            steps {
                sh'''
                    cd k8s/eureka
                    kubectl apply -f eureka-statefulset.yaml -n default
                    kubectl apply -f eureka-service.yaml -n default
                    kubectl apply -f eureka-ingress.yaml -n default
                '''
            }
       }
    }
}

auth-server

pipeline {
    agent {
        label 'node1'
    }
    environment {
       git_url = 'http://192.168.1.112/root/open-capacity-platform.git'
       credentialsId = 'gitlab'
       git_branch = 'master'
    }
    stages {
       stage('pull git code') {
            steps {
                git credentialsId:"${credentialsId}",url:"${git_url}",branch: "${git_branch}"
            }
       }
       stage('unpack code') {
                   steps {
                       sh "mvn -U -pl oauth-center/auth-server -am clean package -DskipTests"
                   }
       }
       stage('make auth-server image') {
            steps {
                sh'''
                   cd oauth-center/auth-server && mvn docker:build
                '''
            }
       }
       stage('upload image') {
                   steps {
                       sh'''
                           docker login 120.78.187.149 -u admin -p tang1611
                           docker tag ocp/auth-server 120.78.187.149/ocp/auth-server
                           docker rmi ocp/auth-server
                           docker push 120.78.187.149/ocp/auth-server
                       '''
                   }
       }
        stage('apply auth-server resource list'){
            steps {
                sh'''
                    cd k8s/auth-server
                    kubectl apply -f auth-server-deployment.yaml -n default
                    kubectl apply -f auth-server-service.yaml -n default
                '''
            }
        }
    }
}

user-center

pipeline {
    agent {
        label 'node1'
    }
    environment {
       git_url = 'http://192.168.1.112/root/open-capacity-platform.git'
       credentialsId = 'gitlab'
       git_branch = 'master'
    }
    stages {
       stage('pull git code') {
            steps {
                git credentialsId:"${credentialsId}",url:"${git_url}",branch: "${git_branch}"
            }
       }
       stage('unpack code') {
                   steps {
                       sh "mvn -U -pl business-center/user-center -am clean package -DskipTests"
                   }
       }
       stage('make user-center image') {
            steps {
                sh'''
                   cd business-center/user-center && mvn docker:build
                '''
            }
       }
       stage('upload image') {
                   steps {
                       sh'''
                           docker login 120.78.187.149 -u admin -p tang1611
                           docker tag ocp/user-center 120.78.187.149/ocp/user-center
                           docker rmi ocp/user-center
                           docker push 120.78.187.149/ocp/user-center
                       '''
                   }
       }
        stage('apply user-center resource list'){
            steps {
                sh'''
                    cd k8s/user-center
                    kubectl apply -f user-center-deployment.yaml -n default
                    kubectl apply -f user-center-service.yaml -n default
                '''
            }
        }
    }
}

log-center

pipeline {
    agent {
        label 'node1'
    }
    environment {
       git_url = 'http://192.168.1.112/root/open-capacity-platform.git'
       credentialsId = 'gitlab'
       git_branch = 'master'
    }
    stages {
       stage('pull git code') {
            steps {
                git credentialsId:"${credentialsId}",url:"${git_url}",branch: "${git_branch}"
            }
       }
       stage('unpack code') {
                   steps {
                       sh "mvn -U -pl monitor-center/log-center -am clean package -DskipTests"
                   }
       }
       stage('make log-center image') {
            steps {
                sh'''
                   cd monitor-center/log-center && mvn docker:build
                '''
            }
       }
       stage('upload image') {
                   steps {
                       sh'''
                           docker login 120.78.187.149 -u admin -p tang1611
                           docker tag ocp/log-center 120.78.187.149/ocp/log-center
                           docker rmi ocp/log-center
                           docker push 120.78.187.149/ocp/log-center
                       '''
                   }
       }
        stage('apply log-center resource list'){
            steps {
                sh'''
                    cd k8s/log-center
                    kubectl apply -f log-center-deployment.yaml -n default
                    kubectl apply -f log-center-service.yaml -n default
                '''
            }
        }
    }
}

api-gateway

pipeline {
    agent {
        label 'node1'
    }
    environment {
       git_url = 'http://192.168.1.112/root/open-capacity-platform.git'
       credentialsId = 'gitlab'
       git_branch = 'master'
    }
    stages {
       stage('pull git code') {
            steps {
                git credentialsId:"${credentialsId}",url:"${git_url}",branch: "${git_branch}"
            }
       }
       stage('unpack code') {
                   steps {
                       sh "mvn -U -pl api-gateway -am clean package -DskipTests"
                   }
       }
       stage('make api-gateway image') {
            steps {
                sh'''
                   cd api-gateway && mvn docker:build
                '''
            }
       }
       stage('upload image') {
                   steps {
                       sh'''
                           docker login 120.78.187.149 -u admin -p tang1611
                           docker tag ocp/api-gateway 120.78.187.149/ocp/api-gateway
                           docker rmi ocp/api-gateway
                           docker push 120.78.187.149/ocp/api-gateway
                       '''
                   }
       }
        stage('apply api-gateway resource list'){
            steps {
                sh'''
                    cd k8s/api-gateway
                    kubectl apply -f api-gateway-deployment.yaml -n default
                    kubectl apply -f api-gateway-service.yaml -n default
                    kubectl apply -f api-gateway-ingress.yaml -n default
                '''
            }
        }
    }
}

back-center

pipeline {
    agent {
        label 'node1'
    }
    environment {
       git_url = 'http://192.168.1.112/root/open-capacity-platform.git'
       credentialsId = 'gitlab'
       git_branch = 'master'
    }
    stages {
       stage('pull git code') {
            steps {
                git credentialsId:"${credentialsId}",url:"${git_url}",branch: "${git_branch}"
            }
       }
       stage('make back-center image') {
            steps {
                sh'''
                   cd web-portal/back-center/
                   docker build -t 120.78.187.149/ocp/back-center .
                '''
            }
       }
       stage('upload image') {
                   steps {
                       sh'''
                           docker login 120.78.187.149 -u admin -p tang1611
                           docker push 120.78.187.149/ocp/back-center
                       '''
                   }
       }
       stage('apply back-center resource list'){
            steps {
                sh'''
                    cd k8s/back-center
                    kubectl apply -f back-center-configmap.yaml -n default
                    kubectl apply -f back-center-deployment.yaml -n default
                    kubectl apply -f back-center-service.yaml -n default
                '''
            }
       }
    }
}

在 windows的hosts文件里面添加记录

192.168.1.110 eureka.ocp.com
192.168.1.110 api-gateway.ocp.com

构建

在构建之前，由于cnych/jenkins:jnlp6镜像没有maven环境，我们在这个镜像基础上重新制作镜像，增加maven环境

# 直接拉取即可,修改jenkins slave的pod模板，填写这个镜像
docker pull tangweifeng/jenkins-slave:latest

还需要再本地宿主机上安装maven，提前准备下载好本地需要的依赖jar

修改本地maven的settings.xml

# 在setting标签下添加localRepository
<localRepository>/nfs/jarHome</localRepository>

执行，依赖会被下载到/nfs/jarHome路径下

mvn clean package -DskipTests

修改jenkins slave 的Pod模板，增加一个volume

如果不这样挂载依赖到容器里面，每次构建都会耗费大量时间在maven打包上

按照顺序依次执行pipeline，mysql_redis - eureka - auth-server - user-center - log-center - api-gateway - back-center

上面的各个pipeline脚本还有很多需要完善的地方，需要根据自己的实际情况进行完善，比如：job的触发器、sonarqube代码质量检查、邮件通知、自动化测试、shell脚本进行一些特定需求的开发、在制作镜像的时候先判断docker上是否存在此镜像（是否需要先进行删除）等等其他需求

最后在pipeline执行完成之，pod等资源被成功创建运行后，在浏览器通过任意node的ip:NodePort（back-center的nodeport）访问