2024-08-11 21:05:06 +08:00
|
|
|
|
<h1><center>CI/CD+Harbor+K8S实现微服务项目持续集成和发布</center></h1>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## 一：项目描述
|
|
|
|
|
|
|
|
|
|
|
|
#### 1.环境介绍
|
|
|
|
|
|
|
|
|
|
|
|
| 服务器 | IP | 角色 | 配置 |
|
|
|
|
|
|
| :--------: | :------------: | :--------------------------------: | :-----: |
|
|
|
|
|
|
| jenkins | 192.168.18.210 | 持续构建项目并发布 | 1Cpu+3G |
|
|
|
|
|
|
| gitlab | 192.168.18.200 | （https://www.xingdian.com）版本库 | 2Cpu+8G |
|
|
|
|
|
|
| harbor | 192.168.18.230 | 镜像仓库 | 1Cpu+3G |
|
|
|
|
|
|
| NFS | 192.168.18.230 | 给k8s提供持久化存储 | 1Cpu+3G |
|
|
|
|
|
|
| k8s-master | 192.168.18.160 | k8s管理节点 | 2Cpu+4G |
|
|
|
|
|
|
| k8s-node-1 | 192.168.18.161 | k8s项目节点 | 1Cpu+3G |
|
|
|
|
|
|
| k8s-node-2 | 192.168.18.162 | k8s项目节点 | 1Cpu+3G |
|
|
|
|
|
|
| k8s-node-3 | 192.168.18.163 | k8s项目节点 | 1Cpu+3G |
|
|
|
|
|
|
|
|
|
|
|
|
#### 2.jenkins部署
|
|
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
|
1.上传jdk
|
|
|
|
|
|
[root@jenkins ~]# tar xzf jdk-8u191-linux-x64.tar.gz -C /usr/local/
|
|
|
|
|
|
[root@jenkins ~]# cd /usr/local/
|
|
|
|
|
|
[root@jenkins local]# mv jdk1.8.0_191/ java
|
|
|
|
|
|
2.安装tomcat
|
|
|
|
|
|
[root@jenkins ~]# wget http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.42/bin/apache-tomcat-8.5.42.tar.gz
|
|
|
|
|
|
[root@jenkins ~]# tar xzf apache-tomcat-8.5.42.tar.gz -C /usr/local/
|
|
|
|
|
|
[root@jenkins ~]# cd /usr/local/
|
|
|
|
|
|
[root@jenkins local]# mv apache-tomcat-8.5.42/ tomcat
|
|
|
|
|
|
3.安装maven
|
|
|
|
|
|
[root@jenkins ~]# wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz
|
|
|
|
|
|
[root@jenkins ~]# tar xzf apache-maven-3.5.4-bin.tar.gz -C /usr/local/java
|
|
|
|
|
|
[root@jenkins ~]# cd /usr/local/java
|
|
|
|
|
|
[root@jenkins java]# mv apache-maven-3.5.4/ maven
|
|
|
|
|
|
设置变量:
|
|
|
|
|
|
[root@jenkins-server ~]# vim /etc/profile
|
|
|
|
|
|
JAVA_HOME=/usr/local/java
|
|
|
|
|
|
MAVEN_HOME=/usr/local/java/maven
|
|
|
|
|
|
PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin
|
|
|
|
|
|
export PATH JAVA_HOME MAVEN_HOME
|
|
|
|
|
|
[root@jenkins-server ~]# source /etc/profile
|
|
|
|
|
|
验证:
|
|
|
|
|
|
[root@jenkins-server ~]# java -version
|
|
|
|
|
|
java version "1.8.0_191"
|
|
|
|
|
|
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
|
|
|
|
|
|
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
|
|
|
|
|
|
[root@jenkins-server ~]# mvn -v
|
|
|
|
|
|
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
|
|
|
|
|
|
Maven home: /usr/local/java/maven
|
|
|
|
|
|
Java version: 1.8.0_191, vendor: Oracle Corporation, runtime: /usr/local/java/jre
|
|
|
|
|
|
Default locale: en_US, platform encoding: UTF-8
|
|
|
|
|
|
OS name: "linux", version: "3.10.0-693.el7.x86_64", arch: "amd64", family: "unix"
|
|
|
|
|
|
下载jenkins的安装包:
|
|
|
|
|
|
安装jenkins:2.332.3 ----通过官网直接下载war包。
|
|
|
|
|
|
官网:http://updates.jenkins-ci.org/download/war/
|
|
|
|
|
|
[root@jenkins-server ~]# wget https://get.jenkins.io/war-stable/2.332.3/jenkins.war
|
|
|
|
|
|
部署jenkins
|
|
|
|
|
|
[root@jenkins-server ~]# cd /usr/local/tomcat/webapps/
|
|
|
|
|
|
[root@jenkins-server webapps]# rm -rf *
|
|
|
|
|
|
[root@jenkins-server webapps]# cp /root/jenkins.war .
|
|
|
|
|
|
[root@jenkins-server webapps]# ./bin/startup.sh
|
|
|
|
|
|
Using CATALINA_BASE: /usr/local/tomcat
|
|
|
|
|
|
Using CATALINA_HOME: /usr/local/tomcat
|
|
|
|
|
|
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
|
|
|
|
|
|
Using JRE_HOME: /usr/local/java
|
|
|
|
|
|
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
|
|
|
|
|
|
Tomcat started.
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### 3.gitlab部署
|
|
|
|
|
|
|
|
|
|
|
|
部署文件地址： https://docs.qq.com/doc/DQ0hScnRCbVN6QW1F
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### 4.harbor部署
|
|
|
|
|
|
|
|
|
|
|
|
部署链接地址：https://docs.qq.com/doc/DQ0l1bUtFdFNQSmdR
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## 二：项目使用
|
|
|
|
|
|
|
|
|
|
|
|
在实现部署之前在各个服务至上进行配置，以下是各个配置详情。
|
|
|
|
|
|
|
|
|
|
|
|
#### 1.jenkins配置
|
|
|
|
|
|
|
|
|
|
|
|
插件安装：
|
|
|
|
|
|
|
|
|
|
|
|
Maven Integration
|
|
|
|
|
|
|
|
|
|
|
|
Generic Webhook Trigger
|
|
|
|
|
|
|
|
|
|
|
|
Deploy to container
|
|
|
|
|
|
|
|
|
|
|
|
Git
|
|
|
|
|
|
|
|
|
|
|
|
Publish Over SSH
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
配置JDK+MAVEN+GIT环境
|
|
|
|
|
|
|
|
|
|
|
|
Dashboard--->Global Tool Configuration
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
注意：需要在服务器上安装git
|
|
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
|
[root@jenkins ~]# yum -y install git
|
|
|
|
|
|
[root@jenkins ~]# git config --global user.email "xingdianvip@gmail.com"
|
|
|
|
|
|
[root@jenkins ~]# git config --global user.name "xingdian"
|
|
|
|
|
|
[root@jenkins ~]# git config --global http.sslVerify "false"
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
例如：
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
jenkisn节点安装docker（因为需要构建镜像到harbor）
|
|
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
|
[root@jenkins ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
|
|
|
|
|
|
[root@jenkins ~]# yum -y install docker-ce
|
|
|
|
|
|
[root@jenkins ~]# systemctl start docker && docker enable docker
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
配置http访问，创建daemon.json指定harbor地址
|
|
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
|
[root@jenkins ~]# cat /etc/docker/daemon.json
|
|
|
|
|
|
{
|
|
|
|
|
|
"insecure-registries":["192.168.18.230:80"]
|
|
|
|
|
|
}
|
|
|
|
|
|
[root@jenkins ~]# systemctl daemon-reload && systemctl restart docker
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
注意：另外一种修改docker.service文件添加--insecure-registry，在这里不生效
|
|
|
|
|
|
|
|
|
|
|
|
额外配置，在企业中每个微服务项目都是独立的，但是此项目源码具有关联性，故需要执行一下操作
|
|
|
|
|
|
|
|
|
|
|
|
将所有的项目源码上传到jenkis服务器，在项目目录下执行以下命令，此过程需要耐心等待
|
|
|
|
|
|
|
|
|
|
|
|
如果想加快速度，百度搜索mvn的国内仓库地址（略）
|
|
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
|
[root@jenkins tensquare_parent]# mvn install
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
创建maven项目：
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
注意：地址来自下面的gitlab部署
|
|
|
|
|
|
|
|
|
|
|
|
创建凭据 （再此添加kubernetes 集群master节点）
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
配置ssh remote hosts
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
取消gitlab配置
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
配置webhook
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
脚本如下：
|
|
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
|
# Jenkins机器：编译完成后，build生成一个新版本的镜像，push到远程docker仓库
|
|
|
|
|
|
|
|
|
|
|
|
# Variables
|
|
|
|
|
|
# 注意下面路径里的maven-docker就是jenkins项目的名称，必须一致
|
|
|
|
|
|
JENKINS_WAR_HOME='/root/.jenkins/workspace/eureka/target'
|
|
|
|
|
|
DOCKERFILE='/root/.jenkins/workspace/eureka/'
|
|
|
|
|
|
# 自己创建下面目录，主要镜像构建
|
|
|
|
|
|
DOCKERFILE_HOME='/root/jenkins/docker-file/eureka_jar'
|
|
|
|
|
|
HARBOR_IP='192.168.18.230'
|
|
|
|
|
|
REPOSITORIES='xingdian/eureka'
|
|
|
|
|
|
HARBOR_USER='admin'
|
|
|
|
|
|
HARBOR_USER_PASSWD='Harbor12345'
|
|
|
|
|
|
HARBOR_USER_EMAIL='xingdianvip@gmail.com'
|
|
|
|
|
|
|
|
|
|
|
|
# Copy the newest war to docker-file directory.
|
|
|
|
|
|
if [ -f /root/jenkins/docker-file/eureka_jar/eureka.jar ];then
|
|
|
|
|
|
rm -rf eureka.jar
|
|
|
|
|
|
\cp -f ${JENKINS_WAR_HOME}/tensquare_eureka_server-1.0-SNAPSHOT.jar ${DOCKERFILE_HOME}/eureka.jar
|
|
|
|
|
|
else
|
|
|
|
|
|
\cp -f ${JENKINS_WAR_HOME}/tensquare_eureka_server-1.0-SNAPSHOT.jar ${DOCKERFILE_HOME}/eureka.jar
|
|
|
|
|
|
fi
|
|
|
|
|
|
# Delete image early version.
|
|
|
|
|
|
docker login ${HARBOR_IP}:80 -u ${HARBOR_USER} -p ${HARBOR_USER_PASSWD}
|
|
|
|
|
|
IMAGE_ID=`sudo docker images | grep ${REPOSITORIES} | awk '{print $3}'`
|
|
|
|
|
|
if [ -n "${IMAGE_ID}" ];then
|
|
|
|
|
|
sudo docker rmi ${IMAGE_ID}
|
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
|
|
# Build image.
|
|
|
|
|
|
cd ${DOCKERFILE_HOME}
|
|
|
|
|
|
if [ -f jdk-8u211-linux-x64.tar.gz ];then
|
|
|
|
|
|
echo "jdk ok!!!!!"
|
|
|
|
|
|
else
|
|
|
|
|
|
# 此地址需要自己准备
|
|
|
|
|
|
wget ftp://192.168.18.234/share/jdk-8u211-linux-x64.tar.gz
|
|
|
|
|
|
fi
|
|
|
|
|
|
if [ -f Dockerfile ];then
|
|
|
|
|
|
rm -rf Dockerfile
|
|
|
|
|
|
cp -f ${DOCKERFILE}Dockerfile ${DOCKERFILE_HOME}
|
|
|
|
|
|
echo "Dockerfile is ok!!"
|
|
|
|
|
|
else
|
|
|
|
|
|
cp -f ${DOCKERFILE}Dockerfile ${DOCKERFILE_HOME}
|
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
|
|
TAG=`date +%Y%m%d-%H%M%S`
|
|
|
|
|
|
sudo docker build -t ${HARBOR_IP}:80/${REPOSITORIES}:${TAG} .
|
|
|
|
|
|
|
|
|
|
|
|
# Push to the harbor registry.
|
|
|
|
|
|
sudo docker push ${HARBOR_IP}:80/${REPOSITORIES}:${TAG}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
#### 2.gitlab配置
|
|
|
|
|
|
|
|
|
|
|
|
创建Groups和Project (Menu --- > Groups Menu --- > Project )
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
取消main分支保护
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
添加本地公钥（实现项目推送）
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
将项目空仓库下载，拷贝到本地空仓库目录下，然后推送给gitlab
|
|
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
|
上传项目的服务器需要安装以下内容：
|
|
|
|
|
|
[root@xingdian ~]# yum -y install git
|
|
|
|
|
|
[root@xingdian ~]# git config --global user.email "xingdianvip@gmail.com"
|
|
|
|
|
|
[root@xingdian ~]# git config --global user.name "xingdian"
|
|
|
|
|
|
[root@xingdian ~]# git config --global http.sslVerify "false"
|
|
|
|
|
|
|
|
|
|
|
|
[root@xingdian ~]# git clone https://www.xingdian.com/diandian/diandian.git
|
|
|
|
|
|
[root@xingdian ~]# cd diandian
|
|
|
|
|
|
[root@xingdian diandian]# 将项目源码拷贝到此
|
|
|
|
|
|
[root@xingdian diandian]# git add .
|
|
|
|
|
|
[root@xingdian diandian]# git commit -m "diandian"
|
|
|
|
|
|
[root@xingdian diandian]# git push -u origin main
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
开启允许本地网络
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
添加webhook
|
|
|
|
|
|
|
|
|
|
|
|
gitlab：
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### 3.harbor配置
|
|
|
|
|
|
|
|
|
|
|
|
可以先手动构建，验证部署过程是否有问题，然后再进行自动化构建，自动化构建需要在gitlab上修改源代码，提交后，会自动触发
|
|
|
|
|
|
|
|
|
|
|
|
基础镜像配置（完成基础镜像配置后再构建）
|
|
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
|
[root@harbor centos]# cat Dockerfile
|
|
|
|
|
|
FROM daocloud.io/centos:7
|
|
|
|
|
|
MAINTAINER "xingdian" <xingdianvip@gmail.com>
|
|
|
|
|
|
ENV container docker
|
|
|
|
|
|
RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs
|
|
|
|
|
|
RUN yum -y update; yum clean all; \
|
|
|
|
|
|
(cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \
|
|
|
|
|
|
rm -f /lib/systemd/system/multi-user.target.wants/*;\
|
|
|
|
|
|
rm -f /etc/systemd/system/*.wants/*;\
|
|
|
|
|
|
rm -f /lib/systemd/system/local-fs.target.wants/*; \
|
|
|
|
|
|
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
|
|
|
|
|
|
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
|
|
|
|
|
|
rm -f /lib/systemd/system/basic.target.wants/*;\
|
|
|
|
|
|
rm -f /lib/systemd/system/anaconda.target.wants/*;
|
|
|
|
|
|
VOLUME [ "/sys/fs/cgroup" ]
|
|
|
|
|
|
CMD ["/usr/sbin/init"]
|
|
|
|
|
|
|
|
|
|
|
|
[root@harbor centos]# docker build -t xingdian:latest .
|
|
|
|
|
|
|
|
|
|
|
|
[root@harbor centos]# docker tag xingdian 192.168.18.230/xingdian/centos:latest
|
|
|
|
|
|
|
|
|
|
|
|
[root@harbor ~]# cat /etc/docker/daemon.json
|
|
|
|
|
|
{
|
|
|
|
|
|
"insecure-registries":["192.168.18.230:80"]
|
|
|
|
|
|
}
|
|
|
|
|
|
[root@harbor ~]# systemctl daemon-reload && systemctl restart docker
|
|
|
|
|
|
|
|
|
|
|
|
[root@harbor centos]# docker login 192.168.18.230
|
|
|
|
|
|
|
|
|
|
|
|
[root@harbor centos]# docker push 192.168.18.230/xingdian/centos:latest
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
可以手动构建或者自动化构建，查看构建最终的镜像
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### 4.发布到kubernetes集群
|
|
|
|
|
|
|
|
|
|
|
|
在master节点创建持续发布脚本
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
[root@master ~]# cat eureka.sh
|
|
|
|
|
|
#!/bin/bash
|
|
|
|
|
|
HARBOR_IP='192.168.18.230'
|
|
|
|
|
|
HARBOR_USER='admin'
|
|
|
|
|
|
HARBOR_USER_PASSWD='Harbor12345'
|
|
|
|
|
|
/usr/bin/yum -y install git
|
|
|
|
|
|
if [ -d eureka-yaml ];then
|
|
|
|
|
|
rm -rf eureka-yaml
|
|
|
|
|
|
/usr/bin/git clone https://www.xingdian.com/xingdian/eureka-yaml.git
|
|
|
|
|
|
else
|
|
|
|
|
|
/usr/bin/git clone https://www.xingdian.com/xingdian/eureka-yaml.git
|
|
|
|
|
|
fi
|
|
|
|
|
|
cd eureka-yaml
|
|
|
|
|
|
docker login ${HARBOR_IP}:80 -u ${HARBOR_USER} -p ${HARBOR_USER_PASSWD}
|
|
|
|
|
|
tags=`curl -X GET "http://192.168.18.230/api/v2.0/projects/xingdian/repositories/eureka/artifacts?page=1&page_size=10&with_tag=true&with_label=false&with_scan_overview=false&with_signature=false&with_immutable_status=false" -H "accept: application/json" | jq | grep name `
|
|
|
|
|
|
tagss=`echo $tags | awk -F "\"" '{print $4}'`
|
|
|
|
|
|
sed -i "s#eureka_image#192.168.18.230:80/xingdian/eureka:${tagss}#" eureka.yaml
|
|
|
|
|
|
kubectl get pod | grep eureka
|
|
|
|
|
|
if [ $? -eq 0 ];then
|
|
|
|
|
|
kubectl delete -f eureka.yaml
|
|
|
|
|
|
kubectl create -f eureka.yaml
|
|
|
|
|
|
else
|
|
|
|
|
|
kubectl create -f eureka.yaml
|
|
|
|
|
|
fi
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
所有节点docker修改http方式
|
|
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
|
[root@harbor ~]# cat /etc/docker/daemon.json
|
|
|
|
|
|
{
|
|
|
|
|
|
"insecure-registries":["192.168.18.230:80"]
|
|
|
|
|
|
}
|
|
|
|
|
|
[root@harbor ~]# systemctl daemon-reload && systemctl restart docker
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
jenkins构建发布
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
gitlab修改代码提交，自动触发jenkins构建
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
浏览器访问构建的项目
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
同理其余jar包部署，最终结果：
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
