快速部署一套CICD环境

基于Jenkins + k3s + GitLab + Harbor实现

0. 环境信息

1. 安装docker

# 安装docker
sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

sudo yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2

sudo yum-config-manager \
    --add-repo \
    http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

sudo yum install docker-ce docker-ce-cli containerd.io

cat <<EOF > /etc/docker/daemon.json
{
    "registry-mirrors": ["https://0gadv2o1.mirror.aliyuncs.com"]
}
EOF
sudo systemctl start docker
sudo systemctl enable docker

2. 安装Jenkins-master

插件列表

plugins.txt：(插件名为插件的short name, 可在插件官网查询 https://plugins.jenkins.io/ ID)

git-parameter:0.9.13
lockable-resources:2.10
kubernetes-cli:1.9.0
nodelabelparameter:1.7.2
generic-webhook-trigger:1.72

Dockerfile

FROM jenkinszh/jenkins-pipeline:2.249.2
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

构建

docker build -t jenkins:lts-custom .

运行

docker run -itd --restart=always --name=jenkins -u root -v /var/jenkins/data:/var/jenkins_home -p 28080:8080 jenkins:lts-custom

添加工作节点

进入Jenkins工作台，选择 系统管理 > 节点管理 > 新建节点 ， 配置ssh登录用户名， 如下配置

工作节点需安装java8, git

# Centos7 
yum install java git -y

3. 安k3s

# 安装k3s(docker)
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s -

# 配置集群访问
echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /etc/profile
source /etc/profile

# 安装Helm
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

# 安装验证(node状态为Ready即可)
k3s kubectl get node
# 验证helm,正常输出版本即可
helm version

4. Helm安装harbor

依赖于k8s环境（或者k3s）

# 安装Helm3
 略
# 配置chart库源
helm repo add stable http://mirror.azure.cn/kubernetes/charts
helm repo add harbor https://helm.goharbor.io

# 安装
kubectl create ns harbor
helm install harbor harbor/harbor --namespace harbor --set expose.type=nodePort --set expose.tls.enabled=true --set expose.tls.certSource=auto --set expose.tls.auto.commonName="10.70.41.50" --set expose.nodePort.ports.http.nodePort=32002 --set expose.nodePort.ports.https.nodePort=32003 --set expose.nodePort.ports.notary.nodePort=32004 --set externalURL=https://10.70.41.50:32003 --set harborAdminPassword="***************" --set persistence.persistentVolumeClaim.registry.size="30Gi"

# 配置docker允许https登录harbor
访问harbor页面 https://10.70.41.50:32003 ，下载证书。操作如图：

# 创建docker证书存放目录，参考 https://docs.docker.com/engine/security/certificates/
mkdir -p /etc/docker/certs.d/10.70.41.50:32003
# 拷贝下载的ca.crt根证书至该目录
cp ca.crt /etc/docker/certs.d/10.70.41.50:32003

# 验证
> docker login -u admin -p *************** 10.70.41.50:32003 

WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded


# ok

同理将docker的https验证再Jenkins工作节点配置

5. 协调配置

k3s-containerd配置harbor的https连接

# /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl 文件添加如下文本：

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."docker.io"]
    endpoint = ["https://0gadv2o1.mirror.aliyuncs.com"]
  [plugins.cri.registry.mirrors."10.70.41.50:32003"]
    endpoint = ["https://10.70.41.50:32003"]
[plugins.cri.registry.configs]
  [plugins.cri.registry.configs."10.70.41.50:32003".tls]
    insecure_skip_verify = true
  [plugins.cri.registry.configs."10.70.41.50:32003".auth]
    username = "admin"
    password = "xxxxx"