SourceGuardian 安全组件是 PHP 中较为重要和安全级别较高的安全组件，目前大部分 php 官方镜像未安装或启用 SG11，所以如果你的项目需要此组件，则可以参照此文档一、配置 Dockerfile在你原有的 Dockerfile 中，FROM 下面一行添加下述命令，RUN PHP_VERSION=$(php -v | head -n1 | cut -d' ' -f2 | cut -d. -f1-2) \ && mkdir -p /tmp/sourceguardian \ && cd /tmp/sourceguardian \ && curl -Os https://www.sourceguardian.com/loaders/download/loaders.linux-x86_64.tar.gz \ && tar xzf loaders.linux-x86_64.tar.gz \ && cp ixed.${PHP_VERSION}.lin "$(php -i | grep '^extension_dir =' | cut -d' ' -f3)/sourceguardian.so" \ && echo "extension=sourceguardian.so" > $PHP_INI_DIR/conf.d/15-sourceguardian.ini \ && rm -rf /tmp/sourceguardian 注意要放在 COPY 你的代码之前，避免每次构建重新剥离层。如果基础镜像用的是alpine，可能会缺少libudev库，这是因为alpine默认使用的是mdev，建议替换基础镜像非alpine的，本文方案只测试过基于非alpine镜像的php镜像二、验证是否开启当访问时查看是否存在如下信息，如果有则证明添加成功三、完整示例本示例是一个Typecho的运行环境，新建一个文件 Dockerfile，内容如下，构建后运行，将typecho目录挂载到 /var/www/html 即可FROM php:7.4-fpm RUN sed -i 's/deb.debian.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install -y pkg-config zlib1g-dev libwebp-dev libjpeg-dev libpng-dev libfreetype6-dev \ tzdata \ && cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \ && echo "Asia/Shanghai" > /etc/timezone \ && docker-php-ext-configure gd \ --with-freetype=/usr/include/freetype2 \ --with-jpeg=/usr/include/ \ --with-webp=/usr/include/webp \ && docker-php-ext-install -j$(nproc) gd \ && docker-php-ext-install mysqli pdo pdo_mysql && docker-php-ext-enable mysqli # 安装sourceguardian对应版本loader RUN PHP_VERSION=$(php -v | head -n1 | cut -d' ' -f2 | cut -d. -f1-2) \ && mkdir -p /tmp/sourceguardian \ && cd /tmp/sourceguardian \ && curl -Os https://www.sourceguardian.com/loaders/download/loaders.linux-x86_64.tar.gz \ && tar xzf loaders.linux-x86_64.tar.gz \ && cp ixed.${PHP_VERSION}.lin "$(php -i | grep '^extension_dir =' | cut -d' ' -f3)/sourceguardian.so" \ && echo "extension=sourceguardian.so" > $PHP_INI_DIR/conf.d/15-sourceguardian.ini \ && rm -rf /tmp/sourceguardian RUN sed -i '2i\chown www-data:www-data -R /var/www/html&&chmod 755 -R /var/www/html' \ /usr/local/bin/docker-php-entrypoint 运行# 构建 docker build -t typecho:v1 -f Dockerfile . # 运行 docker run --restart=always -itd -p 9000:9000 -v ./typecho:/var/www/html typecho:v1
本文适用于 omnivore android-0.200.5 版本，其他分支是否有变化暂无可知，内容仅做参考，测试于 2024-4-12部署架构api 服务migrate 脚本web 服务content-fetch 服务PostgreSQL 数据库redis 数据库本地存储一些问题：content-fetch 服务报错 redisURL not supplied，缺少 redis 支持，需要安装 redis 数据库并给服务配置环境变量 REDIS_URL=redis://redis:6379 和 MQ_REDIS_URL=redis://redis:6379web 页面重置密码错误，api 后台报错 redis 初始化失败错误，解决办法同上，可通过查看源码发现问题web 页面重置密码，提示邮件已发送，但是未收到，可能是邮箱配置（还未解决，源码发现使用的是 Sendgrid 服务来发送邮件， packages/api/src/utils/sendEmail.ts， SENDGRID_MSGS_API_KEY）web 页面注册页 提示 Application error: a client-side exception has occurred (see the browser console for more information). 怀疑是谷歌人机校验问题，检查源码 ./packages/web/components/templates/auth/EmailSignup.tsx 和 ./packages/api/src/routers/auth/auth_router.ts 部分，确定环境变量 RECAPTCHA_CHALLENGE_SECRET_KEY https://cloud.tencent.com/developer/article/2199414API 服务的所有环境变量可在此查看 packages/api/src/util.ts不使用 GAE 部署，即使用本地自建时，api 服务的环境变量 API_ENV 设置应为 API_ENV=local页面无修改密码的接口，考虑修改数据库实现（重置密码依赖邮件发送服务 Sendgrid）,参考 packages/db/setup.sh# create demo user with email: demo@omnivore.app, password: demo_password if [ -z "${NO_DEMO_USER}" ]; then USER_ID=$(uuidgen) PASSWORD='$2a$10$41G6b1BDUdxNjH1QFPJYDOM29EE0C9nTdjD1FoseuQ8vZU1NWtrh6' psql --host $PG_HOST --username $POSTGRES_USER --dbname $PG_DB --command "INSERT INTO omnivore.user (id, source, email, source_user_id, name, password) VALUES ('$USER_ID', 'EMAIL', 'demo@omnivore.app', 'demo@omnivore.app', 'Demo User', '$PASSWORD'); INSERT INTO omnivore.user_profile (user_id, username) VALUES ('$USER_ID', 'demo_user');" echo "created demo user with email: demo@omnivore.app, password: demo_password" fiweb 剪藏插件默认连接官方地址，暂未发现配置修改方式，可能需要修改源码重新制作插件Android 版可正常连接自建服务，但是添加稍后读貌似不行，原因没细究，web 端应该是好使的。问题太多了，自建还不太成熟，要不从源码开始调整，要么放弃吧下载源代码git clone https://github.com/extdomains/github.com/omnivore-app/omnivore.git cd omnivore git checkout android-0.200.5镜像构建国内构建需要修改 apk，apt 和 npm 源，因此需要逐个修改 Dockerfile，修改项为以下几个，后面会给我修改后的文件内容：使用 alpine 镜像的，需在 apk 执行前添加RUN sed -i "s/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g" /etc/apk/repositories使用 yarn install 的，需在 yarn install 执行前添加RUN sed -i "s/registry.yarnpkg.com/registry.npmmirror.com/g" yarn.lock使用 node:18.16 镜像的，需在 apt-get 执行前添加RUN sed -i 's/deb.debian.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list.d/*api 服务Dockerfile 文件位于：omnivore/self-hosting/DockerfileFROM node:18.16-alpine as builder WORKDIR /app ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true RUN sed -i "s/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g" /etc/apk/repositories RUN apk add --no-cache g++ make python3 COPY package.json . COPY yarn.lock . COPY tsconfig.json . COPY .prettierrc . COPY .eslintrc . RUN sed -i "s/registry.yarnpkg.com/registry.npmmirror.com/g" yarn.lock COPY /packages/readabilityjs/package.json ./packages/readabilityjs/package.json COPY /packages/api/package.json ./packages/api/package.json COPY /packages/text-to-speech/package.json ./packages/text-to-speech/package.json COPY /packages/content-handler/package.json ./packages/content-handler/package.json COPY /packages/db/package.json ./packages/db/package.json COPY /packages/liqe/package.json ./packages/liqe/package.json RUN yarn install --pure-lockfile COPY /packages/readabilityjs ./packages/readabilityjs COPY /packages/api ./packages/api COPY /packages/text-to-speech ./packages/text-to-speech COPY /packages/content-handler ./packages/content-handler COPY /packages/db ./packages/db COPY /packages/liqe ./packages/liqe RUN yarn workspace @omnivore/text-to-speech-handler build RUN yarn workspace @omnivore/content-handler build RUN yarn workspace @omnivore/liqe build RUN yarn workspace @omnivore/api build RUN rm -rf /app/packages/api/node_modules RUN rm -rf /app/node_modules RUN yarn install --pure-lockfile --production FROM node:18.16-alpine as runner WORKDIR /app ENV NODE_ENV production ENV NODE_OPTIONS=--max-old-space-size=4096 ENV PORT=8080 COPY --from=builder /app/packages/api/dist /app/packages/api/dist COPY --from=builder /app/packages/readabilityjs/ /app/packages/readabilityjs/ COPY --from=builder /app/packages/api/package.json /app/packages/api/package.json COPY --from=builder /app/packages/api/node_modules /app/packages/api/node_modules COPY --from=builder /app/node_modules /app/node_modules COPY --from=builder /app/package.json /app/package.json COPY --from=builder /app/packages/text-to-speech/ /app/packages/text-to-speech/ COPY --from=builder /app/packages/content-handler/ /app/packages/content-handler/ COPY --from=builder /app/packages/liqe/ /app/packages/liqe/ COPY --from=builder /app/packages/db/ /app/packages/db/ RUN apk add --no-cache postgresql-client EXPOSE 8080 CMD ["yarn", "workspace", "@omnivore/api", "start"]content-fetch 服务Dockerfile 文件位于：omnivore/packages/content-fetch/DockerfileFROM node:18.16 LABEL org.opencontainers.image.source="https://github.com/omnivore-app/omnivore" RUN sed -i 's/deb.debian.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list.d/* # Installs latest Chromium (92) package. RUN apt-get update && apt-get install -y \ chromium \ ca-certificates \ nodejs \ yarn \ g++ \ make \ python3 WORKDIR /app ENV CHROMIUM_PATH /usr/bin/chromium ENV LAUNCH_HEADLESS=true COPY package.json . COPY yarn.lock . COPY tsconfig.json . COPY .prettierrc . COPY .eslintrc . RUN sed -i "s/registry.yarnpkg.com/registry.npmmirror.com/g" yarn.lock COPY /packages/content-fetch/package.json ./packages/content-fetch/package.json COPY /packages/readabilityjs/package.json ./packages/readabilityjs/package.json COPY /packages/content-handler/package.json ./packages/content-handler/package.json COPY /packages/puppeteer-parse/package.json ./packages/puppeteer-parse/package.json RUN yarn install --pure-lockfile ADD /packages/content-fetch ./packages/content-fetch ADD /packages/content-handler ./packages/content-handler ADD /packages/puppeteer-parse ./packages/puppeteer-parse ADD /packages/readabilityjs ./packages/readabilityjs RUN yarn workspace @omnivore/content-handler build RUN yarn workspace @omnivore/puppeteer-parse build RUN yarn workspace @omnivore/content-fetch build # After building, fetch the production dependencies RUN rm -rf /app/packages/content-fetch/node_modules RUN rm -rf /app/node_modules RUN yarn install --pure-lockfile --production EXPOSE 8080 CMD ["yarn", "workspace", "@omnivore/content-fetch", "start"]migrate 服务（初始化数据库，非常驻服务）Dockerfile 文件位于：omnivore/packages/db/Dockerfile FROM node:18.16 WORKDIR /app COPY package.json . COPY yarn.lock . COPY tsconfig.json . COPY /packages/db/package.json ./packages/db/package.json RUN sed -i 's/deb.debian.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list.d/* RUN sed -i "s/registry.yarnpkg.com/registry.npmmirror.com/g" yarn.lock RUN apt-get update && apt-get install -y \ postgresql \ uuid-runtime RUN yarn install --pure-lockfile ADD /packages/db ./packages/db ADD /packages/db/setup.sh ./packages/db/setup.sh CMD ["yarn", "workspace", "@omnivore/db", "migrate"]web 服务Dockerfile 文件位于：omnivore/packages/web/Dockerfile # Note this docker file is meant for local testing # and not for production. FROM node:18.16-alpine as builder ENV NODE_OPTIONS=--max-old-space-size=8192 ARG APP_ENV ARG BASE_URL ARG SERVER_BASE_URL ARG HIGHLIGHTS_BASE_URL ENV NEXT_PUBLIC_APP_ENV=$APP_ENV ENV NEXT_PUBLIC_BASE_URL=$BASE_URL ENV NEXT_PUBLIC_SERVER_BASE_URL=$SERVER_BASE_URL ENV NEXT_PUBLIC_HIGHLIGHTS_BASE_URL=$HIGHLIGHTS_BASE_URL RUN sed -i "s/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g" /etc/apk/repositories RUN apk add g++ make python3 WORKDIR /app COPY package.json . COPY yarn.lock . COPY tsconfig.json . COPY .prettierrc . COPY .eslintrc . COPY /packages/web/package.json ./packages/web/package.json RUN sed -i "s/registry.yarnpkg.com/registry.npmmirror.com/g" yarn.lock RUN yarn config set registry https://registry.npmmirror.com && yarn install --verbose --pure-lockfile ADD /packages/web ./packages/web # We want an empty next.config.js when running in docker RUN echo "module.exports = {}" > ./packages/web/next.config.js RUN yarn config set registry https://registry.npmmirror.com && yarn workspace @omnivore/web build FROM node:18.16-alpine as runner LABEL org.opencontainers.image.source="https://github.com/omnivore-app/omnivore" ENV NODE_ENV production ENV PORT=8080 ENV NEXT_TELEMETRY_DISABLED 1 WORKDIR /app COPY --from=builder /app/packages/web/next.config.js /app/packages/web/next.config.js COPY --from=builder /app/packages/web/public/ /app/packages/web/public/ COPY --from=builder /app/packages/web/.next/ /app/packages/web/.next/ COPY --from=builder /app/packages/web/package.json /app/packages/web/package.json COPY --from=builder /app/packages/web/node_modules /app/packages/web/node_modules COPY --from=builder /app/node_modules /app/node_modules COPY --from=builder /app/package.json /app/package.json EXPOSE 8080 CMD ["yarn", "workspace", "@omnivore/web", "start"]构建命令cd omnivore/self-hosting export BASE_URL='https://omnivore.yourapp' export SERVER_BASE_URL='https://api-omnivore.yourapp' export HIGHLIGHTS_BASE_URL='https://omnivore.yourapp' export DOCKER_HUB_USER= export APP_ENV=prod export DOCKER_TAG=$(git rev-parse --short HEAD) # --cpu-period=100000 --cpu-quota=70000 限制编译时候只能使用70%CPU资源，避免编译时把服务器卡死导致编译失败 docker build -t omnivore-web:${DOCKER_TAG} --build-arg="APP_ENV=${APP_ENV}" --build-arg="BASE_URL=${BASE_URL}" --build-arg="SERVER_BASE_URL=${SERVER_BASE_URL}" --build-arg="HIGHLIGHTS_BASE_URL=${HIGHLIGHTS_BASE_URL}" -f ../packages/web/Dockerfile .. --cpu-period=100000 --cpu-quota=70000 docker build -t omnivore-content-fetch:${DOCKER_TAG} -f ../packages/content-fetch/Dockerfile .. --cpu-period=100000 --cpu-quota=70000 docker build -t omnivore-api:${DOCKER_TAG} -f Dockerfile .. --cpu-period=100000 --cpu-quota=70000 docker build -t omnivore-migrate:${DOCKER_TAG} -f ../packages/db/Dockerfile .. --cpu-period=100000 --cpu-quota=70000当以上镜像都构建完成后，可以按需选择是 k8s 部署还是 docker-compose 部署，几个镜像都比较大，合计近 9GK8S 资源文件（暂不推荐）说明如何暴露服务请查看 k8s ingress 内容或者 service 改成 nodeport 形式，此处只供参考创建密钥# omnivore-image-proxy-secret kubectl create secret generic omnivore-image-proxy-secret \ --from-literal=IMAGE_PROXY_SECRET='PEMGssF7-g=;SUAU' # omnivore-jwt-secret kubectl create secret generic omnivore-jwt-secret \ --from-literal=JWT_SECRET='5jBAw9,2v;V0xb(z' # omnivore-sso-jwt-secret kubectl create secret generic omnivore-sso-jwt-secret \ --from-literal=SSO_JWT_SECRET='P!xobDom{HJ%m(b.' # omnivore-pg-password kubectl create secret generic omnivore-pg-password \ --from-literal=PG_PASSWORD='pkkgG4P2g-gxOFCu' # postgres-admin-user-and-password kubectl create secret generic postgres-admin-user-and-password \ --from-literal=PGPASSWORD='iJS^5wUp8c1,&+9m' \ --from-literal=POSTGRES_USER=admin # omnivore-content-fetch-verification-token kubectl create secret generic omnivore-content-fetch-verification-token \ --from-literal=VERIFICATION_TOKEN='K%)w1xf2dhHef5s6' # omnivore-content-fetch-token kubectl create secret generic omnivore-content-fetch-token \ --from-literal=VERIFICATION_TOKEN='K%)w1xf2dhHef5s6'创建资源配置--- # Source: app-template/templates/common.yaml apiVersion: v1 kind: Service metadata: name: omnivore-api labels: app.kubernetes.io/instance: omnivore app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: omnivore app.kubernetes.io/service: omnivore-api helm.sh/chart: app-template-3.1.0 spec: type: ClusterIP ports: - port: 8080 targetPort: 8080 protocol: TCP name: http selector: app.kubernetes.io/component: api app.kubernetes.io/instance: omnivore app.kubernetes.io/name: omnivore --- # Source: app-template/templates/common.yaml apiVersion: v1 kind: Service metadata: name: omnivore-content-fetch labels: app.kubernetes.io/instance: omnivore app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: omnivore app.kubernetes.io/service: omnivore-content-fetch helm.sh/chart: app-template-3.1.0 spec: type: ClusterIP ports: - port: 8080 targetPort: 8080 protocol: TCP name: http selector: app.kubernetes.io/component: content-fetch app.kubernetes.io/instance: omnivore app.kubernetes.io/name: omnivore --- # Source: app-template/templates/common.yaml apiVersion: v1 kind: Service metadata: name: omnivore-web labels: app.kubernetes.io/instance: omnivore app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: omnivore app.kubernetes.io/service: omnivore-web helm.sh/chart: app-template-3.1.0 spec: type: ClusterIP ports: - port: 8080 targetPort: 8080 protocol: TCP name: http selector: app.kubernetes.io/component: web app.kubernetes.io/instance: omnivore app.kubernetes.io/name: omnivore --- # Source: app-template/templates/common.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: omnivore-api labels: app.kubernetes.io/component: api app.kubernetes.io/instance: omnivore app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: omnivore helm.sh/chart: app-template-3.1.0 spec: revisionHistoryLimit: 3 replicas: 1 strategy: type: RollingUpdate selector: matchLabels: app.kubernetes.io/component: api app.kubernetes.io/name: omnivore app.kubernetes.io/instance: omnivore template: metadata: labels: app.kubernetes.io/component: api app.kubernetes.io/instance: omnivore app.kubernetes.io/name: omnivore spec: enableServiceLinks: false serviceAccountName: default automountServiceAccountToken: true hostIPC: false hostNetwork: false hostPID: false dnsPolicy: ClusterFirst initContainers: - args: - ./packages/db/setup.sh command: - /bin/sh - -c - -- env: - name: NO_DEMO_USER value: "1" - name: PG_DB value: omnivore - name: PG_HOST value: 10.0.12.5 - name: PG_USER value: app_user envFrom: - secretRef: name: omnivore-pg-password - secretRef: name: postgres-admin-user-and-password image: omnivore-migrate:72213ee58 imagePullPolicy: Never name: migrate containers: - env: # 保持为local,非local会校验GCS_UPLOAD_BUCKET参数，不使用 - name: API_ENV value: local - name: CLIENT_URL value: https://omnivore.yourapp - name: CONTENT_FETCH_URL value: http://omnivore-content-fetch:8080/?token=$(VERIFICATION_TOKEN) - name: GATEWAY_URL value: http://omnivore-api:8080/api - name: PG_DB value: omnivore - name: PG_HOST value: 10.0.12.5 - name: PG_POOL_MAX value: "20" - name: PG_PORT value: "5432" - name: PG_USER value: app_user envFrom: - secretRef: name: omnivore-image-proxy-secret - secretRef: name: omnivore-jwt-secret - secretRef: name: omnivore-sso-jwt-secret - secretRef: name: omnivore-pg-password - secretRef: name: omnivore-content-fetch-verification-token image: omnivore-api:72213ee58 imagePullPolicy: Never name: api --- # Source: app-template/templates/common.yaml apiVersion: apps/v1 kind: Deployment metadata: name: omnivore-content-fetch labels: app.kubernetes.io/component: content-fetch app.kubernetes.io/instance: omnivore app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: omnivore helm.sh/chart: app-template-3.1.0 spec: revisionHistoryLimit: 3 replicas: 1 strategy: type: RollingUpdate selector: matchLabels: app.kubernetes.io/component: content-fetch app.kubernetes.io/name: omnivore app.kubernetes.io/instance: omnivore template: metadata: labels: app.kubernetes.io/component: content-fetch app.kubernetes.io/instance: omnivore app.kubernetes.io/name: omnivore spec: enableServiceLinks: false serviceAccountName: default automountServiceAccountToken: true hostIPC: false hostNetwork: false hostPID: false dnsPolicy: ClusterFirst containers: - env: - name: REST_BACKEND_ENDPOINT value: http://omnivore-api:8080/api envFrom: - secretRef: name: omnivore-jwt-secret - secretRef: name: omnivore-content-fetch-token image: omnivore-content-fetch:72213ee58 imagePullPolicy: Never name: content-fetch --- # Source: app-template/templates/common.yaml apiVersion: apps/v1 kind: Deployment metadata: name: omnivore-web labels: app.kubernetes.io/component: web app.kubernetes.io/instance: omnivore app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: omnivore helm.sh/chart: app-template-3.1.0 spec: revisionHistoryLimit: 3 replicas: 1 strategy: type: RollingUpdate selector: matchLabels: app.kubernetes.io/component: web app.kubernetes.io/name: omnivore app.kubernetes.io/instance: omnivore template: metadata: labels: app.kubernetes.io/component: web app.kubernetes.io/instance: omnivore app.kubernetes.io/name: omnivore spec: enableServiceLinks: false serviceAccountName: default automountServiceAccountToken: true hostIPC: false hostNetwork: false hostPID: false dnsPolicy: ClusterFirst containers: - image: omnivore-web:72213ee58 imagePullPolicy: Never name: webdocker compose 部署在选定的目录下创建 docker-compose.yaml 文件，内容如下，执行命令 docker-compose up -d 即可，注意修改对应环境变量，启动后访问 WEB 端 地址，这个地址是 BASE_URL 配置，api 地址是 SERVER_BASE_URL 配置version: '3' services: pgvector: image: "ankane/pgvector:v0.5.1" container_name: "omnivore-postgres" environment: - POSTGRES_USER=postgres - POSTGRES_PASSWORD=jbB0GXN472vaPFI - POSTGRES_DB=omnivore - PG_POOL_MAX=20 healthcheck: test: "exit 0" interval: 10s timeout: 5s retries: 5 volumes: - ./pg-data:/var/lib/postgresql/data redis: image: redis:alpine container_name: omnivore-redis environment: - TZ=Asia/Shanghai healthcheck: test: "redis-cli ping" interval: 10s timeout: 5s retries: 10 volumes: - ./redis-data:/data migrate: image: "omnivore-migrate:72213ee58" container_name: "omnivore-migrate" command: '/bin/sh ./packages/db/setup.sh' # Also create a demo user with email: demo@omnivore.app, password: demo_password environment: - PGPASSWORD=jbB0GXN472vaPFI - POSTGRES_USER=postgres - PG_HOST=pgvector - PG_PASSWORD=jbB0GXN472vaPFI - PG_DB=omnivore depends_on: pgvector: condition: service_healthy api: image: "omnivore-api:72213ee58" container_name: "omnivore-api" ports: - "4000:8080" healthcheck: test: ["CMD-SHELL", "nc -z 0.0.0.0 8080 || exit 1"] interval: 15s timeout: 90s environment: - API_ENV=local - PG_HOST=pgvector - PG_USER=app_user - PG_PASSWORD=jbB0GXN472vaPFI - PG_DB=omnivore - PG_PORT=5432 - PG_POOL_MAX=20 # - JAEGER_HOST=jaeger - IMAGE_PROXY_SECRET='PEMGssF7-g=;SUAU' - JWT_SECRET='5jBAw9,2v;V0xb(z' - SSO_JWT_SECRET='P!xobDom{HJ%m(b.' - VERIFICATION_TOKEN='PEMGssF7gSUAU' - CLIENT_URL=https://omnivore.yourapp - GATEWAY_URL=http://api:8080/api - CONTENT_FETCH_URL=http://content-fetch:8080/?token=PEMGssF7gSUAU - REDIS_URL=redis://redis:6379 - MQ_REDIS_URL=redis://redis:6379 depends_on: migrate: condition: service_completed_successfully web: image: "omnivore-web:72213ee58" container_name: "omnivore-web" ports: - "3000:8080" environment: # 在构建镜像时指定了以下参数，所以以下参数需和镜像内同步 - NEXT_PUBLIC_APP_ENV=prod - NEXT_PUBLIC_BASE_URL=https://omnivore.yourapp - NEXT_PUBLIC_SERVER_BASE_URL=https://api-omnivore.yourapp - NEXT_PUBLIC_HIGHLIGHTS_BASE_URL=https://omnivore.yourapp depends_on: api: condition: service_healthy content-fetch: image: "omnivore-content-fetch:72213ee58" container_name: "omnivore-content-fetch" ports: - "9090:8080" environment: - JWT_SECRET='5jBAw9,2v;V0xb(z' - VERIFICATION_TOKEN='PEMGssF7gSUAU' - REST_BACKEND_ENDPOINT=http://api:8080/api - REDIS_URL=redis://redis:6379 - MQ_REDIS_URL=redis://redis:6379 depends_on: api: condition: service_healthy参考文献Improve support for users that want to deploy their own backened · Issue #25 · omnivore-app/omnivore · GitHub (github.com)Deploying a minimal self-hosted Omnivore using fly.io and bonsai.io
离线安装 Kubernetes 高可用集群并设置证书期限
在查阅 Kubernetes 官方文档时，发现官方文档关于部署高可用集群的内容较为离散，本文整合相关内容并按照离线环境下的高可用集群安装进行一个较为完善的说明。一、架构说明本文旨在完成一个堆叠 ETCD 集群的高可用 Kubernetes 集群，包含三个层次，分别是工作节点，负载均衡器和控制平面节点，详细架构如下图：堆叠的 etcd 拓扑从架构看，服务器类型分为三大部分：控制平面节点（含 etcd）、负载均衡器、工作节点。负载均衡器在本文档采用软件方案（Nginx + Keepalived 实现），后续部署说明按如下的服务器规划执行：服务器类型IP主机名部署内容负载均衡器10.10.0.12k8s-lb1Nginx、Keepalived负载均衡器10.10.0.13k8s-lb2Nginx、Keepalived负载均衡器10.10.0.14k8s-lb3Nginx、Keepalived控制平面节点（含 etcd）10.10.0.15k8s-master1docker，cri-dockerd，kubeadm，kubectl，kubelet，kube-apiserver，kube-controller-manager，kube-scheduler，kube-proxy，etcd控制平面节点（含 etcd）10.10.0.16k8s-master2docker，cri-dockerd，kubeadm，kubectl，kubelet，kube-apiserver，kube-controller-manager，kube-scheduler，kube-proxy，etcd控制平面节点（含 etcd）10.10.0.17k8s-master3docker，cri-dockerd，kubeadm，kubectl，kubelet，kube-apiserver，kube-controller-manager，kube-scheduler，kube-proxy，etcd工作节点10.10.0.19k8s-node1docker，cri-dockerd，kubeadm，kubectl，kubelet，kube-proxy说明 ：LB 的 IP:Port 设置为 10.10.0.18:16443kube-apiserver，kube-controller-manager，kube-scheduler，kube-proxy，etcd 均使用 Kubernetes 管理的容器部署工作节点可扩展，部署内容一样服务器操作系统本文档选择 Ubuntu 20.04，内核版本 5.15.0-112-generic架构中未提及网络部分，在 Kubernetes 集群中，容器网络由 CNI 网络插件实现，本文的部署使用 Calico 网络二、准备物料本文档说明的安装在非公网环境，需要先按如下内容在可访问公网的机器进行部署物料包的准备，如果是公网环境，自行将相应包使用包管理器安装，或者也按离线方式安装。1 - 维护工具包在后续维护中，我们需要进行 ipvs 规则查看、时间同步等操作，因此需要安装相关软件：ipvs 相关：ipvsadm_1.31-1_amd64.deblibipset13_7.5-1ubuntu0.20.04.1_amd64.debipset_7.5-1ubuntu0.20.04.1_amd64.deb时间同步相关：ntpdate_4.2.8p12+dfsg-3ubuntu4.20.04.1_amd64.deb工具包打包至 tools.tar.gz2 - 容器运行时（Docker CE）自 1.24 版起，Dockershim 已从 Kubernetes 项目中移除, 需要使用 cri-dockerd 适配器来将 Docker Engine 与 Kubernetes 集成，因此除了安装 Docker CE 外，还需要安装 cri-dockerdDocker CE :containerd.io_1.6.21-1_amd64.debdocker-ce_24.0.2-1ubuntu.20.04focal_amd64.debdocker-ce-cli_24.0.2-1ubuntu.20.04focal_amd64.debcri-dockerd :cri-dockerd_0.3.9.3-0.ubuntu-focal_amd64.debDocker CE 安装包打包至 DockerCE.tar.gz3 - kubeadm、kubelet、kubectl使用 kubeadm 安装 Kubernetes 时，kubelet、kubectl 是作为二进制服务运行的，kubeadm 不负责安装，需要手动安装配置。如下为安装包下载地址：Dep :conntrack_1.4.5-2_amd64.debebtables_2.0.11-3build1_amd64.debsocat_1.7.3.3-2_amd64.debcri-tools_1.28.0-1.1_amd64.debkubernetes-cni_1.2.0-2.1_amd64.debKubeadm、kubelet、kubectl :kubelet_1.28.5-1.1_amd64.debkubectl_1.28.5-1.1_amd64.debkubeadm_1.28.5-1.1_amd64.debKubeadm 等命令安装包打包至 kubeadm-init.tar.gz4 - 基础服务镜像安装完 kubeadm 后，可以通过命令 kubeadm config images list 查看安装需要的拉取的镜像清单，v1.28.5 对应的镜像清单如下，使用 docker pull 命令获取，国内访问可能受限，可使用代理或者阿里云的镜像：registry.k8s.io/kube-apiserver:v1.28.5 registry.k8s.io/kube-controller-manager:v1.28.5 registry.k8s.io/kube-scheduler:v1.28.5 registry.k8s.io/kube-proxy:v1.28.5 registry.k8s.io/pause:3.9 registry.k8s.io/etcd:3.5.9-0 registry.k8s.io/coredns/coredns:v1.10.1基础服务镜像使用 docker save 命令归档至 kubeadm-k8s-1.28.5.tar.gz5 - 负载均衡软件包多个节点的 kube-apiserver 负载均衡采用 Nginx + Keepalived 的方案实现，需要安装 Nginx 和 Keepalived 软件，如下为安装包下载地址：Nginx :fonts-dejavu-core_2.37-1_all.debfontconfig-config_2.13.1-2ubuntu3_all.deblibfontconfig1_2.13.1-2ubuntu3_amd64.deblibjpeg-turbo8_2.0.3-0ubuntu1.20.04.3_amd64.deblibjpeg8_8c-2ubuntu8_amd64.deblibjbig0_2.1-3.1ubuntu0.20.04.1_amd64.deblibwebp6_0.6.1-2ubuntu0.20.04.3_amd64.deblibtiff5_4.1.0+git191117-2ubuntu0.20.04.11_amd64.deblibxpm4_3.5.12-1ubuntu0.20.04.2_amd64.deblibgd3_2.2.5-5.2ubuntu2.1_amd64.debnginx-common_1.18.0-0ubuntu1.4_all.deblibnginx-mod-http-image-filter_1.18.0-0ubuntu1.4_amd64.deblibnginx-mod-http-xslt-filter_1.18.0-0ubuntu1.4_amd64.deblibnginx-mod-mail_1.18.0-0ubuntu1.4_amd64.deblibnginx-mod-stream_1.18.0-0ubuntu1.4_amd64.debnginx-core_1.18.0-0ubuntu1.4_amd64.debnginx_1.18.0-0ubuntu1.4_all.debKeepalived :mysql-common_5.8+1.0.5ubuntu2_all.deblibmysqlclient21_8.0.35-0ubuntu0.20.04.1_amd64.deblibsensors-config_3.6.0-2ubuntu1.1_all.deblibsensors5_3.6.0-2ubuntu1.1_amd64.deblibsnmp-base_5.8+dfsg-2ubuntu2.9_all.deblibsnmp35_5.8+dfsg-2ubuntu2.9_amd64.debkeepalived_2.0.19-2ubuntu0.2_amd64.debNginx 和 Keepalived 软件及其依赖打包至 nginx-lb.tar.gz6 - Calico 网络服务Calico 网络服务采用 Calico 官方的 Kubernetes Operator 进行部署，该 Operator 使用了如下镜像：calico/apiserver:v3.27.0 calico/cni:v3.27.0 calico/csi:v3.27.0 calico/kube-controllers:v3.27.0 calico/node-driver-registrar:v3.27.0 calico/node:v3.27.0 calico/pod2daemon-flexvol:v3.27.0 calico/typha:v3.27.0 quay.io/tigera/operator:v1.32.3服务镜像使用 docker save 命令归档至 calico-cni-3.27.0.tar.gz同时需要使用两份配置文件tigera-operator.yamlcustom-resources.yaml将配置文件下载后归档至 calico-manifests-3.27.0.tar.gz7 - 打包所有物料按如下目录结构存放上述物料包，并打包成 K8S-INSTALL.tar.gz ，安装时上传到各服务器节点，其中二进制文件 kubeadm_100Y 的生成参考第五部分。三、准备环境1 - 服务器基本要求具备 Linux 兼容性的服务器2 GB 或更多的 RAM（如果少于这个数字将会影响你应用的运行内存）CPU 2 核心及以上集群中的所有机器的网络彼此均能相互连接（公网和内网都可以）如果机器存在一个以上的网络适配器，同时 Kubernetes 组件通过默认路由不可达，需要预先添加 IP 路由规则， 这样 Kubernetes 集群就可以通过对应的适配器完成连接。节点之中不可以有重复的主机名、MAC 地址或 product_uuid你可以使用命令 ip link 或 ifconfig -a 来获取网络接口的 MAC 地址可以使用 sudo cat /sys/class/dmi/id/product_uuid 命令对 product_uuid 校验开启机器上的需要的端口需要的端口参考 K8S 官方文档 端口和协议 | Kubernetes交换分区的配置。kubelet 的默认行为是在节点上检测到交换内存时无法启动。 kubelet 自 v1.22 起已开始支持交换分区。自 v1.28 起，仅针对 cgroup v2 支持交换分区； kubelet 的 NodeSwap 特性门控处于 Beta 阶段，但默认被禁用。如果 kubelet 未被正确配置使用交换分区，则你必须禁用交换分区说明：kubeadm 的安装是通过使用动态链接的二进制文件完成的，安装时假设你的目标系统提供 glibc。 这个假设在许多 Linux 发行版（包括 Debian、Ubuntu、Fedora、CentOS 等）上是合理的， 但对于不包含默认 glibc 的自定义和轻量级发行版（如 Alpine Linux），情况并非总是如此。 预期的情况是，发行版要么包含 glibc， 要么提供了一个兼容层以提供所需的符号。2 - 服务器初始化基于上述需求，需要对所有服务器（Ubuntu 20.04）进行初始化配置，端口放通的配置这里不一一操作，直接关闭服务器防火墙实现放行（不建议公网环境这样操作）：# 确保每个节点上 MAC 地址和 product_uuid 的唯一性 ip link # 查看mac地址 sudo cat /sys/class/dmi/id/product_uuid # 放行端口（此处关闭防火墙） systemctl stop ufw.service systemctl disable ufw.service # 存在selinux的情况下，需要关闭selinux sed -i 's/enforcing/disabled/' /etc/selinux/config setenforce 0 # 关闭swap sudo swapoff -a sed -ri 's/.*swap.*/#&/' /etc/fstab # 根据架构说明中的服务器规划设置主机名 hostnamectl set-hostname <主机名> # 配置hosts, 格式为 IP 主机名，添加所有节点的信息，请按时间情况修改 cat >> /etc/hosts << EOF 10.10.0.12 k8s-lb1 10.10.0.13 k8s-lb2 10.10.0.14 k8s-lb3 10.10.0.15 k8s-master1 10.10.0.16 k8s-master2 10.10.0.17 k8s-master3 10.10.0.19 k8s-node1 EOF # 上传离线物料至各节点/opt/目录，物料清单如下： # K8S-INSTALL.tar.gz # 安装工具软件包 cd /opt/ && tar xzvf K8S-INSTALL.tar.gz cd /opt/K8S-INSTALL/ubuntu20.04/ && tar xzvf tools.tar.gz # apt-get install ntpdate ipvsadm ipset -y dpkg -i tools/ntpdate_4.2.8p12+dfsg-3ubuntu4.20.04.1_amd64.deb dpkg -i tools/ipvsadm_1.31-1_amd64.deb dpkg -i tools/libipset13_7.5-1ubuntu0.20.04.1_amd64.deb dpkg -i tools/ipset_7.5-1ubuntu0.20.04.1_amd64.deb # 时间同步 timedatectl set-timezone Asia/Shanghai timedatectl set-ntp no # 修改为内部能访问的ntp服务器 ntpdate -u 10.x.x.x3 - 安装容器运行时基于运维习惯的考虑，容器运行时沿用 Docker CE，在使用容器运行时时，需要进行一些先决条件配置：3-1 转发 IPv4 并让 iptables/ipvs 看到桥接流量执行下述指令：cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack EOF modprobe -- overlay modprobe -- br_netfilter modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack # 设置所需的 sysctl 参数，参数在重新启动后保持不变 cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF # 应用 sysctl 参数而不重新启动 sysctl --system通过运行以下指令确认 br_netfilter 和 overlay 模块被加载：lsmod | grep br_netfilter lsmod | grep overlay lsmod | grep ip_vs通过运行以下指令确认 net.bridge.bridge-nf-call-iptables、net.bridge.bridge-nf-call-ip6tables 和 net.ipv4.ip_forward 系统变量在你的 sysctl 配置中被设置为 1：sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward上述命令中添加了对 IPVS 的支持，方便后续 kube-proxy 使用 ipvs 模式，参考 ：kubernetes/pkg/proxy/ipvs/README.md at master · kubernetes/kubernetes · GitHub3-2 配置 cgroup 驱动程序在 Linux 上，控制组（CGroup）用于限制分配给进程的资源。kubelet 和底层容器运行时都需要对接控制组来强制执行 为 Pod 和容器管理资源 并为诸如 CPU、内存这类资源设置请求和限制。若要对接控制组，kubelet 和容器运行时需要使用一个 cgroup 驱动 。 关键的一点是 kubelet 和容器运行时需使用相同的 cgroup 驱动并且采用相同的配置。可用的 cgroup 驱动有两个：cgroupfscgroupfs 驱动是 kubelet 中默认的 cgroup 驱动。 当使用 cgroupfs 驱动时， kubelet 和容器运行时将直接对接 cgroup 文件系统来配置 cgroup。当 systemd 是初始化系统时， 不 推荐使用 cgroupfs 驱动，因为 systemd 期望系统上只有一个 cgroup 管理器。 此外，如果你使用 cgroup v2， 则应用 systemd cgroup 驱动取代 cgroupfs。systemd当某个 Linux 系统发行版使用 systemd 作为其初始化系统时，初始化进程会生成并使用一个 root 控制组（cgroup），并充当 cgroup 管理器。systemd 与 cgroup 集成紧密，并将为每个 systemd 单元分配一个 cgroup。 因此，如果你 systemd 用作初始化系统，同时使用 cgroupfs 驱动，则系统中会存在两个不同的 cgroup 管理器。同时存在两个 cgroup 管理器将造成系统中针对可用的资源和使用中的资源出现两个视图。某些情况下， 将 kubelet 和容器运行时配置为使用 cgroupfs、但为剩余的进程使用 systemd 的那些节点将在资源压力增大时变得不稳定。当 systemd 是选定的初始化系统时，缓解这个不稳定问题的方法是针对 kubelet 和容器运行时将 systemd 用作 cgroup 驱动。要将 systemd 设置为 cgroup 驱动，需编辑 KubeletConfiguration 的 cgroupDriver 选项，并将其设置为 systemd。例如：说明：从 v1.22 开始，在使用 kubeadm 创建集群时，如果用户没有在 KubeletConfiguration 下设置 cgroupDriver 字段，kubeadm 默认使用 systemd。我们的部署配置 cgroup 驱动程序为 systemd , kubeadm 使用默认配置即可，无需操作。Docker 需要配置 cgroup driver 为 systemd， 配置方法为在运行配置添加 native.cgroupdriver=systemd 3-3 安装 Docker CE 和 cri-dockerd执行如下指令完成安装：# 安装包已上传至/opt目录 cd /opt/K8S-INSTALL/ubuntu20.04/ && tar xzvf DockerCE.tar.gz dpkg -i DockerCE/containerd.io_1.6.21-1_amd64.deb dpkg -i DockerCE/docker-ce-cli_24.0.2-1~ubuntu.20.04~focal_amd64.deb dpkg -i DockerCE/docker-ce_24.0.2-1~ubuntu.20.04~focal_amd64.deb dpkg -i DockerCE/cri-dockerd_0.3.9.3-0.ubuntu-focal_amd64.deb # 配置docker, 限制日志大小，调整工作目录为 /home 或其他数据盘, 配置运行时的 cgroup driver 为 systemd # log-opts 需要和 kubelet 的KubeletConfiguration配置中容器日志文件相关配置保持一致避免冲突， # max-size <-> containerLogMaxSize 默认 10m # max-file <-> containerLogMaxFiles 默认 5 # 参考：https://github.com/Mirantis/cri-dockerd/issues/35 cat > /etc/docker/daemon.json << EOF { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "5" }, "data-root": "/home/docker", "exec-opts": ["native.cgroupdriver=systemd"], "insecure-registries": ["dockerhub.kubekey.local"] } EOF systemctl restart docker # 修改 cri-dockerd 默认的基础镜像，以适配当前版本的 Kubernetes 的 pause 镜像 cat > /usr/lib/systemd/system/cri-docker.service << EOF [Unit] Description=CRI Interface for Docker Application Container Engine Documentation=https://docs.mirantis.com After=network-online.target firewalld.service docker.service Wants=network-online.target Requires=cri-docker.socket [Service] Type=notify ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.k8s.io/pause:3.9 ExecReload=/bin/kill -s HUP \$MAINPID TimeoutSec=0 RestartSec=2 Restart=always # Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229. # Both the old, and new location are accepted by systemd 229 and up, so using the old location # to make them work for either version of systemd. StartLimitBurst=3 # Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230. # Both the old, and new name are accepted by systemd 230 and up, so using the old name to make # this option work for either version of systemd. StartLimitInterval=60s # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity # Comment TasksMax if your systemd version does not support it. # Only systemd 226 and above support this option. TasksMax=infinity Delegate=yes KillMode=process [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl restart cri-docker通过运行如下命令查看 cri-dockerd 的运行情况, cri-dockerd 依赖 conntrack 二进制，后续在安装 kubeadm 等软件时安装systemctl status cri-docker通过运行如下命令查看 docker ce 的运行情况systemctl status docker通过运行如下命令查看 docker ce 的 cgroupdriver 是否配置为 systemddocker info | grep 'Cgroup Driver'4 - 安装 kubeadm、kubelet 和 kubectl本章节说明如何安装 kubeadm 工具箱。假设安装包已上传至节点的 /opt 目录，执行如下命令完成安装：cd /opt/K8S-INSTALL/ubuntu20.04/ && tar xzvf kubeadm-init.tar.gz dpkg -i kubeadm-init/conntrack_1.4.5-2_amd64.deb dpkg -i kubeadm-init/ebtables_2.0.11-3build1_amd64.deb dpkg -i kubeadm-init/socat_1.7.3.3-2_amd64.deb dpkg -i kubeadm-init/cri-tools_1.28.0-1.1_amd64.deb dpkg -i kubeadm-init/kubernetes-cni_1.2.0-2.1_amd64.deb dpkg -i kubeadm-init/kubectl_1.28.5-1.1_amd64.deb dpkg -i kubeadm-init/kubelet_1.28.5-1.1_amd64.deb dpkg -i kubeadm-init/kubeadm_1.28.5-1.1_amd64.deb # 锁定版本 apt-mark hold kubelet kubeadm kubectl # 重启 cri-dockerd，解决其 conntrack 依赖问题 systemctl restart cri-dockerkubelet 现在每隔几秒就会重启，因为它陷入了一个等待 kubeadm 指令的死循环。5 - 替换 kubeadm 为证书 100 年有效期修改版(对应 kubeadm v1.28.5)默认 kubeadm 生成的证书是 1 年有效期，通过修改源码重新编译默认生成 100 年有效期证书的 kubeadm ，保证集群的稳定，执行以下命令完成命令的备份和替换：mv /usr/bin/kubeadm /usr/bin/kubeadm_backup cp /opt/K8S-INSTALL/kubeadm_100Y /usr/bin/kubeadm chown root:root /usr/bin/kubeadm chmod +x /usr/bin/kubeadm # 验证版本 kubeadm version三、部署 Nginx Keepalived 高可用负载均衡层使用 https://10.10.0.18:16443 作为 Kubernetes 控制面的负载均衡地址，在选定的三台机器（10.10.0.13,10.10.0.12,10.10.0.14）上部署如下负载均衡组件：# 三个负载均衡节点安装nginx和keepalived cd /opt && tar xzvf nginx-lb.tar.gz # apt-get install nginx -y dpkg -i nginx-lb/fonts-dejavu-core_2.37-1_all.deb dpkg -i nginx-lb/fontconfig-config_2.13.1-2ubuntu3_all.deb dpkg -i nginx-lb/libfontconfig1_2.13.1-2ubuntu3_amd64.deb dpkg -i nginx-lb/libjpeg-turbo8_2.0.3-0ubuntu1.20.04.3_amd64.deb dpkg -i nginx-lb/libjpeg8_8c-2ubuntu8_amd64.deb dpkg -i nginx-lb/libjbig0_2.1-3.1ubuntu0.20.04.1_amd64.deb dpkg -i nginx-lb/libwebp6_0.6.1-2ubuntu0.20.04.3_amd64.deb dpkg -i nginx-lb/libtiff5_4.1.0+git191117-2ubuntu0.20.04.11_amd64.deb dpkg -i nginx-lb/libxpm4_3.5.12-1ubuntu0.20.04.2_amd64.deb dpkg -i nginx-lb/libgd3_2.2.5-5.2ubuntu2.1_amd64.deb dpkg -i nginx-lb/nginx-common_1.18.0-0ubuntu1.4_all.deb dpkg -i nginx-lb/libnginx-mod-http-image-filter_1.18.0-0ubuntu1.4_amd64.deb dpkg -i nginx-lb/libnginx-mod-http-xslt-filter_1.18.0-0ubuntu1.4_amd64.deb dpkg -i nginx-lb/libnginx-mod-mail_1.18.0-0ubuntu1.4_amd64.deb dpkg -i nginx-lb/libnginx-mod-stream_1.18.0-0ubuntu1.4_amd64.deb dpkg -i nginx-lb/nginx-core_1.18.0-0ubuntu1.4_amd64.deb dpkg -i nginx-lb/nginx_1.18.0-0ubuntu1.4_all.deb # apt-get install keepalived -y dpkg -i nginx-lb/mysql-common_5.8+1.0.5ubuntu2_all.deb dpkg -i nginx-lb/libmysqlclient21_8.0.35-0ubuntu0.20.04.1_amd64.deb dpkg -i nginx-lb/libsensors-config_3.6.0-2ubuntu1.1_all.deb dpkg -i nginx-lb/libsensors5_3.6.0-2ubuntu1.1_amd64.deb dpkg -i nginx-lb/libsnmp-base_5.8+dfsg-2ubuntu2.9_all.deb dpkg -i nginx-lb/libsnmp35_5.8+dfsg-2ubuntu2.9_amd64.deb dpkg -i nginx-lb/keepalived_2.0.19-2ubuntu0.2_amd64.deb # 配置nginx负载均衡master节点 cat > /etc/nginx/nginx.conf << EOF user www-data; worker_processes auto; pid /run/nginx.pid; include /etc/nginx/modules-enabled/*.conf; events { worker_connections 1024; } # 四层负载均衡，为多台Master apiserver组件提供负载均衡 stream { log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent'; access_log /var/log/nginx/k8s-access.log main; upstream k8s-apiserver { server 10.10.0.15:6443 weight=1 max_fails=2 fail_timeout=30s; # Master1 APISERVER IP:PORT server 10.10.0.16:6443 weight=1 max_fails=2 fail_timeout=30s; # Master2 APISERVER IP:PORT server 10.10.0.17:6443 weight=1 max_fails=2 fail_timeout=30s; # Master3 APISERVER IP:PORT } server { listen 16443; # 如果nginx与master节点复用，这个监听端口不能是6443，否则会冲突 proxy_pass k8s-apiserver; } } EOF # 启动nginx systemctl start nginx systemctl enable nginx # 所有节点配置检查nginx进程状态的脚本 cat > /etc/keepalived/check_nginx.sh << "EOF" #!/bin/bash count=$(ss -antp |grep 16443 |egrep -cv "grep|$$") if [ "$count" -eq 0 ];then exit 1 else exit 0 fi EOF chmod +x /etc/keepalived/check_nginx.sh # 配置10.10.0.12的keepalived（非抢占式） cat > /etc/keepalived/keepalived.conf << EOF global_defs { script_user root enable_script_security router_id LVS_DEVEL } vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh" } vrrp_instance VI_1 { state BACKUP interface bond0 # 修改为实际网卡名 virtual_router_id 55 # VRRP 路由 ID实例，每个实例是唯一的 priority 100 nopreempt # 非抢占式 advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒 unicast_peer{ 10.10.0.13 10.10.0.14 } authentication { auth_type PASS auth_pass kube@123 } virtual_ipaddress { 10.10.0.18 } track_script { check_nginx } } EOF # 配置10.10.0.13的keepalived（非抢占式） cat > /etc/keepalived/keepalived.conf << EOF global_defs { script_user root enable_script_security router_id LVS_DEVEL } vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh" } vrrp_instance VI_1 { state BACKUP interface eno1np0 # 修改为实际网卡名 virtual_router_id 55 # VRRP 路由 ID实例，每个实例是唯一的 priority 90 nopreempt # 非抢占式 advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒 unicast_peer{ 10.10.0.12 10.10.0.14 } authentication { auth_type PASS auth_pass kube@123 } virtual_ipaddress { 10.10.0.18 } track_script { check_nginx } } EOF # 配置10.10.0.14的keepalived（非抢占式） cat > /etc/keepalived/keepalived.conf << EOF global_defs { script_user root enable_script_security router_id LVS_DEVEL } vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh" } vrrp_instance VI_1 { state BACKUP interface bond0 # 修改为实际网卡名 virtual_router_id 55 # VRRP 路由 ID实例，每个实例是唯一的 priority 70 nopreempt # 非抢占式 advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒 unicast_peer{ 10.10.0.12 10.10.0.13 } authentication { auth_type PASS auth_pass kube@123 } virtual_ipaddress { 10.10.0.18 } track_script { check_nginx } } EOF # 启动keepalived systemctl restart keepalived systemctl enable keepalived 四、部署 Kubernetes使用 kubeadm 安装 Kubernetes 控制平面，本文档旨在安装一个三个控制面多个工作节点的 Kubernetes 集群，因此安装命令参数需要添加如下内容：--control-plane-endpoint= 10.10.0.18:16443 ：计划将单个控制平面 kubeadm 集群升级成高可用， 你应该指定 --control-plane-endpoint 为所有控制平面节点设置共享端点。 端点可以是负载均衡器的 DNS 名称或 IP 地址。--pod-network-cidr=192.168.0.0/16 : Pod 网络插件使用 Calico，Calico 默认使用的 Pod 网络地址范围为 192.168.0.0/16--cri-socket=unix:///var/run/cri-dockerd.sock : 使用了 cri-dockerd 适配器将 Docker CE 集成进 Kubernetes，需要指定使用的容器运行时连接 socket 为 cri-dockerd 的 unix:///var/run/cri-dockerd.sock--kubernetes-version=v1.28.5 : 指定安装的 Kubernetes 版本kube-proxy 使用 ipvs 模式--upload-certs 标志用来将在所有控制平面实例之间的共享证书上传到集群。 如果正好相反，你更喜欢手动地通过控制平面节点或者使用自动化工具复制证书， 请删除此标志并参考如下部分证书分配手册。命令参数说明：kubeadm init | Kubernetes3-1 生成 kubeadm init 配置注意：本节步骤只需要在主节点 1 执行，作为第一个安装的控制节点。我们使用 kubeadm config print init-defaults > kubeadm_init_config.yaml 命令生成默认配置，在此基础上修改或添加我们自定义的参数配置，修改后的文件 kubeadm_init_config.yaml 内容如下：cd /opt/K8S-INSTALL/ cat > /opt/K8S-INSTALL/kubeadm_init_config.yaml << EOF apiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration localAPIEndpoint: bindPort: 6443 nodeRegistration: # 配置 docker ce 为容器运行时 criSocket: unix:///var/run/cri-dockerd.sock imagePullPolicy: IfNotPresent taints: null --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: {} etcd: local: # 必要时修改etcd存储路径 dataDir: /var/lib/etcd imageRepository: registry.k8s.io kind: ClusterConfiguration # 配置集群控制面负载地址 controlPlaneEndpoint: 10.10.0.18:16443 # 安装 Kubernetes 1.28.5 版本 kubernetesVersion: 1.28.5 networking: dnsDomain: cluster.local # Calico 网络默认的Pod子网 podSubnet: 192.168.0.0/16 serviceSubnet: 192.169.0.0/12 scheduler: {} --- # 配置kube-proxy为 ipvs 模式 apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: ipvs EOF3-2 导入基础镜像假设已经镜像包上传至各节点的 /opt 目录，在所有 Kubernetes 节点执行如下命令完成镜像的导入：docker load -i /opt/K8S-INSTALL/images/kubeadm-k8s-1.28.5.tar.gz3-3 执行安装命令在主节点 1 上执行如下命令开始安装 Kubernetes 控制节点：kubeadm init --config /opt/K8S-INSTALL/kubeadm_init_config.yaml --upload-certs完成后输出如下内容：Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/You can now join any number of the control-plane node running the following command on each as root: kubeadm join 10.10.0.18:16443 --token g8v4vv.jez3btz6kb6oy9rn \ --discovery-token-ca-cert-hash sha256:d976b88a2fab6538f38b9f934b6f889b136605861e35d5d3295bae2c1173126b \ --control-plane --certificate-key afafa37b37ef47d8f80bfe6e865c75dc80e84fdec30d068adbdef9ae7a56a2f3Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward.Then you can join any number of worker nodes by running the following on each as root:kubeadm join 10.10.0.18:16443 --token g8v4vv.jez3btz6kb6oy9rn \ --discovery-token-ca-cert-hash sha256:d976b88a2fab6538f38b9f934b6f889b136605861e35d5d3295bae2c1173126b按照说明执行如下命令完成 kubectl 的连接配置：mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config3-4 添加其他控制面节点根据 3-3 部署完成后输出的命令提示，在其他控制面节点使用如下命令完成控制面节点部署并加入集群，注意指定 --cri-socket unix:///var/run/cri-dockerd.sock ，certificate-key 的有效期为 2 小时，过期后使用命令 kubeadm init phase upload-certs --upload-certs 重新获取：# 添加节点 kubeadm join 10.10.0.18:16443 --token f8yzvp.ng0pnlnvhbohvttv \ --discovery-token-ca-cert-hash sha256:19d95c39b9630ffa896e5209e0e0d2682b5fffc449c20fd3b7b54bef7339972b \ --control-plane --certificate-key cc89ea3987e5a078652ea24ef864d6084c862459555927ac42e672dc342efbef \ --cri-socket unix:///var/run/cri-dockerd.sock命令执行成功后输出如下内容：This node has joined the cluster and a new control plane instance was created:Certificate signing request was sent to apiserver and approval was received.The Kubelet was informed of the new secure connection details.Control plane label and taint were applied to the new node.The Kubernetes control plane instances scaled up.A new etcd member was added to the local/stacked etcd cluster.To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/configRun 'kubectl get nodes' to see this node join the cluster.按照说明执行如下命令完成 kubectl 的连接配置：mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config此时可以执行 kubectl get nodes 命令获取已安装的控制名节点，节点在未安装完成网络插件前状态都将是 NotReady ：root@k8s-master3:~# kubectl get nodesNAME STATUS ROLES AGE VERSIONk8s-master1 NotReady control-plane 11m v1.28.5k8s-master2 NotReady control-plane 4m58s v1.28.5k8s-master3 NotReady control-plane 4m8s v1.28.53-5 添加工作节点根据 3-3 部署完成后输出的命令提示，在工作节点使用如下命令完成工作节点部署并加入集群，注意指定 --cri-socket unix:///var/run/cri-dockerd.sock ，指定的 token 有效期为 24 小时：kubeadm join 10.10.0.18:16443 --token f8yzvp.ng0pnlnvhbohvttv \ --discovery-token-ca-cert-hash sha256:19d95c39b9630ffa896e5209e0e0d2682b5fffc449c20fd3b7b54bef7339972b \ --cri-socket unix:///var/run/cri-dockerd.sock命令执行成功后输出如下内容：This node has joined the cluster:Certificate signing request was sent to apiserver and a response was received.The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.此时可以在控制节点执行 kubectl get nodes 命令获取已安装的控制名节点，节点在未安装完成网络插件前状态都将是 NotReady ：root@k8s-master3:~# kubectl get nodes NAME STATUS ROLES AGE VERSIONk8s-master1 NotReady control-plane 18m v1.28.5k8s-master2 NotReady control-plane 12m v1.28.5k8s-master3 NotReady control-plane 11m v1.28.5k8s-node1 NotReady 106s v1.28.53-6 部署 Calico 网络插件参考：Quickstart for Calico on Kubernetes | Calico Documentation (tigera.io)Calico 是一种开源网络和网络安全解决方案，适用于容器，虚拟机和基于主机的本机工作负载。Calico 支持广泛的平台，包括 Kubernetes，docker，OpenStack 和裸机服务。Calico 后端支持多种网络模式。BGP 模式：将节点做为虚拟路由器通过 BGP 路由协议来实现集群内容器之间的网络访问。IPIP 模式：在原有 IP 报文中封装一个新的 IP 报文，新的 IP 报文中将源地址 IP 和目的地址 IP 都修改为对端宿主机 IP。cross-subnet：Calico-ipip 模式和 calico-bgp 模式都有对应的局限性，对于一些主机跨子网而又无法使网络设备使用 BGP 的场景可以使用 cross-subnet 模式，实现同子网机器使用 calico-BGP 模式，跨子网机器使用 calico-ipip 模式。部署完成后默认使用 calico-ipip 的模式各节点加载镜像 :以下命令在所有 Kubernetes 节点执行：docker load -i /opt/K8S-INSTALL/images/calico-cni-3.27.0.tar.gz安装 Calico ：如下指令在任意一个控制面节点执行：cd /opt/K8S-INSTALL/ && tar xzvf calico-manifests-3.27.0.tar.gz kubectl create -f calico-manifests-3.27.0/tigera-operator.yaml # 如果 pod 网段不是 192.168.0.0/16 , 对应修改 custom-resources.yaml 文件内容 kubectl create -f calico-manifests-3.27.0/custom-resources.yaml执行如下命令等待所有 Pod 的 STATUS 栏变成 Running 即完成 Calico 的安装 :watch kubectl get pods -n calico-systemEvery 2.0s: kubectl get pods -n calico-system k8s-master1: Thu Jan 11 15:46:28 2024NAME READY STATUS RESTARTS AGEcalico-kube-controllers-78b56f45bc-b2skt 1/1 Running 0 3m34scalico-node-cxh5z 1/1 Running 0 3m34scalico-node-jgdjh 1/1 Running 0 3m34scalico-node-qcjr5 1/1 Running 0 3m34scalico-node-z6bxl 1/1 Running 0 3m34scalico-typha-5df98d9d45-mfkpv 1/1 Running 0 3m25scalico-typha-5df98d9d45-z2f7t 1/1 Running 0 3m34scsi-node-driver-c5bjh 2/2 Running 0 3m34scsi-node-driver-fvs5b 2/2 Running 0 3m34scsi-node-driver-gktvt 2/2 Running 0 3m34scsi-node-driver-ktg54 2/2 Running 0 3m34s执行如下命令等待所有 Pod 的 STATUS 栏变成 Running 即完成 Calico api-server 的安装 :watch kubectl get pods -n calico-apiserverEvery 2.0s: kubectl get pods -n calico-apiserverk8s-master1: Thu Jan 11 16:09:22 2024NAME READY STATUS RESTARTS AGEcalico-apiserver-747499f6b8-bz8f2 1/1 Running 0 76scalico-apiserver-747499f6b8-g9l8n 1/1 Running 0 2m14s所有 Pod 状态均为 Running 后，执行如下命令获取各节点状态将为 Ready :kubectl get nodesNAME STATUS ROLES AGE VERSIONk8s-master1 Ready control-plane 106m v1.28.5k8s-master2 Ready control-plane 100m v1.28.5k8s-master3 Ready control-plane 99m v1.28.5k8s-node1 Ready 89m v1.28.5至此，Kubernetes 高可用集群就安装完成了 ，后续如果需要添加工作节点或者控制节点，按 3-4 、3-5 操作即可，注意修改 token 和 certificate-key 为新生成的。# 重新创建 token 命令 kubeadm token create --print-join-command # 重新创建 certificate-key 命令 kubeadm init phase upload-certs --upload-certs # 获取 discovery-token-ca-cert-hash 的值 openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex | sed 's/^.* //'五、自编译 Kubeadm 使证书默认 100 年有效期Kubernetes v1.25 编译 kubeadm 修改证书有效期到 100 年 - sysin - 博客园 (cnblogs.com)kubeadm 安装的 k8s 证书过期升级为 100 年证书_k8s 证书改 100 年-CSDN 博客默认的 kubeadm 命令生成的证书有效期是 1 年，虽然谷歌推荐每年使用 kubeadm 续期，但实际环境对 Kubernetes 集群的改动较小，为了环境稳定，使用一个 10 年以上的证书可以减少麻烦。常用有效的办法就行修改 kubeadm 的源码，修改代码中默认的有效期为 100 年，然后编译成二进制文件替换默认版本。以下说明如何进行修改和编译。5-1 下载对应版本源码访问：https://github.com/kubernetes/kubernetes/releases，下载特定版本源码，这里获取 v1.28.5 版本的wget https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.28.5.tar.gz tar -zxvf kubernetes-1.28.5.tar.gz mv kubernetes-1.28.5 kubernetes cd kubernetes5-2 修改源码主要有两个地方需要修改(v1.28.5 版本后是否会变动暂不可知)5-2-1 修改 CA 有效期为 100 年（默认为 10 年）vim ./staging/src/k8s.io/client-go/util/cert/cert.go // 这个方法里面 NotAfter: now.Add(duration365d * 10).UTC() // 默认有效期就是 10 年，改成 100 年 (sysin) // 输入 /NotAfter 查找，回车定位 func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) { now := time.Now() tmpl := x509.Certificate{ SerialNumber: new(big.Int).SetInt64(0), Subject: pkix.Name{ CommonName: cfg.CommonName, Organization: cfg.Organization, }, NotBefore: now.UTC(), // NotAfter: now.Add(duration365d * 10).UTC(), NotAfter: now.Add(duration365d * 100).UTC(), KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, BasicConstraintsValid: true, IsCA: true, } certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key) if err != nil { return nil, err } return x509.ParseCertificate(certDERBytes) }5-2-2 修改证书有效期为 100 年（默认为 1 年）vim ./cmd/kubeadm/app/constants/constants.go // 就是这个常量定义 CertificateValidity，改成 * 100 年 (sysin) // 输入 /CertificateValidity 查找，回车定位 const ( // KubernetesDir is the directory Kubernetes owns for storing various configuration files KubernetesDir = "/etc/kubernetes" // ManifestsSubDirName defines directory name to store manifests ManifestsSubDirName = "manifests" // TempDirForKubeadm defines temporary directory for kubeadm // should be joined with KubernetesDir. TempDirForKubeadm = "tmp" // CertificateValidity defines the validity for all the signed certificates generated by kubeadm // CertificateValidity = time.Hour * 24 * 365 CertificateValidity = time.Hour * 24 * 365 * 100 // CACertAndKeyBaseName defines certificate authority base name CACertAndKeyBaseName = "ca" // CACertName defines certificate name CACertName = "ca.crt" // CAKeyName defines certificate name CAKeyName = "ca.key"验证一下已经正确修改：cat ./staging/src/k8s.io/client-go/util/cert/cert.go | grep NotAfter cat ./cmd/kubeadm/app/constants/constants.go | grep CertificateValidity5-3 安装编译环境官方编译说明 ： kubernetes/build/README.md at 4b94168c0f724bbf1edfd846a8e788a32d9a5c49 · kubernetes/kubernetes · GitHubapt-get install makedkpg -i docker-buildx-plugin_0.11.2-1\~ubuntu.20.04~focal_amd64.debdocker pull registry.k8s.io/build-image/kube-cross:v1.28.0-go1.20.12-bullseye.0 (编译镜像)其他镜像，但只编译二进制文件不需要其他的5-4 编译二进制我们只需生成新的 kubeadm 二进制文件，不需要整个 Release 包，执行如下命令只编译二进制文件# kubernetes 源码根目录执行 build/make-clean.sh build/run.sh makeroot@k8s-master1:/opt/kubernetes# build/run.sh make+++ [0112 10:13:26] Verifying Prerequisites....+++ [0112 10:13:26] Building Docker image kube-build:build-9146bf4b41-5-v1.28.0-go1.20.12-bullseye.0+++ [0112 10:13:27] Syncing sources to container+++ [0112 10:13:29] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.+++ [0112 10:13:29] Running build command... Go version: go version go1.20.12 linux/amd64+++ [0112 10:13:29] Building go targets for linux/amd64 k8s.io/kubernetes/cmd/kube-proxy (static) k8s.io/kubernetes/cmd/kube-apiserver (static) k8s.io/kubernetes/cmd/kube-controller-manager (static) k8s.io/kubernetes/cmd/kubelet (non-static) k8s.io/kubernetes/cmd/kubeadm (static) k8s.io/kubernetes/cmd/kube-scheduler (static) k8s.io/component-base/logs/kube-log-runner (static) k8s.io/kube-aggregator (static) k8s.io/apiextensions-apiserver (static) k8s.io/kubernetes/cluster/gce/gci/mounter (non-static) k8s.io/kubernetes/cmd/kubectl (static) k8s.io/kubernetes/cmd/kubectl-convert (static) github.com/onsi/ginkgo/v2/ginkgo (non-static) k8s.io/kubernetes/test/e2e/e2e.test (test) k8s.io/kubernetes/test/conformance/image/go-runner (non-static) k8s.io/kubernetes/cmd/kubemark (static) github.com/onsi/ginkgo/v2/ginkgo (non-static) k8s.io/kubernetes/test/e2e_node/e2e_node.test (test) Env for linux/amd64: GOOS=linux GOARCH=amd64 GOROOT=/usr/local/go CGO_ENABLED= CC= Coverage is disabled. Coverage is disabled.+++ [0112 10:14:21] Placing binaries+++ [0112 10:14:41] Syncing out of container root@k8s-master1:/opt/kubernetes# build/make-clean.sh+++ [0112 10:18:58] Verifying Prerequisites....+++ [0112 10:18:58] Deleting container kube-build-data-9146bf4b41-5-v1.28.0-go1.20.12-bullseye.0+++ [0112 10:19:00] Deleting image kube-build:build-9146bf4b41-5-v1.28.0-go1.20.12-bullseye.0+++ [0112 10:19:00] Cleaning all untagged docker images+++ [0112 10:19:00] Removing _output directory root@k8s-master1:/opt/kubernetes# build/run.sh make+++ [0112 10:19:05] Verifying Prerequisites....+++ [0112 10:19:05] Building Docker image kube-build:build-9146bf4b41-5-v1.28.0-go1.20.12-bullseye.0+++ [0112 10:19:06] Creating data container kube-build-data-9146bf4b41-5-v1.28.0-go1.20.12-bullseye.0+++ [0112 10:19:10] Syncing sources to container+++ [0112 10:19:20] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.+++ [0112 10:19:20] Running build command... Go version: go version go1.20.12 linux/amd64+++ [0112 10:19:21] Building go targets for linux/amd64 k8s.io/kubernetes/cmd/kube-proxy (static) k8s.io/kubernetes/cmd/kube-apiserver (static) k8s.io/kubernetes/cmd/kube-controller-manager (static) k8s.io/kubernetes/cmd/kubelet (non-static) k8s.io/kubernetes/cmd/kubeadm (static) k8s.io/kubernetes/cmd/kube-scheduler (static) k8s.io/component-base/logs/kube-log-runner (static) k8s.io/kube-aggregator (static) k8s.io/apiextensions-apiserver (static) k8s.io/kubernetes/cluster/gce/gci/mounter (non-static) k8s.io/kubernetes/cmd/kubectl (static) k8s.io/kubernetes/cmd/kubectl-convert (static) github.com/onsi/ginkgo/v2/ginkgo (non-static) k8s.io/kubernetes/test/e2e/e2e.test (test) k8s.io/kubernetes/test/conformance/image/go-runner (non-static) k8s.io/kubernetes/cmd/kubemark (static) github.com/onsi/ginkgo/v2/ginkgo (non-static) k8s.io/kubernetes/test/e2e_node/e2e_node.test (test) Env for linux/amd64: GOOS=linux GOARCH=amd64 GOROOT=/usr/local/go CGO_ENABLED= CC= Coverage is disabled. Coverage is disabled.+++ [0112 10:27:10] Placing binaries+++ [0112 10:27:27] Syncing out of container编译完产物在 _output/dockerized/bin/linux/amd64/ 目录下5-5 备份原版，安装新版mv /usr/bin/kubeadm /usr/bin/kubeadm_backup cp _output/local/bin/linux/amd64/kubeadm /usr/bin/kubeadm #chmod +x /usr/bin/kubeadm # 验证版本 kubeadm version # 压缩保存到用户主目录下 tar zcvf ~/kubeadm-1.25.0.tgz -C /usr/bin/ kubeadm # 在其他(含工作节点)节点上替换原有版本 mv /usr/bin/kubeadm /usr/bin/kubeadm_backup tar zxvf ./kubeadm-1.25.0.tgz -C /usr/bin/5-6 【原版 kubeadm 安装前】正常使用修改后的命令安装集群即可5-6 【原版 kubeadm 安装后】执行命令更新证书如果是使用原版 kubeadm 安装之后，可以手动执行命令更新证书有效期到 100 年。可以先备份证书，证书在 /etc/kubernetes/pki# 续订全部证书 kubeadm certs renew all # 重启 kube-apiserver, kube-controller-manager, kube-scheduler , etcd kubectl -n kube-system delete pod kube-scheduler-k8s-master1 kube-scheduler-k8s-master2 kube-scheduler-k8s-master3 kube-controller-manager-k8s-master1 kube-controller-manager-k8s-master2 kube-controller-manager-k8s-master3 kube-apiserver-k8s-master1 kube-apiserver-k8s-master2 kube-apiserver-k8s-master3 etcd-k8s-master1 etcd-k8s-master2 etcd-k8s-master3 # 查看当前有效期，ca证书是init的时候生成的，这个命令刷新不了 kubeadm certs check-expirationroot@k8s-master1:/opt# kubeadm certs check-expiration[check-expiration] Reading configuration from the cluster...[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGEDadmin.conf Dec 19, 2123 02:46 UTC 99y ca noapiserver Dec 19, 2123 02:46 UTC 99y ca noapiserver-etcd-client Dec 19, 2123 02:46 UTC 99y etcd-ca noapiserver-kubelet-client Dec 19, 2123 02:46 UTC 99y ca nocontroller-manager.conf Dec 19, 2123 02:46 UTC 99y ca noetcd-healthcheck-client Dec 19, 2123 02:46 UTC 99y etcd-ca noetcd-peer Dec 19, 2123 02:46 UTC 99y etcd-ca noetcd-server Dec 19, 2123 02:46 UTC 99y etcd-ca nofront-proxy-client Dec 19, 2123 02:46 UTC 99y front-proxy-ca noscheduler.conf Dec 19, 2123 02:46 UTC 99y ca noCERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGEDca Jan 08, 2034 06:21 UTC 9y noetcd-ca Jan 08, 2034 06:21 UTC 9y nofront-proxy-ca Jan 08, 2034 06:21 UTC 9y noapiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration ... cgroupDriver: systemd
Docker安装SonarQube并配置阿里P3C规范SonarQube Mysql安装安装docker run -itd --restart=always -p 17007:3306 -e MYSQL_ROOT_PASSWORD=sonarQube@2020 -v sonarqube-mysql-conf:/etc/mysql/conf.d -v sonarqube-mysql-data:/var/lib/mysq --name sonarqube-mysql mysql:5.7配置mysqlmysql.cnf添加[mysqld] sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES max_allowed_packet=100M innodb_log_file_size=640M 初始化登录mysql执行如下sql： CREATE DATABASE sonar CHARACTER SET utf8 COLLATE utf8_general_ci; CREATE USER 'sonar' IDENTIFIED BY 'sonar'; GRANT ALL ON sonar.* TO 'sonar'@'%' IDENTIFIED BY 'sonar'; GRANT ALL ON sonar.* TO 'sonar'@'localhost' IDENTIFIED BY 'sonar'; FLUSH PRIVILEGES;SonarQube 安装配置主机内核参数（大于等于以下值即可）sysctl -w vm.max_map_count=262144 sysctl -w fs.file-max=65536 ulimit -n 65536 ulimit -u 4096安装SonarQubedocker run -itd --restart=always -p 9000:9000 -e SONARQUBE_JDBC_URL='jdbc:mysql://10.0.55.117:17007/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance&useSSL=false' -e SONARQUBE_JDBC_USERNAME=sonar -e SONARQUBE_JDBC_PASSWORD=sonar -v sonarqube-data:/opt/sonarqube/data -v sonarqube-logs:/opt/sonarqube/logs -v sonarqube-extensions:/opt/sonarqube/extensions --name sonarqube sonarqube:7.4-communitySonarQube Sancenr安装(zip)wget https://github.com/SonarSource/sonar-scanner-cli/releases/download/4.4.0.2170/sonar-scanner-cli-4.4.0.2170-linux.zip unzip sonar-scanner-cli-4.4.0.2170-linux.zip mkdir /sonarQubeScanner mv sonar-scanner-cli-4.4.0.2170-linux /sonarQubeScanner修改配置文件，连接sonar服务器vi /sonarQubeScanner/sonar-scanner-4.4.0.2170-linux/conf/sonar-scanner.properties #添加如下内容 # 默认SonarQube服务地址，地址根据实际情况更改 sonar.host.url=http://10.0.45.220:9000 # 默认源代码编码 sonar.sourceEncoding=UTF-8配置环境变量vim /etc/profile 末尾添加一行 export PATH="$PATH:/sonarQubeScanner/sonar-scanner-4.4.0.2170-linux/bin" source /etc/profileSonarQube Sancenr安装(docker)参考： https://docs.sonarqube.org/latest/analysis/scan/sonarscanner/docker run \ --rm \ -e SONAR_HOST_URL="http://${SONARQUBE_URL}" \ -v "${YOUR_REPO}:/usr/src" \ sonarsource/sonar-scanner-cliOK配置pmd 阿里规范文档：阿里p3c规范.xml<?xml version='1.0' encoding='UTF-8'?><profile><name>阿里p3c</name><language>java</language><rules><rule><repositoryKey>checkstyle</repositoryKey><key>com.puppycrawl.tools.checkstyle.checks.coding.EqualsHashCodeCheck</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>common-java</repositoryKey><key>FailedUnitTests</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AbstractClassShouldStartWithAbstractNamingRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AbstractMethodOrInterfaceMethodMustUseJavadocRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AccessorClassGeneration</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AppendCharacterWithChar</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ArrayNamingShouldHaveBracketRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AssignmentToNonFinalStatic</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidAccessibilityAlteration</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidApacheBeanUtilsCopyRule</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidArrayLoops</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidBranchingStatementAsLastInLoop</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidCallStaticSimpleDateFormatRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidCommentBehindStatementRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidComplexConditionRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidConcurrentCompetitionRandomRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidFinalLocalVariable</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidInstantiatingObjectsInLoops</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidMissUseOfMathRandomRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidNewDateGetTimeRule</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidPatternCompileInMethodRule</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidReturnInFinallyRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidStartWithDollarAndUnderLineNamingRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidSynchronizedAtMethodLevel</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidThreadGroup</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidUseTimerRule</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidUsingNativeCode</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidUsingShortType</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>AvoidUsingVolatile</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>BadComparison</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>BeanMembersShouldSerialize</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>BigIntegerInstantiation</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>BooleanGetMethodName</key><priority>MAJOR</priority><parameters><parameter><key>checkParameterizedMethods</key><value>false</value></parameter></parameters></rule><rule><repositoryKey>pmd</repositoryKey><key>BooleanInstantiation</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>BooleanPropertyShouldNotStartWithIsRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ByteInstantiation</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CallSuperFirst</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CallSuperInConstructor</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CallSuperLast</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CheckResultSet</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ClassCastExceptionWithSubListToArrayListRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ClassCastExceptionWithToArray</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ClassCastExceptionWithToArrayRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ClassMustHaveAuthorRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ClassNamingShouldBeCamelRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CloneMethodMustBePublic</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CloneMethodReturnTypeMustMatchClassName</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CollectionInitShouldAssignCapacityRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CommentContent</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CommentDefaultAccessModifier</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CommentRequired</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CommentSize</key><priority>MINOR</priority><parameters><parameter><key>maxLineLength</key><value>80</value></parameter><parameter><key>maxLines</key><value>6</value></parameter></parameters></rule><rule><repositoryKey>pmd</repositoryKey><key>CommentsMustBeJavadocFormatRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ConcurrentExceptionWithModifyOriginSubListRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ConfusingTernary</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ConsecutiveAppendsShouldReuse</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ConsecutiveLiteralAppends</key><priority>MINOR</priority><parameters><parameter><key>threshold</key><value>1</value></parameter></parameters></rule><rule><repositoryKey>pmd</repositoryKey><key>ConstantFieldShouldBeUpperCaseRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>CountDownShouldInFinallyRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>DataflowAnomalyAnalysis</key><priority>MAJOR</priority><parameters><parameter><key>maxViolations</key><value>100</value></parameter><parameter><key>maxPaths</key><value>1000</value></parameter></parameters></rule><rule><repositoryKey>pmd</repositoryKey><key>DefaultPackage</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>DoNotHardCodeSDCard</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>DoNotUseThreads</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>DontModifyInForeachCircleRule</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>DontUseFloatTypeForLoopIndices</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>DoubleCheckedLocking</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>EmptyMethodInAbstractClassShouldBeAbstract</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>EnumConstantsMustHaveCommentRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>EqualsAvoidNullRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ExceptionClassShouldEndWithExceptionRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>GodClass</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>GuardDebugLogging</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>GuardLogStatement</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>GuardLogStatementJavaUtil</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ImmutableField</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>InefficientEmptyStringCheck</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>InefficientStringBuffering</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>InsufficientStringBufferDeclaration</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>IntegerInstantiation</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>LawOfDemeter</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>LocalHomeNamingConvention</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>LocalInterfaceSessionNamingConvention</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>LocalVariableCouldBeFinal</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>LongInstantiation</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>LowerCamelCaseVariableNamingRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>MDBAndSessionBeanNamingConvention</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>MethodReturnWrapperTypeRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>MissingStaticMethodInNonInstantiatableClass</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>NPathComplexity</key><priority>MAJOR</priority><parameters><parameter><key>minimum</key><value>200</value></parameter></parameters></rule><rule><repositoryKey>pmd</repositoryKey><key>NeedBraceRule</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>NullAssignment</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>OptimizableToArrayCall</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>PackageNamingRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>PojoMustOverrideToStringRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>PojoMustUsePrimitiveFieldRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>PojoNoDefaultValueRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>RedundantFieldInitializer</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>RemoteInterfaceNamingConvention</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>RemoteSessionInterfaceNamingConvention</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>RemoveCommentedCodeRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ServiceOrDaoClassShouldEndWithImplRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ShortInstantiation</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>SimpleDateFormatNeedsLocale</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>SimplifiedTernary</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>SimplifyConditional</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>SimplifyStartsWith</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>SingleMethodSingleton</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>SingletonClassReturningNewInstance</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>SingularField</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>StaticEJBFieldShouldBeFinal</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>StringConcatRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>StringInstantiation</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>SuspiciousOctalEscape</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>SwitchStatementRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>TestClassShouldEndWithTestNamingRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ThreadLocalShouldRemoveRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ThreadPoolCreationRule</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>ThreadShouldSetNameRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>TooManyFields</key><priority>MAJOR</priority><parameters><parameter><key>maxfields</key><value>15</value></parameter></parameters></rule><rule><repositoryKey>pmd</repositoryKey><key>TooManyStaticImports</key><priority>MAJOR</priority><parameters><parameter><key>maximumStaticImports</key><value>4</value></parameter></parameters></rule><rule><repositoryKey>pmd</repositoryKey><key>TransactionMustHaveRollbackRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UndefineMagicConstantRule</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UnnecessaryFinalModifier</key><priority>INFO</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UnnecessaryFullyQualifiedName</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UnnecessaryReturn</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UnsupportedExceptionWithModifyAsListRule</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UnusedModifier</key><priority>INFO</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UnusedNullCheckInEquals</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UpperEllRule</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UseArraysAsList</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UseConcurrentHashMap</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UseIndexOfChar</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UseLocaleWithCaseConversions</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UseProperClassLoader</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UseStringBufferForStringAppends</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UseStringBufferLength</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UseVarargs</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UselessOperationOnImmutable</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>UselessQualifiedThis</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>pmd</repositoryKey><key>WrapperTypeEqualityRule</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>ClassVariableVisibilityCheck</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>ObjectFinalizeCheck</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>ObjectFinalizeOverridenCallsSuperFinalizeCheck</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1143</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1148</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1175</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1201</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1206</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1217</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1221</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1226</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1244</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1313</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1317</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1444</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1656</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1751</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1764</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1849</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1860</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1862</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1872</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S1989</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2039</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2055</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2060</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2061</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2066</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2068</key><priority>BLOCKER</priority><parameters><parameter><key>credentialWords</key><value>password,passwd,pwd</value></parameter></parameters></rule><rule><repositoryKey>squid</repositoryKey><key>S2070</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2089</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2092</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2097</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2109</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2110</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2111</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2114</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2116</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2118</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2122</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2123</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2127</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2134</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2141</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2142</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2151</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2153</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2154</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2159</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2162</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2164</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2167</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2168</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2175</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2177</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2183</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2184</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2189</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2200</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2201</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2204</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2222</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2225</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2226</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2229</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2230</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2236</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2245</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2251</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2252</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2254</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2255</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2257</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2258</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2272</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2273</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2275</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2276</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2277</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2278</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2386</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2441</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2445</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2446</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2583</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2637</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2639</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2653</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2658</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2674</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2676</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2677</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2695</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2755</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2757</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2761</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2789</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2885</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2886</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S2976</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3020</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3034</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3046</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3066</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3067</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3281</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3306</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3329</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3330</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3346</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3355</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3369</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3374</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3422</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3436</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3438</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3439</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3510</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3518</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3599</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3655</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3749</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3750</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3751</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3752</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3753</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3923</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3958</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3959</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3981</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3984</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S3986</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4143</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4275</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4347</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4348</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4351</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4423</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4424</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4426</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4432</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4433</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4434</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4435</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4499</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4502</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4507</key><priority>MINOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4508</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4510</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4529</key><priority>MAJOR</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4544</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4601</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4602</key><priority>BLOCKER</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S4684</key><priority>CRITICAL</priority><parameters/></rule><rule><repositoryKey>squid</repositoryKey><key>S899</key><priority>MINOR</priority><parameters/></rule></rules></profile>参考： https://blog.csdn.net/zuozewei/article/details/90232808安装gitlab插件见常见插件包安装微信机器人https://github.com/sjdy521/Mojo-Weixin/blob/master/Docker.md
方案说明首先inotify监控文件系统的各种变化，当文件有任何变动时，就触发rsync同步，则该源服务器为主动，那它就是客户端，备份服务器为被动方，那它就是服务端(-deamon)。如此达到实时同步目的。实现源服务器文件变动后同步更新至备份服务器环境说明服务器均为 Centos7rsync 安装yum -y install rsync firewall-cmd --add-port=873/tcp --permanent firewall-cmd --reload systemctl start rsyncd.service systemctl enable rsyncd.service systemctl status rsyncd.service netstat -lnp|grep 873备份服务器（server端 192.168.1.102）rsync 配置建立用户与密码认证文件mkdir -p /etc/rsyncd/ echo "backup:bk_password" > /etc/rsyncd/rsyncd.passwd chmod 0600 /etc/rsyncd/rsyncd.passwd建立rsync配置文件# vi /etc/rsyncd.conf uid = root gid = root use chroot = no max connections = 4 strict modes = yes hosts allow = * port = 873 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log ignore nonreadable = yes dont compress = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2 [test] path = /opt/test ignore errors read only = false write only = false list = false auth users = backup secrets file = /etc/rsyncd/rsyncd.passwd PS：注意这里要将 read only 修改为 false, 因为是要将文件同步到该模块的，必须可写重新启动systemctl restart rsyncd.service源服务器 （client端 192.168.1.101）rsync 安装rsync 安装yum -y install rsync firewall-cmd --add-port=873/tcp --permanent firewall-cmd --reload建立访问server端的密码认证文件mkdir -p /etc/rsyncd/ echo "bk_password" > /etc/rsyncd/rsync.passwd chmod 600 /etc/rsyncd/rsync.passwd执行如下命令初始同步 /opt/test 目录rsync -avzr --delete --progress --password-file=/etc/rsyncd/rsync.passwd /opt/dockerServiceLog backup@192.168.1.102::test inotify 安装查看是否支持inotifyll /proc/sys/fs/inotify/ -rw-r--r--. 1 root root 0 May 9 11:46 max_queued_events -rw-r--r--. 1 root root 0 May 9 11:46 max_user_instances -rw-r--r--. 1 root root 0 May 9 11:46 max_user_watches 有这几个文件表示支持编译安装wget https://github.com/inotify-tools/inotify-tools/releases/download/3.20.2.2/inotify-tools-3.20.2.2.tar.gz tar xzvf inotify-tools-3.20.2.2.tar.gz cd inotify-tools-3.20.2.2 ./configure --prefix=/usr/local/inotify make && make install编写监控文件变化脚本vi /etc/rsyncd/watchdir.sh#!/bin/bash #filename watchdir.sh path=$1 mkdir -p /var/log/rsync/ /usr/local/inotify/bin/inotifywait -mrq --timefmt '%d/%m/%y/%H:%M' --format '%T %w %f' -e modify,delete,create,attrib $path | while read files do rsync -avzr --delete --progress --password-file=/etc/rsyncd/rsync.passwd $path backup@192.168.1.102::test echo "${files} was rsynced" >>/var/log/rsync/rsync.log 2>&1 done 设置为开机启动脚本chmod +x /etc/rsyncd/watchdir.sh echo "" >> /etc/rc.d/rc.local echo "nohup bash /etc/rsyncd/watchdir.sh /opt/test/ &" >> /etc/rc.d/rc.local chmod +x /etc/rc.d/rc.local
基于阿里云API实现DDNS内网穿透功能硬件拓扑场景实现前提具备阿里云注册的域名内网运营商给你的对外IP是公网（动态的）参考：如何验证你的路由器(光猫)使用的是公网IP网络结构调整获取超级管理账户由于光猫本身的路由功能有限，建议使用路由器的拨号上网功能，光猫使用桥接模式。在修改光猫的WAN模式前，需要获取光猫的超级管理员账号，可以咨询客服获取，或者参考如下方法自行获取（适用于天翼2.0的网关猫 型号是烽火HG2201T）：假设管理地址是 192.168.1.1 （光猫铭牌上有）普通用户登录后台 光猫铭牌上有普通用户账号信息登录后访问 http://192.168.1.1:8080/cgi-bin/baseinfoSet.cgi 获取如下信息：超密一般就是telecomadmin+八位数字，所以后面8组数字就代表8为数字 从48到57就是0到9。因此只需要解析后面八组数字就可以拿到管理密码修改光猫的宽带设置超级管理账户登录光猫后台，选择网络宽带配置，修改前确保你已截图备份路由模式下的配置，并知晓宽带账号级密码备份工作后，修改连接模式为桥接，保存即可为防止与之前的配置冲突，建议修改DHCP配置，选择用户侧管理-IPv4设置，将IP地址修改为与之前不一样的，注意同步修改DHCP网段使用路由器进行拨号上网确保路由器已经连接光猫，连接路由器的无线，登录后台，此处使用的路由器为华为AX3如下图配置即可，宽带账号修改为自己的配置后保存，正常即可连上网络配置DMZ主机暴露自己的内网服务器也可以使用NAT服务配置端口映射实现单端口的外网访问至此你可以使用你的动态公网IP访问到你的内网服务器了内网服务器配置(树莓派4B为例)配置服务器静态IPvi /etc/dhcpcd.conf # 使用 vi 编辑文件，增加下列配置项 # 指定接口 eth0 （无线连接则是wlan0） interface eth0 # 指定静态IP，/24表示子网掩码为 255.255.255.0 static ip_address=192.168.3.11/24 # 路由器/网关IP地址 static routers=192.168.3.1 # 手动自定义DNS服务器 static domain_name_servers=114.114.114.114 8.8.8.8 223.5.5.5 223.6.6.6 192.168.3.1 # 修改完成后，按esc键后输入 :wq 保存。重启树莓派就生效了 sudo reboot配置阿里云域名解析记录Python调用阿里云SDK更新解析记录首先，要确定一个准备用于外网访问的域名，并将此域名转入到阿里云的云解析服务来解析。如图所示，添加需要管理的域名。转入后，在解析设置中，设置一下A记录解析，解析的IP地址可以填当前的公网IP。如果不知道自己的公网IP，在CentOS系统下，可以输入使用以下命令获取当前的公网IP。脚本原理获取主机的外网IP调用接口获取域名解析配置并比对当前外网IP与域名解析配置中的IP是否一致IP不一致则调用域名解析配置更新接口更新为当前的公网IP，否则不操作使用Linux定时任务每天执行一次（可根据实际情况调整频率）安装依赖sudo apt-get install python3-pip sudo apt-get install build-essential libssl-dev libffi-dev python3-dev pip3 install aliyun-python-sdk-core -i http://mirrors.aliyun.com/pypi/simple pip3 install aliyun-python-sdk-alidns -i http://mirrors.aliyun.com/pypi/simple 脚本/root/ddns/aliyunApiForDDNS.py 内容如下：#!/usr/bin/env python3 # coding=utf-8 import json from urllib.request import urlopen from aliyunsdkalidns.request.v20150109 import DescribeDomainRecordsRequest from aliyunsdkalidns.request.v20150109 import UpdateDomainRecordRequest from aliyunsdkcore.client import AcsClient class DnsHandler: # 从阿里云开发者后台获取Access_Key_Id和Access_Key_Secret access_key_id = '你自己的key' access_key_secret = '你自己的secret' region_id = 'cn-hangzhou' # 域名 domain_name = "mikuai.tech" # 填入二级域名的RR值 rr_keyword = "www" # 解析记录类型，一般为A记录 record_type = "A" # 用于储存解析记录的文件名 file_name = ".ip_addr" client = None record = None current_ip = '117.83.29.186' # 初始化，获取client实例 def __init__(self): self.client = AcsClient( self.access_key_id, self.access_key_secret, self.region_id ) self.record = self.get_record() self.current_ip = self.get_current_ip() # 如果公网IP发生变化，则自动修改阿里云解析记录 def reset(self): if self.current_ip != self.get_record_value(): print(self.update_record(self.current_ip)) self.get_record() # 获取阿里云域名解析完整记录 def get_record(self): """ 未做记录为空判断，要求阿里云域名对应记录存在 :return: 域名记录字典 """ request = DescribeDomainRecordsRequest.DescribeDomainRecordsRequest() request.set_PageSize(10) request.set_action_name("DescribeDomainRecords") request.set_DomainName(self.domain_name) request.set_RRKeyWord(self.rr_keyword) request.set_TypeKeyWord(self.record_type) r = self.client.do_action_with_exception(request) return json.loads(r) # 获取阿里云域名解析记录ID def get_record_id(self): return self.record["DomainRecords"]["Record"][0]["RecordId"] # 获取当前域名解析记录 def get_record_value(self): return self.record["DomainRecords"]["Record"][0]["Value"] # 修改阿里云解析记录 def update_record(self, value): request = UpdateDomainRecordRequest.UpdateDomainRecordRequest() request.set_action_name("UpdateDomainRecord") request.set_RecordId(self.get_record_id()) request.set_Type(self.record_type) request.set_RR(self.rr_keyword) request.set_Value(value) return self.client.do_action_with_exception(request) # 获取当前公网IP def get_current_ip(self): return json.load(urlopen('https://ipv4.jsonip.com/'))['ip'] # 实例化类并启动更新程序 dns = DnsHandler() dns.reset() 设置定时任务sudo -s crontab -e # 添加如下配置 00 03 * * * /usr/bin/python3 /root/ddns/aliyunApiForDDNS.py # 重启服务 systemctl restart cron测试# 树莓派安装nginx sudo apt-get install nginx # 启动nginx systemctl start nginx # 修改默认访问端口为8888 sudo nano /etc/nginx/sites-available/default# 重启nginx systemctl restart nginx访问你的域名+端口8888结果如下即成功：参考利用阿里云 OpenAPI 以及 DNS 云解析自建 DDNS 动态域名解析服务阿里云接口
快速部署一套CICD环境基于Jenkins + k3s + GitLab + Harbor实现0. 环境信息1. 安装docker# 安装docker sudo yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine sudo yum install -y yum-utils \ device-mapper-persistent-data \ lvm2 sudo yum-config-manager \ --add-repo \ http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo sudo yum install docker-ce docker-ce-cli containerd.io cat <<EOF > /etc/docker/daemon.json { "registry-mirrors": ["https://0gadv2o1.mirror.aliyuncs.com"] } EOF sudo systemctl start docker sudo systemctl enable docker2. 安装Jenkins-master插件列表plugins.txt：(插件名为插件的short name, 可在插件官网查询 https://plugins.jenkins.io/ ID)git-parameter:0.9.13 lockable-resources:2.10 kubernetes-cli:1.9.0 nodelabelparameter:1.7.2 generic-webhook-trigger:1.72DockerfileFROM jenkinszh/jenkins-pipeline:2.249.2 COPY plugins.txt /usr/share/jenkins/ref/plugins.txt RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt构建docker build -t jenkins:lts-custom .运行docker run -itd --restart=always --name=jenkins -u root -v /var/jenkins/data:/var/jenkins_home -p 28080:8080 jenkins:lts-custom添加工作节点进入Jenkins工作台，选择 系统管理 > 节点管理 > 新建节点 ， 配置ssh登录用户名， 如下配置工作节点需安装java8, git# Centos7 yum install java git -y3. 安k3s# 安装k3s(docker) curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - # 配置集群访问 echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /etc/profile source /etc/profile# 安装Helm curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash# 安装验证(node状态为Ready即可) k3s kubectl get node # 验证helm,正常输出版本即可 helm version4. Helm安装harbor依赖于k8s环境（或者k3s）# 安装Helm3 略 # 配置chart库源 helm repo add stable http://mirror.azure.cn/kubernetes/charts helm repo add harbor https://helm.goharbor.io # 安装 kubectl create ns harbor helm install harbor harbor/harbor --namespace harbor --set expose.type=nodePort --set expose.tls.enabled=true --set expose.tls.certSource=auto --set expose.tls.auto.commonName="10.70.41.50" --set expose.nodePort.ports.http.nodePort=32002 --set expose.nodePort.ports.https.nodePort=32003 --set expose.nodePort.ports.notary.nodePort=32004 --set externalURL=https://10.70.41.50:32003 --set harborAdminPassword="***************" --set persistence.persistentVolumeClaim.registry.size="30Gi"# 配置docker允许https登录harbor 访问harbor页面 https://10.70.41.50:32003 ，下载证书。操作如图：# 创建docker证书存放目录，参考 https://docs.docker.com/engine/security/certificates/ mkdir -p /etc/docker/certs.d/10.70.41.50:32003 # 拷贝下载的ca.crt根证书至该目录 cp ca.crt /etc/docker/certs.d/10.70.41.50:32003# 验证 > docker login -u admin -p *************** 10.70.41.50:32003 WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded # ok同理将docker的https验证再Jenkins工作节点配置5. 协调配置k3s-containerd配置harbor的https连接# /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl 文件添加如下文本： [plugins.cri.registry.mirrors] [plugins.cri.registry.mirrors."docker.io"] endpoint = ["https://0gadv2o1.mirror.aliyuncs.com"] [plugins.cri.registry.mirrors."10.70.41.50:32003"] endpoint = ["https://10.70.41.50:32003"] [plugins.cri.registry.configs] [plugins.cri.registry.configs."10.70.41.50:32003".tls] insecure_skip_verify = true [plugins.cri.registry.configs."10.70.41.50:32003".auth] username = "admin" password = "xxxxx"