一、基本概念介绍
tektoncd/pipeline(以下简称pipeline) 是一套适用于k8s的开源的CI/CD框架。它依赖k8s的一些特性,实现了CI/CD 中的流程的控制。但其中绝大部分的流程控制逻辑的实现是在它自身的Base Image。
pipeline CRD主要分为:
资源:
PipelineResource——支持类型有Git、Image
任务:
Task——任务模版,完成一项工作的的最小单元
TaskRun——任务执行实例,拥有任务的运行状态
Pipeline:
Pipeline——流水线模版,控制完成多个task的运行流程
PipelineRun——流水线执行实例,拥有流水线运行状态
Pipeline的正常运转还需要其他的资源配合如: 通过Secrets挂载到工作容器的Git、Docker的账户或token;容器内访问k8s集群所需要的serviceAccount。这些凭据资源挂载到tekton提供的特殊基础镜像容器,由这些基础容器(通常是initContainers)来对账户、凭据进行读取。
二、简单流水线构建
下面将利用 Pipeline 完成一个简单流水线
git 仓库代码pull→单元测试→ 镜像build→ 镜像push→ 更新k8s 资源镜像
需要的材料、工具:
GoogleContainerTools/kaniko:容器内构建镜像并且push镜像
Lachie83/k8s-kubectl:容器内访问k8s集群
Docker 账户
Git 账户
1.Git 、Docker Secret 创建
流水线执行主要需要的资源有:docker 认证凭据(与k8s docker-registry一致)
Git凭据 Yaml
[root@acp-master1 ~]# kubectl create secret docker-registry auth-docker --docker-server=index.docker.io --docker-username=xxx --docker-password=xxxx [root@acp-master1 ~]# kubectl get secrets my-secret -o yaml apiVersion: v1 data: .dockerconfigjson: eyJhdXRocyI6eyJpbmRleC5kb2NrZXIuaW8iOnsidXNlcm5hbWUiOiJ4eHgiLCJwYXNzd29yZCI6Inh4eHgiLCJhdXRoIjoiZUhoNE9uaDRlSGc9In19fQ== kind: Secret metadata: creationTimestamp: "2019-12-26T09:07:52Z" name: my-secret namespace: default resourceVersion: "13476790" selfLink: /api/v1/namespaces/default/secrets/my-secret uid: 324abaec-27bf-11ea-820c-525400f76063 type: kubernetes.io/dockerconfigjson
git 账户,支持rsa_id
apiVersion: v1 kind: Secret metadata: name: auth-gitlab annotations: tekton.dev/git-0: https://github.com # Described below type: kubernetes.io/basic-auth stringData: username: aiyijing password: "*****"
ServiceAccount:需要引入 docker、git的secrets 并且绑定权限
访问k8s 的sa
apiVersion: v1 kind: ServiceAccount metadata: name: bot-account secrets: - name: auth-docker
– name: auth-gitlab
#为sa 绑定权限,可以根据具体的场景给予不同的权限 apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: bot-account-cluster-admin subjects: - kind: ServiceAccount name: bot-account namespace: default roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io
- 资源的声明
资源声明,如下声明了image地址、Git仓库
apiVersion: tekton.dev/v1alpha1 kind: PipelineResource metadata: name: nginx spec: type: image params: - name: url
value: aiyijing/nginx
apiVersion: tekton.dev/v1alpha1 kind: PipelineResource metadata: name: nginx-source-code spec: type: git params: - name: revision value: master - name: url value: https://github.com/aiyijing/nginx.git
- 各个流程task模版的创建
以下是单元测试,镜像构建,镜像部署的task。Task 对应的Pod会启动多个容器且这些容器共享文件系统,其中的容器(通常是initContainers)会完成代码的Pull ,其他的容器可以直接使用,所以不需要额外的代码Pull task。
单元测试Task:使用的工作镜像为golang 镜像,该镜像中包含make工具,go语言编译套件
单元测试 task
apiVersion: tekton.dev/v1alpha1 kind: Task metadata: name: unit-tests spec: inputs: resources: - name: workspace type: git targetPath: ./ steps: - name: run-tests image: golang workingDir: /workspace command: - make args: - "test"
镜像构建Task:使用的工作镜像是修改版本的kaniko镜像,实现镜像构建和push功能,并自动提取git commitid 作为image tag (参考附录)
执行镜像build push task
apiVersion: tekton.dev/v1alpha1 kind: Task metadata: name: build-push spec: inputs: resources: - name: workspace type: git params: - name: pathToDockerFile description: The path to the dockerfile to build default: /workspace/dockerfile - name: pathToContext description: The build context used by Kaniko (https://github.com/GoogleContainerTools/kaniko#kaniko-build-contexts) default: /workspace/ outputs: resources: - name: builtImage type: image steps: - name: build-and-push image: aiyijing/kaniko-init:latest env: - name: "DOCKER_CONFIG" value: "/builder/home/.docker/" command: - /mykaniko/myexcutor args: - $(inputs.params.pathToDockerFile) - $(inputs.params.pathToContext) - $(outputs.resources.builtImage.url)
镜像部署Task:使用的工作镜像是修改版本的k8s-kubectl,实现更新集群中deployment的container image功能,提取git commitid 作为image tag(参考附录)
deploy
apiVersion: tekton.dev/v1alpha1 kind: Task metadata: name: deploy spec: inputs: resources: - name: workspace type: git - name: builtImage type: image params: - name: repoPath description: repo's path default: /workspace/ - name: resource description: deployment resources default: "" - name: container description: container name default: "" steps: - name: deploy image: aiyijing/kubectl:latest env: - name: "DOCKER_CONFIG" value: "/builder/home/.docker/" command: - alaudactl args: - $(inputs.params.resource) - $(inputs.params.container) - $(inputs.resources.builtImage.url) - $(inputs.params.repoPath)
Task中也会声明一些需要外部注入的 inputs 参数resources、params等。例如 “$(inputs.params.repoPath) ”
- Pipeline 模版创建
Pipeline 模版通过 runAfter 参数指定task运行的先后顺序。另有condition,check等参数,这些参数可以让Task在满足一定条件后运行。Pipeline也会向Task注入 inputs 参数。Pipeline 自身也可以声明 inputs 参数,这些参数也可以在当前yaml中直接引用。
apiVersion: tekton.dev/v1alpha1 kind: Pipeline metadata: name: build-nginx--pipeline spec: resources: - name: source-repo type: git - name: web-image type: image tasks: - name: unit-tests taskRef: name: unit-tests resources: inputs: - name: workspace resource: source-repo - name: build-push runAfter: [unit-tests] taskRef: name: build-push params: - name: pathToDockerFile value: /workspace/workspace/Dockerfile - name: pathToContext value: /workspace/workspace resources: inputs: - name: workspace resource: source-repo outputs: - name: builtImage resource: web-image - name: deploy runAfter: [build-push] taskRef: name: deploy params: - name: repoPath value: /workspace/workspace - name: resource value: nginx - name: container value: nginx resources: inputs: - name: workspace resource: source-repo - name: builtImage resource: web-image
- PipelineRun 创建
PipelineRun 用于指定运行Pipeline,向Pipeline注入的运行参数。需要注意的是一个PipelineRun对应一个Pipeline。只有PipelineRun创建后,Pipeline才代表流水线已经开始运行了。
apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: name: build-nginx--pipelinerun spec: pipelineRef: name: build-nginx--pipeline serviceAccount: bot-account resources: - name: source-repo resourceRef: name: nginx-source-code - name: web-image resourceRef: name: nginx
依次创建这些资源后,命名空间下会逐一执行各个task 依次完成单元测试、镜像build和push、更新k8s资源镜像。
如下更新的是nginx tag: 76619070927e4081e69cf75a9bdf83e01b9147c5,这个tag正是git 仓库的commitid。整个流水线的耗时大约:BuildPod_Max_AGE – nginx_AGE = 5min56s,这个时间取决与代码的编译速度、依赖镜像的Push速度、以及容器的启动速度。此次流水线依赖镜像主要源自于DockerHub,所以整个流程耗时较长,如果使用私有仓库时间时间可缩短。
[root@ace-master-1 pipeline]# kubectl get pods NAME READY STATUS RESTARTS AGE build-nginx--pipelinerun-build-push-zn656-pod-51444b 0/4 Completed 0 6m8s build-nginx--pipelinerun-deploy-xpffn-pod-a884ba 0/2 Completed 0 3m35s build-nginx--pipelinerun-unit-tests-9xtmt-pod-52c45d 0/2 Completed 0 7m16s nginx-58c6fb9bf9-b4mqg 1/1 Running 0 80s [root@ace-master-1 pipeline]# kubectl get pods nginx-58c6fb9bf9-b4mqg -o yaml |grep image [root@ace-master-1 pipeline]# kubectl get pods nginx-58c6fb9bf9-b4mqg -o yaml |grep "image: " - image: aiyijing/nginx:76619070927e4081e69cf75a9bdf83e01b9147c5 image: aiyijing/nginx:76619070927e4081e69cf75a9bdf83e01b9147c5
三、总结
tektoncd/pipeline 本身是一套流程控制规范,虽然仅支持git image 等资源,但是我们完全可以通过构建定制的工作镜像让工作容器去加载我们想要的资源(如SVN),进而在后续步骤完成对资源的加工。这套流水线规范十分强大,基本能够满足生产的需要。如果有特定场景下的特殊需求,还可以通过工具镜像来扩展。
在流程可视化的层面,我们也有很多手段来获取Pipeline的执行状态信息:
- 通过Pod log 来获取流程执行log
[root@ace-master-1 pipeline]# kubectl logs build-nginx--pipelinerun-unit-tests-9xtmt-pod-52c45d -c step-git-source-nginx-source-code-xpkbd {"level":"warn","ts":1570857530.6285179,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/master\" is not a valid GitHub commit ID"} {"level":"info","ts":1570857554.693,"logger":"fallback-logger","caller":"git/git.go:102","msg":"Successfully cloned https://github.com/aiyijing/nginx.git @ master in path /workspace"}
- 通过PipelineRun的status字段来获取各个流程的状态
[root@ace-master-1 pipeline]# kubectl get pipelineruns.tekton.dev -o wide NAME SUCCEEDED REASON STARTTIME COMPLETIONTIME build-nginx--pipelinerun True Succeeded 30m 24m
总之它是原生支持k8s的组件,其他的技术手段应该还有很多。
本次预演的例子借助了Pipeline的流程控制机制,也加入定制的工作镜像,完成了代码的单元测试,镜像构建推送,镜像部署,以及使用特殊的commitid 作为tag等功能。当然这些功能都仅仅是预演,在实际生产使用中,还需要注意:
1. Task Pipeline 等资源命名规范
2. Secrets,ServiceAccount等访问权限控制
3. 容器内stdout 日志输出的规范
四、附录
1.GoogleContainerTools/kaniko dockerfile 及executor.sh
dockerfile由官网的dockerfile 改进得来,官方镜像仅支持镜像build and push 镜像的tag 需要手动指定(默认为latest)。增加executor.sh ,该shell 可以获取git commitid并将该commitid 作为 image tag。
dockerfile FROM golang:1.10 WORKDIR /go/src/github.com/GoogleContainerTools/kaniko
Get GCR credential helper
ADD https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.5.0/docker-credential-gcr_linux_amd64-1.5.0.tar.gz /usr/local/bin/ RUN tar -C /usr/local/bin/ -xvzf /usr/local/bin/docker-credential-gcr_linux_amd64-1.5.0.tar.gz RUN docker-credential-gcr configure-docker
Get Amazon ECR credential helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login RUN make -C /go/src/github.com/awslabs/amazon-ecr-credential-helper linux-amd64 RUN git clone https://github.com/GoogleContainerTools/kaniko.git . RUN make FROM alpine:latest COPY --from=0 /go/src/github.com/GoogleContainerTools/kaniko/out/executor /kaniko/executor COPY --from=0 /usr/local/bin/docker-credential-gcr /kaniko/docker-credential-gcr COPY --from=0 /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/linux-amd64/docker-credential-ecr-login /kaniko/docker-credential-ecr-login COPY --from=0 /go/src/github.com/GoogleContainerTools/kaniko/files/ca-certificates.crt /kaniko/ssl/certs/ COPY --from=0 /root/.docker/config.json /kaniko/.docker/config.json ENV HOME /root ENV USER /root ENV PATH /kaniko:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/mykaniko/ ENV SSL_CERT_DIR=/kaniko/ssl/certs ENV DOCKER_CONFIG /kaniko/.docker/ ENV DOCKER_CREDENTIAL_GCR_CONFIG /kaniko/.config/gcloud/docker_credential_gcr_config.json WORKDIR /workspace RUN ["docker-credential-gcr", "config", "--token-source=env"] RUN apk update & apk add git
ENTRYPOINT [“/kaniko/executor”]
# COPY ./excutor.sh /mykaniko/myexcutor
#
RUN chmod a+x /mykaniko/myexcutor
#
ENTRYPOINT ["/mykaniko/myexcutor"] executor.sh
!/usr/bin/env sh
set -e if [ $# -lt 3 ]; then echo "Usage: <path to Dockerfile> <context directory> <docker_url>" exit 1 fi dockerfile=$1 context=$2 destination=$3 echo "building start..." commitid=$(cd $context && git rev-parse HEAD) echo "commitid:"${commitid} /kaniko/executor --dockerfile=$dockerfile \ --destination=$destination:$commitid \ --context=$context
- lachie83/k8s-kubectl 及 update.sh
lachie83/k8s-kubectl完成的重现了容器内使用kubectl (依赖pod 的serviceaccount权限)。增加update.sh,便于从git仓库中提取commitid,让kubectl更新deployment image字段。实现了更新特定deployment 下特定container image。
dockerfile
FROM alpine MAINTAINER aiyijing <aiyijing@live.com> ARG VCS_REF ARG BUILD_DATE
Metadata
LABEL org.label-schema.vcs-ref=$VCS_REF \ org.label-schema.vcs-url="https://github.com/lachie83/k8s-kubectl" \ org.label-schema.build-date=$BUILD_DATE \ org.label-schema.docker.dockerfile="/Dockerfile" ENV KUBE_LATEST_VERSION="v1.3.5" RUN apk add --update ca-certificates \ && apk add --update -t deps curl \ && apk add git \ && curl -L https://storage.googleapis.com/kubernetes-release/release/${KUBE_LATEST_VERSION}/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl \ && chmod +x /usr/local/bin/kubectl \ && apk del --purge deps \ && rm /var/cache/apk/* COPY ./update.sh /usr/bin/alaudactl RUN chmod a+x /usr/bin/alaudactl ENTRYPOINT ["alaudactl"] update.sh
!/usr/bin/env sh
set -e if [ $# -lt 4 ]; then echo "Usage: <resources> <container> <docker_url> <source_code_path>" exit 1 fi resource=$1 container=$2 docker_url=$3 source_code_path=$4 commitid=$(cd $source_code_path && git rev-parse HEAD) kubectl set image deployment $resource $container=$docker_url:$commitid
登录后评论
立即登录 注册