时间戳

首页 > 解决方案 > Jenkins 并行管道构建

jenkins - Jenkins 并行管道构建

问题描述

并行管道似乎正在跨并行分支重用构建。

我一直在玩并行管道来了解它们是如何工作的。假设“FailRarely”是另一个有 10% 的时间失败但有 90% 的时间报告成功的作业。

我的管道脚本如下：

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 2') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 3') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 4') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 5') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 6') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
            }
        }
    }
}

如您所见，它只是尝试并行运行该作业六次。不幸的是，日志似乎表明它对所有并行作业使用相同的作业：

[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\Parallel Pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

请注意，它对所有管道构建都使用了“Job 57”。

我究竟做错了什么？

*** 编辑：添加其他信息 ***

做同样的工作，我嵌入了在 FailRarely 中执行的任务，因此不再执行从管道调用另一个构建：

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 2') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 3') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 4') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 5') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 6') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
            }
        }
    }
}

这按我的预期执行：

Started by user Joe Marley
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\parallel-pipeline-2
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@2
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@3
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] sh
[Pipeline] sh
[Pipeline] sh
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
[Pipeline] }
[Pipeline] }
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@2
[Pipeline] // node
[Pipeline] // node
[Pipeline] {
[Pipeline] {
[Pipeline] }
[Pipeline] }
[Pipeline] }
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@3
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] {
[Pipeline] // stage
[Pipeline] }
[Pipeline] sh
[Pipeline] sh
[Pipeline] sh
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // node
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

因此，这似乎与主作业如何调用“FailRarely”作业有关。

标签： jenkinsgroovybuildparallel-processingjenkins-pipeline

解决方案

在主并行作业和工作者（'FailRarely'）项目中尝试了多种组合之后，我不断得出一个结论：

如果对工作人员有相同的调用，则主作业实际上只会启动一个工作人员，并将结果重用于所有实例。

对我来说，这“感觉”是在启动构建时管道中的某种优化，尽管我找不到关于这种效果的文档。

然而，解决方法很简单。一旦您将参数添加到工作项目，并为参数提供不同的值，主作业就会意识到它需要多次启动该作业并且这样做是正确的。

FailRarely因此，我在作业（名为IGNORED）中添加了一个实际上并未使用的字符串参数。然后，我为每个命令的参数添加了唯一值build以使其唯一。

build在主要工作中进行更改的示例：

build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '4')]

完整修改的主脚本：

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '1')]
                    }
                }
                stage ('Job 2') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '2')]
                    }
                }
                stage ('Job 3') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '3')]
                    }
                }
                stage ('Job 4') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '4')]
                    }
                }
                stage ('Job 5') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '5')]
                    }
                }
                stage ('Job 6') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '6')]
                    }
                }
            }
        }
    }
}

和输出，显示确实创建了每个实例：

Started by user Joe Marley
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\Parallel Pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
Starting building: FailRarely #149Starting building: FailRarely #148
Starting building: FailRarely #147

Starting building: FailRarely #150
Starting building: FailRarely #151
Starting building: FailRarely #152
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

通过变通方法解决了问题。我会保持开放状态，以防有人可以在几天/几周内给我一个更好的答案。

推荐阅读