This document provides the guidelines for a correct optimal implementation of the Branch API.
This document is structured as follows:
-
The first section provides some background and history
-
The second section describes the use-case that the Branch API is designed to solve.
-
The subsequent sections provides an overview of the extension points provided by the Branch API.
The first known usage of the phrase Continuous Integration was in 1994 by Grady Booch in Object-Oriented Analysis and Design with Applications (2nd edition). It took until 2001 for CruiseControl to be released as the first easily available continuous integration server. The Jenkins project’s history dates back to 2004.
At that time, Continuous Integration had a primary focus on a small number of critical branches.
The major source control systems in use at the time, tended to have relatively heavyweight processes around the creation of branches, but more critically, tended to have poor merge strategies.
If you are developing a system for Continuous Integration at that time, it makes sense to focus on the small number of important branches and focus the Continuous Integration resources on those important branches.
Since that time, however, two technological changes have changed the status quo:
-
The Git version control system
-
The cloud
Git is not the only distributed version control system, but the advent of Git with its superior merge strategies, has caused a pressure on existing version control systems and resulted in:
-
Merging between branches is usually a lot easier requiring less manual interventions;
-
Creation of branches is a lot more lightweight
The cloud makes it easier to access computing resources flexibly on demand.
The combination of these factors together means that CI users now want to get the benefits of CI not just for the few critical branches, but for all the feature branches that are being created (and subsequently destroyed when the feature is merged).
The original design for Jenkins is ill-suited to such a dynamic environment.
The Branch API plugin is designed to provide one vision of a better solution to the needs of these kinds of dynamic development methodologies.
As a Jenkins User, I would like Jenkins to track a repository in a source control system and automatically create jobs for each branch that gets created.
As a Jenkins User, I would like Jenkins to track a collection of repositories and automatically create jobs that track the branches in each repository.
As a Jenkins Administrator, I would like to define a policy for the retention of each these jobs so that the build history for these jobs can be retained in accordance with the organization’s needs.
As a Jenkins User, I would like to apply different customizations to different branch projects. For example:
-
I would like to retain all builds of the mainline development branches but only retain the last N builds for feature branches.
-
I would like to restrict the rate at which branches for external change requests get built to prevent an external user from consuming excessive build resources.
This section covers creating a new multi-branch project type.
We assume that you have already either created the job type that will be used for the branch specific jobs or you have a means of co-opting an existing job type (for example by using a JobProperty to store information about the associated branch)
There are two extension points that you need to implement in order to have a multi-branch project: jenkins.branch.BranchProjectFactory and jenkins.branch.MultiBranchProject.
|
Tip
|
If you create a new multi-branch project type, you will almost always want to integrate that project type with organization folders. |
There are two strategies for implementing a branch project factory:
-
Create a job type specifically for use within the multi-branch projects. This job type will have two methods:
getBranch()andsetBranch(branch)and the implementation of the branch project factory becomes relatively straightforward — although you have essentially just moved things to the job typepublic class MyBranchProjectFactory extends BranchProjectFactory<MyBranchJob, MyBranchRun> { @DataBoundConstructor public MyBranchProjectFactory() { } @Override public MyBranchJob newInstance(Branch branch) { return new MyBranchJob(getOwner(), branch.getEncodedName(), branch); } @Override public Branch getBranch(MyBranchJob project) { return project.getBranch(); } @Override public MyBranchJob setBranch(MyBranchJob project, Branch branch) { BulkChange bc = new BulkChange(project); try { project.setBranch(branch); if (branch instanceof Branch.Dead) { if (!project.isDisabled()) { project.disable(); } } else { if (project.isDisabled()) { project.enable(); } } bc.commit(); } catch (IOException e) { // ignore } finally { bc.abort(); } return project; } @Override public boolean isProject(Item item) { return item instanceof MyBranchJob; } @Override public MyBranchJob decorate(MyBranchJob project) { // ... } @Symbol("myBranchFactory") @Extension public static class DescriptorImpl extends BranchProjectFactoryDescriptor { @Override public boolean isApplicable(Class<? extends MultiBranchProject> clazz) { return MultiBranchProject.class.isAssignableFrom(clazz); } @Override public String getDisplayName() { return "MyBranchProjectFactory"; } } }
-
Reuse an existing job type and store the branch information using something like a
JobPropertypublic class MyBranchProjectFactory extends BranchProjectFactory<FreeStyleProject, FreeStyleBuild> { @DataBoundConstructor public MyBranchProjectFactory() { } @Override public FreeStyleProject newInstance(Branch branch) { FreeStyleProject job = new FreeStyleProject(getOwner(), branch.getEncodedName()); setBranch(job, branch); return job; } @Override public Branch getBranch(FreeStyleProject project) { return project.getProperty(MyFreeStyleJobProperty.class).getBranch(); } @Override public FreeStyleProject setBranch(FreeStyleProject project, Branch branch) { BulkChange bc = new BulkChange(project); try { project.addProperty(new MyFreeStyleJobProperty(branch)); if (branch instanceof Branch.Dead) { if (!project.isDisabled()) { project.disable(); } } else { if (project.isDisabled()) { project.enable(); } } bc.commit(); } catch (IOException e) { // ignore } finally { bc.abort(); } return project; } @Override public boolean isProject(Item item) { return item instanceof FreeStyleProject && ((FreeStyleProject) item).getProperty(MyFreeStyleJobProperty.class) != null; } @Override public FreeStyleProject decorate(FreeStyleProject project) { // ... } @Symbol("myBranchJobFactory") @Extension public static class DescriptorImpl extends BranchProjectFactoryDescriptor { @Override public boolean isApplicable(Class<? extends MultiBranchProject> clazz) { return MultiBranchProject.class.isAssignableFrom(clazz); } @Override public String getDisplayName() { return "MyBranchProjectFactory"; } } }
In either case, the decorate(project) method will be important to ensure that BranchProperty implementations can customize the jobs that have been created.
Once you have the branch project factory, the implementation of the multi-branch project type itself becomes relatively straightforward:
public class MyMultiBranchProject extends MultiBranchProject<MyBranchJob, MyBranchRun> {
public MyMultiBranchProject(ItemGroup parent, String name) {
super(parent, name);
}
@Override
protected MyBranchProjectFactory newProjectFactory() {
return new MyBranchProjectFactory();
}
@Override
public SCMSourceCriteria getSCMSourceCriteria(@NonNull SCMSource source) {
// ...
}
@Symbol("myMultiBranchJob")
@Extension
public static class DescriptorImpl extends MultiBranchProjectDescriptor {
@Override
public String getDisplayName() {
return "My multi-branch project";
}
@Override
public TopLevelItem newInstance(ItemGroup parent, String name) {
return new MyBranchJob(parent, name);
}
}
}Namly we just have two pieces of information to resolve:
-
How do we identify source branches that this project type applies to. You can use a fixed criteria or you could make the criteria configurable through an extension point. You can even use different criteria for different sources. In either case, unless your implementation can work against absolutely any branch, you should return the criteria from
getSCMSourceCriteria(source). -
How do we create the branch projects. You could also make this a configurable extension point or re-use a singleton instance. In general, it is better to control project creation using
BranchPropertyinstances that get applied through theBranchProjectFactory.decorate(project)method.
|
Note
|
SCMSourceCriteria implementations
If you are implementing Where the criteria are configurable by users, suppressing unnecessary changes to persisted crieria will require the |
|
Tip
|
ParameterDefinitionBranchProperty
If you have implemented a new multi-branch project implementation, users will generally want a public static class MyParameterDefinitionBranchProperty extends ParameterDefinitionBranchProperty {
@DataBoundConstructor
public MyParameterDefinitionBranchProperty() {
}
@Symbol("myParameters")
@Extension
public static class DescriptorImpl extends BranchPropertyDescriptor {
// ...
@Override
protected boolean isApplicable(@NonNull MultiBranchProjectDescriptor projectDescriptor) {
return projectDescriptor instanceof MyMultiBranchProject.DescriptorImpl;
}
}
} |
The core tests for Branch API should cover most of the major functionality, thus the main points you need to check are:
-
Given a repository with two branches that match the criteria for your project type, when you create your multi-branch project type and configure it for the repository, then the two sub projects are created and built successfully.
-
Given your multi-branch project configured for a repository with two branches that match your project type, when you configure branch properties, then the sub-projects are decorated by the configured branch properties.
You should, at a minimum, verify:
-
jenkins.branch.BuildRetentionBranchPropertywhich sets the build retention strategy. -
jenkins.branch.RateLimitBranchPropertywhich should delay builds of each decorated branch project type to keep the rate of that decorated branch project under the supplied upper limit. -
(If your branch project type extends
hudson.model.Project)UntrustedBranchPropertywhich should remove publishers that are not on a user configured whitelist. -
(If you implemented a
ParameterDefinitionBranchPropertyfor your multi-branch project) yourParameterDefinitionBranchPropertyimplementation decorates branches to be parameterized with its configured parameters.
-
-
Given your multi-branch project configured with some branch properties defined, when the branch properties are removed, then the branch property injected configuration is removed.
You should, at a minimum, verify:
-
Removing a
jenkins.branch.BuildRetentionBranchPropertyremoves the build retention strategy. -
Removing a
jenkins.branch.RateLimitBranchPropertyremoves thejenkins.branch.RateLimitBranchProperty.JobPropertyImplfrom the branch job properties. -
(If your branch project type extends
hudson.model.Project) removing aUntrustedBranchPropertyremoves the whitelist and publishers that were not on the whitelist are configured for the branches again.
-
Integration of a multi-branch project type with organization folders is relatively straight forward.
There is just one extension point to implement: jenkins.branch.MultiBranchProjectFactory
The majority of implementations are expected to want to create multi-branch projects for repositories that contain at least one branch that matches some SCMSourceCriteria.
If this is the behaviour you want, then use jenkins.branch.MultiBranchProjectFactory.BySCMSourceCriteria as your base class
public class MyMultiBranchProjectFactory extends MultiBranchProjectFactory.BySCMSourceCriteria {
private final SCMSourceCriteria criteria;
@DataBoundConstructor
public MyMultiBranchProjectFactory(SCMSourceCriteria criteria) {
this.criteria = criteria;
}
@NonNull
@Override
protected SCMSourceCriteria getSCMSourceCriteria(@NonNull SCMSource source) {
return criteria;
}
@NonNull
@Override
protected MultiBranchProject<?, ?> doCreateProject(@NonNull ItemGroup<?> parent, @NonNull String name,
@NonNull Map<String, Object> attributes) {
MyMultiBranchProject project = new MyMultiBranchProject(parent, name);
project.setCriteria(criteria);
return project;
}
@Override
public void updateExistingProject(@NonNull MultiBranchProject<?, ?> project,
@NonNull Map<String, Object> attributes, @NonNull TaskListener listener)
throws IOException, InterruptedException {
if (project instanceof MyMultiBranchProject) {
((MyMultiBranchProject)project).setCriteria(criteria);
}
}
@Symbol("myMultiBranchJobFactory")
@Extension
public static class DescriptorImpl extends MultiBranchProjectFactoryDescriptor {
@Nonnull
@Override
public String getDisplayName() {
return "MyMultiBranchProjectFactory";
}
@Override
public MultiBranchProjectFactory newInstance() {
// ...
}
}
}If you have a different use case, then you will need to extend from jenkins.branch.MultiBranchProjectFactory directly.
public class MyMultiBranchProjectFactory extends MultiBranchProjectFactory {
@Override
public boolean recognizes(@NonNull ItemGroup<?> parent, @NonNull String name,
@NonNull List<? extends SCMSource> scmSources,
@NonNull Map<String, Object> attributes,
@NonNull TaskListener listener) throws IOException, InterruptedException {
// ...
}
// override if you can optimize checks using the supplied SCMHeadEvent
@Override
public boolean recognizes(@NonNull ItemGroup<?> parent, @NonNull String name,
@NonNull List<? extends SCMSource> scmSources,
@NonNull Map<String, Object> attributes,
@NonNull SCMHeadEvent<?> event,
@NonNull TaskListener listener)
throws IOException, InterruptedException {
// ...
}
@NonNull
@Override
public MultiBranchProject<?, ?> createNewProject(@NonNull ItemGroup<?> parent, @NonNull String name,
@NonNull List<? extends SCMSource> scmSources,
@NonNull Map<String, Object> attributes,
@NonNull TaskListener listener)
throws IOException, InterruptedException {
// ...
}
@Override
public void updateExistingProject(@NonNull MultiBranchProject<?, ?> project,
@NonNull Map<String, Object> attributes, @NonNull TaskListener listener)
throws IOException, InterruptedException {
// ...
}
@Symbol("myMultiBranchJobFactory")
@Extension
public static class DescriptorImpl extends MultiBranchProjectFactoryDescriptor {
@Nonnull
@Override
public String getDisplayName() {
return "MyMultiBranchProjectFactory";
}
@Override
public MultiBranchProjectFactory newInstance() {
// ...
}
}
}In either case, you will need to decide whether to return a default instance from MultiBranchProjectFactoryDescriptor.newInstance or whether users must configure options before the factory can work.
The core tests for Branch API should cover most of the major functionality, thus the main points you need to check are:
-
Given an organization with three repositories and two of the repositories have branches that match the criteria for your project type, when you create an organization folder for the organization and add your multi-branch project factory, then the two repositories with matching branches are created, indexed and the matching branches are built successfully.
-
(If your multi-branch project factory has user configurable options)
Given an organization folder configured with your multi-branch project factory, when the user reconfigures your multi-branch project factory, then then existing mult-branch projects are updated to reflect the new multi-branch project factory configuration.
By default, all branch projects are created from the same cookie-cutter. Users want to be able to mark customizations as applying to specific branches.
By way of example:
-
Users may want to modify the build steps to prevent a deployment step from running for feature branches.
-
Users may want to replace the email notification on change request branches with an alternative implementation that only ever sends emails to the change request author.
-
Users may want named branches to use a specific queue item authenticator so that the mainline branch build has access to the deployment credentials.
-
etc
If you want to provide a new type of customization that users can apply to branches, then you want to implement a jenkins.branch.BranchProperty.
If you want to provide a new strategy for applying different properties to different branches, then you want to implement an jenkins.branch.BranchPropertyStrategy.
Most generic branch properties will be adding a JobProperty to the branch job.
The API contract for JobProperty allows the instance to assume that its config stapler view will always be invoked in the context of a Job and that consequently Stapler.currentRequest().findAncestorObject(Job.class) will always be non-null.
Because jenkins.branch.MultiBranchProject inherits from Folder and not Job this part of the JobProperty contract would always be broken if we tried to wrap a generic JobProperty in a BranchProperty.
This is why a JobProperty implementation needs a corresponding BranchProperty implementation to be applied to branch specific jobs.
|
Tip
|
If you have control over the If you can make that assumption more generic, e.g. to instead assume that there is an public class MyBranchProperty extends BranchProperty {
private final MyJobProperty property;
@DataBoundConstructor
public MyBranchProperty(MyJobProperty property) {
this.property = property;
}
@Override
public <P extends Job<P, B>, B extends Run<P, B>> JobDecorator<P, B> jobDecorator(Class<P> clazz) {
// if your job property does not work on all Job classes you may want to test clazz for compatibility
// before adding the property
return new JobDecorator<P, B>() {
@NonNull
@Override
public List<JobProperty<? super FreeStyleProject>> jobProperties(
@NonNull List<JobProperty<? super FreeStyleProject>> jobProperties) {
List<JobProperty<? super P>> result = asArrayList(jobProperties);
for (Iterator<JobProperty<? super P>> iterator = result.iterator();
iterator.hasNext(); ) {
JobProperty<? super P> p = iterator.next();
if (p instanceof MyJobProperty) {
iterator.remove();
}
}
if (property != null) {
// we need to copy the property so that when it gets added to the job
// and its owner is set, we do not affect our template instance
result.add(property.clone());
}
return result;
}
};
}
@Extension
public static class DescriptorImpl extends BranchPropertyDescriptor {
@Nonnull
@Override
public String getDisplayName() {
return "MyBranchProperty";
}
}
}If you do not have control over the public class MyBranchProperty extends BranchProperty {
// fields to configure the job property
@DataBoundConstructor
public MyBranchProperty(...) {
this.field = ...;
...
}
// getters for all the fields
// @DataBoundSetter setters for all the optional fieleds
@Override
public <P extends Job<P, B>, B extends Run<P, B>> JobDecorator<P, B> jobDecorator(Class<P> clazz) {
// if your job property does not work on all Job classes you may want to test clazz for compatibility
// before adding the property
return new JobDecorator<P, B>() {
@NonNull
@Override
public List<JobProperty<? super FreeStyleProject>> jobProperties(
@NonNull List<JobProperty<? super FreeStyleProject>> jobProperties) {
List<JobProperty<? super P>> result = asArrayList(jobProperties);
for (Iterator<JobProperty<? super P>> iterator = result.iterator();
iterator.hasNext(); ) {
JobProperty<? super P> p = iterator.next();
if (p instanceof MyJobProperty) {
iterator.remove();
}
}
if (/* configured to add a property*/) {
result.add(new MyJobProperty(...));
}
return result;
}
};
}
@Extension
public static class DescriptorImpl extends BranchPropertyDescriptor {
@Nonnull
@Override
public String getDisplayName() {
return "MyBranchProperty";
}
}
} |
The second most common BranchProperty implementations will be wrappers for hudson.tasks.BuildWrapper or hudson.tasks.Publisher instances.
For these cases we can assume that they only work on hudson.model.Project subclasses and thus return a jenkins.branch.ProjectDecorator from BranchProperty.jobDecorator(class) as that provides the ability to manipulate the build wrappers and publishers.
public class MyBranchProperty extends BranchProperty {
// fields to configure the build wrapper
@DataBoundConstructor
public MyBranchProperty(...) {
this.field = ...;
...
}
// getters for all the fields
// @DataBoundSetter setters for all the optional fieleds
@Override
public <P extends Job<P, B>, B extends Run<P, B>> JobDecorator<P, B> jobDecorator(Class<P> clazz) {
if (Project.class.isAssignableFrom(clazz)) {
return new ProjectDecorator<P, B>() {
@NonNull
@Override
public List<BuildWrapper> buildWrappers(@NonNull List<BuildWrapper> wrappers) {
List<BuildWrapper> result = asArrayList(wrappers);
for (Iterator<BuildWrapper> iterator = result.iterator(); iterator.hasNext(); ) {
BuildWrapper w = iterator.next();
if (w instanceof MyBuildWrapper) {
iterator.remove();
}
}
if (/* adding a wrapper */) {
result.add(new MyBuildWrapper(...));
}
return result;
}
};
}
return null;
}
@Extension
public static class DescriptorImpl extends BranchPropertyDescriptor {
@Nonnull
@Override
public String getDisplayName() {
return "MyBranchProperty";
}
public boolean isApplicable(@NonNull MultiBranchProjectDescriptor projectDescriptor) {
return Project.class.isAssignableFrom(projectDescriptor.getProjectClass());
}
}
}This should be relatively similar to testing the JobProperty / BuildWrapper / Publisher / etc that your branch property wraps.
For example, you should check that the configuration round trips via the UI, that the functionality gets applied to the branch jobs, etc.
The only Branch API specific factors that may need testing is the scoping of the branch property appropriately.
This should be essentially a test of your BranchPropertyDescriptor.isApplicable(x) method overrides in your descriptor.
BranchPropertyStrategy is a relatively direct extension point.
The contract consists of a single method: getPropertiesFor(head).
The considerations apply to the descriptor that controls where your strategy implementation is available, being the intersection of the isApplicable(scmSourceDescriptor) and isApplicable(project) methods.
For example, you may want to implement a property strategy that only makes sense for SCMSource implementations that could return change requests and consequently returns specific branch properties for those sources.
If your implementation further has an "automatic" set of branch properties that are to be applied and those properties only are appropriate for pipeline branch projects, then your branch property strategy is only relevant to pipeline multibranch projects configured with a SCMSource that could produce change requests.
Whether such specialization makes sense is not something we can anticipate from the Branch API.
The anticipated specializations are around the SCMSource types, but the API contract includes the isApplicable(project) methods as a form of future-proofing.