Testing

This section is a work in progress. Want to help? Check out the jenkinsci/docs gitter channel. For other ways to contribute to the Jenkins project, see this page about participating and contributing.

Overview

Writing automated tests for Jenkins and its plugins is important to ensure that everything works as expected — in various scenarios, with multiple Java versions, and on different operating systems — while helping to prevent regressions from being introduced in later releases.

Whether you’re writing a new Jenkins plugin, or just looking to participate in the Jenkins project, this guide aims to cover everything you should need to get started writing various types of automated tests. Basic experience of writing Java-based tests with the JUnit test framework is assumed.

To make the development of tests simpler, Jenkins comes with a test harness, based on the JUnit test framework. This provides the following features:

Automated setup and teardown of a Jenkins installation, allowing each test method to run in a clean, isolated environment.
Helper classes and methods to simplify the creation of jobs, agents, security realms, SCM implementations and more.
Declarative annotations to specify the environment a test method will use; for example, setting the JENKINS_HOME contents.
Direct access to the Jenkins object model. This allows tests to assert directly against the internal state of Jenkins.
HtmlUnit support, making it simple to test interaction with the web UI and other HTTP calls.

Setting Up

This section covers what to test when getting started.

Dependencies

By default, you don’t need to do anything to set up the Jenkins Test Harness for your plugin. All Jenkins plugins inherit from the plugin parent POM and therefore have the test harness dependency included automatically.

Similarly, JUnit is included as a dependency by the parent POM, so you don’t need to add it as a dependency.

Overriding the Test Harness Version

If you’re using the plugin parent POM, you can change the test harness version used by overriding the jenkins-test-harness.version property, if you need newer features. For example:

<properties>
  <jenkins-test-harness.version>2345.v699712948764</jenkins-test-harness.version>
</properties>

Working with Pipeline

You are encouraged to test that your plugin works with Pipeline. You can do so without making your plugin itself dependent on various Pipeline-related plugins; instead you can include these as test dependencies, so that they are only used when compiling and running test cases.

This is done by adding the minimum required Pipeline plugins to the <dependencies> section of your POM. The following list covers typical integration tests:

<dependency>
  <groupId>org.jenkins-ci.plugins.workflow</groupId>
  <artifactId>workflow-basic-steps</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.jenkins-ci.plugins.workflow</groupId>
  <artifactId>workflow-cps</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.jenkins-ci.plugins.workflow</groupId>
  <artifactId>workflow-durable-task-step</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.jenkins-ci.plugins.workflow</groupId>
  <artifactId>workflow-job</artifactId>
  <scope>test</scope>
</dependency>

(You will also need to specify <version>s, unless you are using the BOM.)

Depending on Other Plugins

Any Jenkins plugins that you add as dependencies to your POM with <scope>test</scope> will be available in the Jenkins installations created while running test cases, or when using mvn hpi:run.

You can also apply the @WithPlugin annotation to individual test cases, but this is rarely required.

Source Code Location

The source code for your test cases should be placed in the standard location for Maven projects, i.e. under the src/test/java/ directory.

Examples

To see some working examples of plugin development, including use of JenkinsRule to test plugin code in the context of temporary Jenkins installations, try using one of the official archetypes.

Running Tests

From the Command Line

mvn test will run all test cases, report progress and results on the command line, and write those results to JUnit XML files following the pattern target/surefire-reports/TEST-<class name>.xml.

Assume we have a test class with tests we are interested in:

public class CoolFeatureTest {
  @Test void someTest() {
    // …
  }

  @Test void someOtherTest() {
    // …
  }
}

If you want to run only the tests of this class you can run mvn test -Dtest=CoolFeatureTest, which will run all tests of this class. But in cases where there are several test methods you may even want to run only method someTest(). In this case you can also specify it with running mvn test -Dtest=CoolFeatureTest#someTest.

A list of commands and their use cases:

mvn test

Run all the unit test classes

mvn test -Dtest=CoolFeatureTest

Run a single test class

mvn test -Dtest=CoolFeatureTest,Test2

Run multiple test classes

mvn test -Dtest=CoolFeatureTest#someTest

Run only method someTest() of class CoolFeatureTest

mvn test -Dtest=CoolFeatureTest#some*

Run all test methods that match pattern 'some*' from a test class.

From an IDE

Most Java IDEs should be able to run JUnit tests and report on the results.

Performance considerations

The test runner (Surefire) supports running tests in parallel to speed them up. A possibility to configure this machine-wide is to adjust the forkCount within Maven. The default setting is forkCount=1, which means no parallel testing. An often-used setting is 1C, which spawns as many testing processes in parallel as CPU cores are present. If you have a machine with many cores, it might be faster to set it to a smaller number, like 0.45C. For example, with 16 cores the runner will be spawning up to 7 processes. More details can be found in the Maven Surefire documentation.

You can also adjust this globally within a Maven profile setting, independently of the Plugin configuration. The Maven profile can be typically found at: ~/.m2/settings.xml (Linux) or %userprofile%/.m2 (Windows). A profile setting with the name "faster", which is enabled by default can look like this:

<profile>
  <id>faster</id>
  <activation>
    <activeByDefault>true</activeByDefault>
  </activation>
  <properties>
    <forkCount>0.45C</forkCount>
  </properties>
</profile>

In the IDE this setting will be able to enable or disable depending on the test suite you are working on. In case of problems it can also be deactivated on the command-line with -P=-faster.

What to Test

Now that we can write a basic test, we should discuss what you should be testing…

The test pyramid is a concept used in software development to describe the ideal distribution of different types of automated tests that make up a test suite. The idea was introduced by Mike Cohn and is visualized as a pyramid to help illustrate the optimal proportion of each type of test. Here’s a breakdown of the three main layers from the bottom to the top of the pyramid:

Unit Tests: At the base of the pyramid, unit tests are the most numerous. These tests are focused on checking the smallest parts of an application, such as individual functions or methods. Unit tests are quick to execute and help ensure that each component of the software works as expected in isolation.
Integration Tests: The middle layer of the pyramid consists of integration tests. These are fewer in number compared to unit tests. Integration tests verify that different parts of the system work together as intended. For Jenkins, this makes use of the JenkinsRule to test the integration of the plugin into Jenkins. For Jenkins, integration tests including the UI make use of the JenkinsRule, JenkinsSessionRule or RealJenkinsRule in combination with the createWebClient()-call, which is used to create a call to an HTML-Unit endpoint to then assert something on the result. The different Rule types get more and more realistic towards a real Jenkins controller, but are still for integration tests. This can be a test for a new build step and testing the results in the UI with a test pipeline.
End-to-End Tests (E2E) and UI Tests: At the top of the pyramid, end-to-end tests are the fewest in number but typically the most complex. These tests simulate real user scenarios from start to finish. They interact with the application as a user would, testing the complete flow of the system. The End-To-End Tests are done with a real browser in combination with the acceptance-test-harness, as can be seen in the warnings-ng ui-tests.

These tests are typically not used in Jenkins plugins because they are too complex to create and maintain.

Purpose of the Test Pyramid: The pyramid aims to encourage developers to write a larger number of lower-level tests (unit tests) and fewer high-level tests (E2E tests). This distribution is recommended because lower-level tests tend to be faster, cheaper to automate, and more reliable, whereas higher-level tests are slower, more expensive, and can be flakier.

By adhering to the test pyramid model, developers can ensure they have a balanced test suite that optimizes resources and maximizes the efficiency and effectiveness of the testing process.

Common Patterns

This section covers patterns that you will commonly use in your test cases, plus scenarios that you should consider testing.

Configuration Round-trip Testing

For Freestyle jobs, where users have to configure projects via the web interface, if you’re writing a Builder, Publisher or similar, it’s a good idea to test that your configuration form works properly. The process to follow is:

Start up a Jenkins installation and programmatically configure your plugin.
Open the relevant configuration page in Jenkins via HtmlUnit.
Submit the configuration page without making any changes.
Verify that your plugin is still identically configured.

This can be done easily with the configRoundtrip convenience methods in JenkinsRule. Use archetypes to see examples.

Providing Environment Variables

In Jenkins, you can set environment variables on the System page, which then become available during builds. To recreate the same configuration from a test method, you can do the following:

@Rule public JenkinsRule j = new JenkinsRule();

@Test public void someTest() {
  EnvironmentVariablesNodeProperty prop = new EnvironmentVariablesNodeProperty();
  EnvVars env = prop.getEnvVars();
  env.put("DEPLOY_TARGET", "staging");
  j.jenkins.getGlobalNodeProperties().add(prop);
  // …
}

Providing Test Data

In order to test parts of your plugin, you may want certain files to exist in the build workspace, or that Jenkins is configured in a certain way. This section covers various ways to achieve this using the Jenkins Test Harness.

Customizing the Build Workspace

Within a Pipeline

Pipeline projects don’t have the concept of a single SCM, like Freestyle projects do, but offer a variety of ways to places files into a workspace.

At its most simple, you can use the writeFile step from the Pipeline: Basic Steps plugin. For example:

@Rule public JenkinsRule j = new JenkinsRule();

@Test public void customizeWorkspace() throws Exception {
    // Create a new Pipeline with the given (Scripted Pipeline) definition
    WorkflowJob project = j.createProject(WorkflowJob.class);
    project.setDefinition(new CpsFlowDefinition(""  
        "node {"   (1)
        "  writeFile text: 'hello', file: 'greeting.txt'"  
        "  // …"  
        "}", true));
    // …
}

1	The `node` allocates a workspace on an agent, so that we have somewhere to write files to.

Alternatively, you can use the unzip step from the Pipeline Utility Steps plugin to copy multiple files and folders into the workspace.

First, add the plugin to your POM as a test dependency — you can find the groupId and artifactId values in the plugin dependency tab:

<dependency>
  <groupId>org.jenkins-ci.plugins</groupId>
  <artifactId>pipeline-utility-steps</artifactId>
  <scope>test</scope>
</dependency>

In general, we recommend to use the managed version from the plugin bom, so there is no need to specify the version. But if you need a specific version you can also add a specific version with: <version>2.16.2</version>

Afterwards, you can write a test that begins with extracting the zip file. For example:

import io.jenkins.myplugin;

public class PipelineWorkspaceExampleTest {
  @Rule public JenkinsRule j = new JenkinsRule();

  @Test public void customizeWorkspaceFromZip() throws Exception {
      // Get a reference to the zip file from the `src/test/resources/io/jenkins/myplugin/files-and-folders.zip`
      URL zipFile = getClass().getResource("files-and-folders.zip");

      // Create a new Pipeline with the given (Scripted Pipeline) definition
      WorkflowJob project = j.createProject(WorkflowJob.class);
      project.setDefinition(new CpsFlowDefinition(""  
          "node {"   (1)
          "  unzip '"   zipFile.getPath()   "'"   (1)
          "  // …"  
          "}", true));
      // …
  }
}

1	The path to the zip file is dynamic, so we pass it into the Pipeline definition.

Using `FilePath`

TODO: Expand this section, and explain the below example.

FilePath workspace = j.jenkins.getWorkspaceFor(job);
FilePath report = workspace.child("target").child("lint-results.xml");
report.copyFrom(getClass().getResourceAsStream("lint-results_r20.xml"));

Customizing the `JENKINS_HOME` Directory

TODO: Write this section.

Using `@LocalData`

Runs a test case with a data set local to test method or the test class. This recipe allows your test case to start with the preset JENKINS_HOME data loaded either from your test method or from the test class.

So let’s check a test class with a case where some test.xml-file is needed, and we want to do something with it.

package org.my.plugin;

public class TestJenkinsWithConfigFile {

    @Rule
    public JenkinsRule r = new JenkinsRule();

    @LocalData
    @Test
    public void testConfig() {
        final File rootDir = r.jenkins.getRootDir();
        final String[] list = rootDir.list();
        // check that data is really there
        assertThat(list, hasItemInArray("test.xml"));

        // do something with that data or verify that jenkins did something during startup with that data
    }
}

The @LocalData now takes care that the files under a specific folder are copied in the JENKINS_HOME. For the example above to work, the file test.xml needs to be placed either in:

(1) org/my/plugin/TestJenkinsWithConfigFile/foo/test.xml to be available only for the testConfig() method.
(3) org/my/plugin/TestJenkinsWithConfigFile/test.xml to be available for all methods in the test class.

If you need more than just one file, you can use zipped files, which will be expanded for you in JENKINS_HOME So, if you have several files, e.g.(test.xml, test2.xml) in a test.zip, you can put this in the directory:

(2) org/my/plugin/TestJenkinsWithConfigFile/foo/test.zip to be available only for the testConfig() method.
(4) org/my/plugin/TestJenkinsWithConfigFile/test.zip to be available for all methods in the test class.

Search is performed in the order of the brackets at the line start. The fallback mechanism allows you to write one test class that interacts with different aspects of the same data set, by associating the dataset with a test class, or have a data set local to a specific test method. The choice of zip and directory depends on the nature of the test data, as well as the size of it.

Configuring an SCM

Projects typically check out code from an SCM before running the build steps, and the test harness provides a few dummy SCM implementations which make it easy to "check out" files into the workspace.

Using a Dummy SCM

The simplest of these is the SingleFileSCM which, as its name suggests, provides a single file during checkout. For example:

@Rule public JenkinsRule j = new JenkinsRule();

@Test public void customizeWorkspaceWithFile() throws Exception {
  // Create a Freestyle project with a dummy SCM
  FreeStyleProject project = j.createFreeStyleProject();
  project.setScm(new SingleFileSCM("greeting.txt", "hello"));
  // …
}

Once a build of this project starts, the file greetings.txt with the contents hello will be added to the workspace during the SCM checkout phase.

There are additional variants of the SingleFileSCM constructor which let you create the file contents from a byte array, or by reading a file from the resources folder, or another URL source. For example:

import io.jenkins.myplugin;

// Reads the contents from `src/test/resources/io/jenkins/myplugin/test.json`
project.setScm(new SingleFileSCM("data.json", getClass().getResource("test.json")));

// Reads the contents from `src/test/resources/test.json` — note the slash prefix
project.setScm(new SingleFileSCM("data.json", getClass().getResource("/test.json")));

If you want to provide more than a single file, you can use ExtractResourceSCM, which will extract the contents of a given zip file into the workspace:

import io.jenkins.myplugin;

// Extracts `src/test/resources/io/jenkins/myplugin/files-and-folders.zip` into the workspace
project.setScm(new ExtractResourceSCM(getClass().getResource("files-and-folders.zip")));

Using a Git SCM

You can create a Git repository during a test using @GitSampleRepoRule. This can be found in the git-plugin.

Using Agents

Agents can be used in tests creating a DumbSlave object to be used later in a Pipeline or in a Freestyle job. After that, the agent can be connected to the Controller using the JenkinsRule methods or retrieving the Computer object from the agent.

import org.jenkinsci.plugins.workflow.job.WorkflowJob;
import org.junit.Test;
import org.jvnet.hudson.test.JenkinsRule;
import org.jvnet.hudson.test.recipes.LocalData;

import hudson.model.Computer;
import hudson.model.FreeStyleProject;
import hudson.model.labels.LabelAtom;
import hudson.slaves.DumbSlave;
import hudson.slaves.JNLPLauncher;

public class WithAgentTest {

  @Rule
  public JenkinsRule jenkinsRule = new JenkinsRule();

  @Test
  @LocalData // Suppose you have already created a Freestyle and a WorkflowJob
  public void test_agent() throws Exception {
    // Creating the agent with a specific label
    LabelAtom testingLabel = new LabelAtom("testing");
    DumbSlave agent = jenkinsRule.createSlave(testingLabel);
    /* If you need to change the Launcher
    agent.setLauncher(new JNLPLauncher(true));
    agent.save();
    */
    // Connecting the agent
    jenkinsRule.waitOnline(agent);
    /*
    Alternative to waitOnline would be to use the Computer object, but you must check the agent becomes
    online. Better use waitOnline

    Computer computer = agent.toComputer();
    while (!computer.isOnline()) {
      computer.connect(true);
    }
    */

    FreeStyleProject freeStyleProject = jenkinsRule.jenkins.getItemByFullName("my-freestyle", FreeStyleProject.class);
    freeStyleProject.setAssignedLabel(testingLabel);
    // Testing the job

    WorkflowJob workflowJob = jenkinsRule.jenkins.getItemByFullName("my-workflow", WorkflowJob.class);
    // Testing the job
  }
}

For the pipeline, just make sure the definition uses the label set in the test. For the example above, the pipeline might be:

pipeline {
  agent {
    label 'testing'
  }

  stages {
    stage('Hello') {
      steps {
        sh 'echo Hello world'
      }
    }
  }
}

The agent can be disconnected as follows using the OfflineCause that fits the test.

computer.disconnect(new OfflineCause.IdleOfflineCause());

Enabling security

If you need a security realm for testing you can use a MockAuthorizationStrategy() where you can grant rights as needed for your test. In the following example everyone gets READ on the Jenkins controller and users alice and bob get individual rights. Using LocalData presets can be used if you need to setup a lot for your test.

import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;

import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;

import org.jvnet.hudson.test.JenkinsRule;
import org.jvnet.hudson.test.MockAuthorizationStrategy;

public class MyTest {

  @Rule
  public JenkinsRule j = new JenkinsRule();

  @Test
  public void testAccess() throws Exception {
    // create a dummy security realm
    j.jenkins.setSecurityRealm(j.createDummySecurityRealm());
    // setup a MockAuthorizationStrategy
    MockAuthorizationStrategy authorizationStrategy = new MockAuthorizationStrategy();
    authorizationStrategy.grant(Jenkins.READ).onRoot().toEveryone();
    authorizationStrategy.grant(Item.DISCOVER).everywhere().to("alice");
    authorizationStrategy.grant(Item.READ).everywhere().to("bob");
    j.jenkins.setAuthorizationStrategy(authorizationStrategy);

    // create a freestyle project for test
    j.createFreeStyleProject("myproject");

    // alice can discover project
    JenkinsRule.WebClient alice = j.createWebClient().login("alice");
    FailingHttpStatusCodeException e = assertThrows(FailingHttpStatusCodeException.class, () -> alice.goTo("bypass/myproject"));
    Assert.assertEquals("alice can discover", 403, e.getStatusCode());

    // bob can read project
    JenkinsRule.WebClient bob = j.createWebClient().login("bob");
    bob.goTo("bypass/myproject"); // success
  }

Verifying Logs

You can verify log messages using @LoggerRule. This can also be useful for temporarily enabling certain loggers during interactive testing. For example:

import java.util.logging.Level;
import org.junit.Rule;
import org.junit.Test;
import org.jvnet.hudson.test.LoggerRule;

import static org.hamcrest.Matchers.containsString;
import static org.junit.Assert.assertThat;
import static org.jvnet.hudson.test.LoggerRule.recorded;

public class MyTest {

  public @Rule LoggerRule l = new LoggerRule();

  @Test
  public void testLogs() throws Exception {
    l.capture(3).record("my.logger.name", Level.ALL);
    doThingThatLogs();
    assertThat(l, recorded(Level.INFO, containsString("Thing started successfully")));
  }
}

Verifying compatibility with Configuration as Code (JCasC)

The Configuration as Code plugin is an opinionated way to configure Jenkins based on human-readable declarative configuration files. Writing such a file can be done without being a Jenkins expert by translating a configuration process one is familiar with executing in the web UI into code. If the plugin supports JCasC, it also tests compatibility. More details can be found in the configuration-as-code-plugin documentation.

Increasing Tests Timeout

The default timeout for Jenkins integration tests is 180 seconds.

The tests that require more resources may fail on a slower computer.

To increase the timeout of the tests you can use the following commands: mvn -Djenkins.test.timeout=250 verify

Performance Testing

Starting from Jenkins Test Harness 2.50, the framework provides ways to run microbenchmarks using Java Microbenchmark Harness.

Developer Documentation

Plugin Tutorial

How-To Guides

Reference Topics

Resources

Extensions Index

Javadoc

Taglibs

Tools