Forem: Piyusha

Jenkins Vs. GoCD: Battle Of CI/CD Tools

Piyusha — Thu, 01 Jul 2021 08:28:49 +0000

If you focus on continuous delivery or continuous deployment, you might have come across tools like Jenkins and GoCD. Jenkins is a potent tool that allows you to use plugins available from its vast store. However, the ride to get started with Jenkins is tough, whereas GoCD has an effortless learning curve for beginners and experienced folks. But which one to choose for your project?

We know this question can perplex you while choosing the right CD tool for your DevOps testing project. That is why we have brought you this comprehensive guide on Jenkins vs GoCD to help you choose the most appropriate CD tool for your project’s Selenium automation testing.

Jenkins Vs GoCD: A High-Level Comparison

Jenkins

With over 17.4K GitHub stars and 6.8k forks on GitHub, Jenkins is the most popular open-source automation server written in Java. It supports a plethora of plugins, over 1500, across different software categories. This has made Jenkins the go-to choice for the automation process of CI/CD in the software development process of over 2000 companies, like Facebook, Netflix, and Instacart.

But with the vast amount of plugins, Jenkins also has a huge learning curve to start with, which is why developers and companies are still on the lookout for the best CI/CD tool.

To help you out, here are a few Pros of Jenkinsconsiderable that would help you get an overview on the reasons to assess Jenkins.

Available for cross-platform, i.e., Windows, Linux, and macOS platforms.
Free and open-source. Hence, it is preferable for startups as well as large-scale organisations.
16.3K Star & 6.5k Forked Repo on GitHub.
Highly extensible.
It has a thriving plugin ecosystem (1500+ plugins) and a best-in-class community.
Integrates with popular cloud platforms such as AWS, Google Cloud, Azure, Digital Ocean, and more.
It can be leveraged for performing work in parallel and realising complex CD requirements.
The installer in the .war format is a stand-alone Java application and works out of the box.

Jenkins was developed as continuous integration (CI) tool initially for DevOps testing. So to work with anything other than CI, like CD, a plugin is required. And even though it is enriched with several plugins, a piece of in-depth knowledge is needed to make sure the correct plugin can be installed. Thus, developers and companies are looking out for other CI/CD tools to replace Jenkins, such as GoCD.

GoCD

If you are looking for continuous delivery or continuous deployment in the software cycle, GoCD can be a good Jenkins alternative. It is used to automate the entire phase of build-test-release, from code check-in to deployment. Being an open-source tool with over 6.1K GitHub stars and 884 GitHub forks, GoCD is used in 40+ companies, including Thoughtworks, Hazeorid, and OpenX.

There are certain advantages of GoCD that make it a prominent Jenkins alternative for continuous delivery/deployment, such as:

It allows for flexible and easy setup for the deployment Pipeline.
It is highly customizable to diverse requirements.
It supports both Linux and Windows systems.
We can configure environment variables for each step.
Extensively used for continuous deployment and testing management.

Even though GoCD is easy to use and can be easily set up for the CD pipeline, its deficiency in the field of CI makes it a challenge for the companies and developers to opt GoCD alternatives such as Jenkins.

Below table is a quick guide to what features you can look for and are supported by GoCD and Jenkins.

Jenkins Vs GoCD: Which One To Choose And Why

Both Jenkins and GoCD have their own pros and cons, but which one to choose is still a dilemma for many. So let’s do a head-on comparison of Jenkins vs GoCD, and help you choose the right CI/CD tool without a blink.

Philosophy

With the aim to support the common scenarios of continuous delivery/deployment without use of any plugin, GoCD is one the best CD tools to look out for. The credit goes to the GoCD’s model which maps to all the core concepts of CD pipelines.

On the other hand Jenkins is a general all-purpose CI/CD tool. Whether it is CI or CD, you can definitely depend upon Jenkins due to the availability of 1000+ plugins.

Jenkins vs GoCD: Which one to choose based on Philosophy?

With a wide range of applications and maximum flexibility, Jenkins definitely has the edge over GoCD. But if you are looking for a single-purposed, focused tool, you can use GoCD.

Continuous Delivery

As the name suggests, GoCD is single handedly devoted towards CD or continuous delivery. Their aim is to make GoCD as the most prominent tool for continuous delivery.

Since Jenkins was developed with the initial goal of serving the purpose of CI, CD is still a challenge with Jenkins. The only practical way of using CD over Jenkins is with the help of some plugin. With the introduction of Jenkins 2.0, Jenkins has made a lot of improvement, but still, it is not clear if CD fits in this.

Jenkins vs GoCD: Which one to choose based on Continuous Delivery?

With the single point focus over CD, and with wide CD community support, GoCD is definitely the right choice if you are looking for only a CD tool. But since CI and CD are usually intermingled in the software development cycle, and since there is a huge demand for an all-purpose CI/CD tool, Jenkins will serve you well.

Getting Started

GoCD has a getting started tutorial which is pretty easy to follow. Due to this GoCD has a very smooth learning curve that can easily pick up beginners as well as experienced.

Jenkins has improved the getting started experience by a lot from the release of Jenkins 2.0, but still installing the right mix of plugins can be a challenge based on the use case. Also, it’s not sure with Jenkins that a single plugin will solve all the purpose. You might need to switch between plugins depending on the scenario.

Jenkins vs GoCD: Which one to choose based on Getting Started?

Both Jenkins and GoCD can be challenging if you want to dive right in. GoCD’s model can be difficult to work with if switching from a pure CI tool. But GoCD has a better getting started curve than Jenkins. Jenkins has hurdles around configuring required tools, unintuitive SCM configuration, and installing additional plugins. Although once you get hold of the plugins in Jenkins, it’s pretty easy to work with and can accomplish more use cases than GoCD.

Plugins

GoCD aims to make implementing CD easy, and to do so it has few extension points like SCM, task, notifications, authentication, authorization, configuration, and elastic agents. But they are not focused on providing the pool of plugins as with the case with Jenkins, which makes GoCD less flexible.

Jenkins, on the other hand, has a wide pool of plugins available to make it infinitely configurable, flexible and extendable. It would be not incorrect to say that the high popularity of Jenkins is gained from its availability of plugins. But, plugins are the same reason which makes Jenkins hard to use as well. The getting started and support for each plugin is different and this can be a challenge to the users.

Jenkins vs GoCD: Which one to choose based on Plugins?

If you are looking for just a CD tool, and nothing else, with no scope of flexibility whatsoever, GoCD is the right choice for you, as it clearly has no intention of supporting a pool of plugins. But if you are looking for a much more flexible CI/CD tool, which is configurable, you should opt for Jenkins, if it matches your needs and uses case.

Jenkins Vs GoCD: Side-By-Side Comparison

To summarize, the below table charts the primary comparisons between GoCD & Jenkins based on various parameters.

Jenkins vs GoCD: Which CI/CD Tool Should You Prefer?

Both Jenkins and GoCD have their own pros and cons but your final choice between the two CI/CD tools depends upon the project requirements and specification. With their advantages and disadvantages, both the tools aim to accomplish the same goal: automating processes for CI/CD (Continuous Integration & Continuous Delivery/Deployment).

Jenkins is for CI and CD whereas GoCD is for CD only. So apart from the comparison of features among the two, you should also focus on the in-house requirements to choose the most appropriate tool for DevOps testing.

Fun Fact: From a DevOps testing perspective, it doesn’t matter whether you choose Jenkins or GoCD, as LambdaTest offers integrations with Jenkins, GoCD, as well as other CI/CD tools via its cloud based Selenium Grid. The best part is, you can test and validate your new builds in the pipeline on over 2000+ real browsers, browser versions, and operating systems and expedite your Selenium automation testing once you have chosen the best DevOps testing tool.

You can also check out the rest of the LambdaTest CI/CD integrations.

Happy Testing!

How To Create & Run A Job In Jenkins Using Jenkins Freestyle Project?

Piyusha — Thu, 20 May 2021 10:44:23 +0000

As per the official Jenkins wiki information, a Jenkins freestyle project is a typical build job or task. This may be as simple as building or packaging an application, running tests, building or sending a report, or even merely running few commands. Collating data for tests can also be done by Jenkins.

For instance, a real-world scenario could involve Jenkins allowing you to submit reports to log management at any specified stage concerning management, which may include details about artifacts or shipping application logs. To get a better understanding, we will dive deeper into how to create a job in Jenkins & eventually a Jenkins freestyle project.

Let’s find out more about Jenkins Build Job before we begin creating a Freestyle project.

What is a Build Job?

The Build jobs are at the heart of any Jenkins build process. To put it plainly, imagine a Jenkins build job as a particular task or step in your entire build process. The task or step could involve merely compiling the source code and running your unit tests over. Or you can configure a build job to do other co-related tasks, such as

Measuring code coverage or code quality metrics.
Running your integration tests.
Generating technical documentation.
And even deploying the application to a web server.

Generally, a real project requires many separate but related build jobs in a CI or DevOps cycle.

Let us first understand what the Jenkins Freestyle project is and how we can create it.

What is Jenkins Freestyle Project?

Jenkins supports different types of build jobs. The two most commonly used Jenkins build jobs are-

Freestyle builds
Maven 2/3 builds

The best part about the Jenkins freestyle project is that it allows you to configure just about any build job, thus making them highly flexible and configurable. The Maven 2/3 builds, on the other hand, understand the Maven project structure and let you set up Maven build jobs quickly with added features.

In a freestyle build job, you can break down your build job into several smaller build steps, making it easier to organize builds in separate stages. E.g., in one step, a build might run a suite of functional tests and then tag the build in a second step if all functional tests are successful. It may be appropriate to invoke an Ant task or a Maven target or run a shell script in a technical terms build phase.

Various other Jenkins plugins also let you use additional types of build steps. These involve Grails, Gant, Rake, Gradle, Ruby, MSBuild, etc. But what highlights a Jenkins Freestyle project is that it lets you create general-purpose build jobs with maximum flexibility.

So the bottom line is- the freestyle projects allow you to configure just about any build job.

A Jenkins project involves steps and post-build actions and is a repeatable build job. There are limitations in the types of actions you can perform in a build step or post-build action. Within a Jenkins freestyle project, there are several standard plugins available to help you overcome this problem. These plugins allow you to configure build triggers and offer project-based security for the Jenkins project.

When to Use a Jenkins Freestyle Project?

An access to the shell or a batch environment propagates a high level of security access for the Jenkins freestyle project. In freestyle project environments, it’s convenient to create a script that instructs a Jenkins freestyle job to FTP a file from one server to another, compile a directory of Java code, or even run a SonarQube test. A Jenkins freestyle job can be as powerful and complex as any build job built with a Jenkins pipeline or a Groovy DSL. But the only drawback to freestyle projects is that the user must know how to script all of these actions, and developers need to learn how to manage these scripts.

Plugins such as Git, Maven, and SonarQube are the preferred way to access resources. And technologically, every developer can write a script to access those resources within a Jenkins freestyle project. However, it is recommended that every developer in the development team should follow the guided security best practices and inhibit from adapting a scripted approach.

Creating a Freestyle Build Job

To set up freestyle projects using Jenkins, we need to ensure that we have Jenkins installed and up and running. Also, it is mandatory to be authenticated as the ‘Admin.’ You can refer to our hands-on Jenkins tutorial to learn more.

Once you have Jenkins up & ready, let us create a Jenkins freestyle job.

Step 1.

Get logged on to your Jenkins dashboard through the Jenkins installation path.
Unless you have defined a private host, it will be hosted on the localhost at http://localhost:8080.
In case your Jenkins is installed in another path, make sure to use the appropriate URL to access the dashboard.

Step 2. Go to the “ New Item ” option at the top left-hand side of your main dashboard.

Step 3. On the next screen,

Here enter the name of the item you want to create. Let us use the “Hello world.”
Select ‘ Freestyle project ’ as the option for this new Item.
Click OK.

When we hit Ok, Jenkins automatically takes us to the project configuration view, where we need to configure our project details. Additionally, we can provide intricate details through tab options.

The first tab is the general details of the project. So the next step is to enter project details.

Step 4. Enter the project details in the General tab, including the name and description of the project that needs to be tested.

Next, let’s understand what the Source Code Management tab does. It checks out code from version control hosts, which implies that if your code is hosted on GitHub or any other repositories, you have to add the repository details. Jenkins will clone the repository.

The blue question mark icons present at the right-hand bottom side serve us with quick tips, and they can be very resourceful and provide more clarification at times.

Step 5. Under Source Code Management (SCM) tab,

Select Git as a repository source and enter your Git Repository URL.
In case you have your repository created locally, it is permissible to use a local repository.
Suppose the GitHub repository you are using is private. In that case, Jenkins will validate the login credentials with GitHub, and upon successful validation, it will then pull the source code from your GitHub repository.

After you have provided all the project & repo details, the next part is to build the code. You can manage the settings under the Build section to build the code at your desired time. You can even schedule the build to run periodically if needed.

Step 6. Go to the Build section and click on the Add build step.

Under the build section,

Click on the “ Add build step.”

Then, “ Execute Windows batch command ” and add the commands you want to execute during the build process.

E.g., Java compile batch commands.

Step 7. When you have entered all the data,

Click Apply.
Save the project.

Step 8. On the left-hand side panel, click the Build Now button to build the source code.

Step 9. We can see the status of the build under Build History.

Step 10. We can verify the history of the executed build under the Build History by clicking the build number. By clicking on the Build Number –> Console Output , it should show you a success or failure message depending on how the job ran.

To sum up, we ran a ‘HelloWorld’ program on GitHub. Jenkin pulled up the source code from the remote repository and continuously built it at a frequency you define.

Jenkins Freestyle Project for Docker Container Images

The below demonstration is about creating a Jenkins Freestyle project that first builds a Docker image and then scans it for any vulnerabilities and compliance issues as a test over.

Go to the Jenkins top page.
Create a new project.
- Click on ‘ New Item.’
- Enter a name for your project in ‘ Enter an item name.’
- Select ‘ Freestyle project.’
- Then click, OK.

Next, add a build step.
- In the Build section → Add build step → select Execute shell → Command.
- Enter the below command in the Command text box.

echo "Creating Dockerfile..."
echo "FROM imiell/bad-dockerfile:latest" > Dockerfile
docker build --no-cache -t test/test-image:0.1 .

Include a build step that will scan for vulnerabilities in the container image(s).
- Select Scan Prisma Cloud Images in the Add Create Phase drop-down list.
- Select the image to be scanned in the Image field by specifying the repository and tag.
- In case the image you want to scan is generated beyond this build, or if you want to scan for an image for each build, even if a new image is not generated by the build, then press Advanced and select Ignore Image Development Time.
To directly publish the scan results in Jenkins, add a post-build action.

This post-build step depends on a file created by the previous scan-build step, which contains the scan results. In the Jenkins build tool, this step explicitly makes the results available for review. Note that the results have already been published in the Console during the last scan phase, and they are ready for analysis there.

Select Publish Prisma Cloud Analysis Results from the drop-down menu under Add Post-Build Action.
Accept the default in the Scan Result Folders.
The publishing stage does not remove the result files from the scan. In the workspace, they remain.
1. Click Save , then Build Now.
2. Examine the results after the build completes. In the following places, scan reports are available:
Prisma Cloud Console: Log into Console, go to Monitor –> Vulnerabilities –> Images –> CI.
Jenkins: Scroll down into the build job –> then click Image Vulnerabilities to see a detailed report.

Jenkins Freestyle Project for Serverless Functions

It is similar to the procedure for container images to set up Jenkins to scan serverless functions, except that you can use the Scan Prisma Cloud Functions construct step.

Here:

Function Path — Path to the function’s ZIP archive to scan.
Function Name — (This is Optional) Identifier string to align policy rules in the Console with scanned functions. When you build policy rules in the Console, the function name will target specific rules to specific functions. If this field is left undefined, the plugin fits the first rule function, where the name of the function is a wildcard.
AWS CloudFormation template file — (This is Optional) Path to a template file in either JSON or YAML format with CloudFormation. Prisma Cloud checks the source code of the function for the AWS service APIs that are used, compares the APIs used with the permissions of the function, and reports when the functions have permissions for APIs they do not need.

Converting a Project from Jenkins Freestyle project to a Declarative Pipeline

In Jenkins, managing freestyle jobs is cumbersome. Declarative Pipelines offer a more modern and suggested solution. It is time-consuming and error-prone, however, to try to convert Freestyle jobs to Declarative Pipelines manually. You can use the Migration Assistant Declarative Pipeline plugin to streamline this operation. During the conversion, the Declarative Pipeline Migration Assistant uses a best-effort approach: supported configurations are converted automatically in Freestyle projects, and placeholder stages are generated for plugins that are not yet supported.

The Declarative Pipeline Migration Assistant’s advantages include:

Generating a Jenkinsfile from a project for Freestyle.
A Jenkinsfile customization.
Creating a Jenkins pipeline project and inserting a Jenkins file.

Generating a Jenkinsfile from a project for Freestyle

A centralized configuration file, called a Jenkinsfile, is the basis for pipeline or multi-branch pipeline projects. In the text editor or GUI, a Jenkinsfile can be generated. The file is maintained either with the project code or, for example, a software configuration management (SCM) tool like Git in a separate repository. Using an SCM to store the file gives the configuration file a centralized location, enables code review and provides an audit trail for the pipeline. To build an initial Jenkinsfile, the Declarative Pipeline Migration Assistant uses information from a Freestyle project.

The Declarative Pipeline Migration Assistant is only accessible from the Jenkins UI and not from the Jenkins CLI.

See Defining Pipeline for more information about pipelines.

Prerequisites: Two plugins, namely

The Pipeline plugins
The Declarative Pipeline Migration Assistant plugin

Steps to generate a Jenkinsfile from a Freestyle project-

Go to the –> Freestyle project.
From the Freestyle project tab, choose To Declarative from the left navigation menu.

A Jenkinsfile will be supplied for review once the conversion is complete.

Currently, the Declarative Pipeline Migration Assistant plugin supports a small number of plugins.

It was unable to convert: if the conversion lists a warning for plugins,

The plugin is not compatible with pipelines. You should review the documentation of the plugin to see if it is Pipeline-compatible. Use the shell step as a replacement option if the plugin is not compatible with the pipeline.
The plugin is compatible with Pipeline and appears in the Snippet Generator. (See table below for a list of compatible plugins.) To construct the correct syntax, use the Snippet Generator.
The plugin is compatible with the pipeline and will not appear in the Snippet Generator. For more detail on implementing the plugin, refer to the Pipeline documentation; see Customizing a Jenkinsfile.
You can write plugins for the Declarative Pipeline Migration Assistant for reusability; see Extending the plugin for Declarative Pipeline Migration Assistant.

The table below exhibits a list of the supported plugins.


Type	Step
scm	git
step	shell step
step	batch step
step	Maven build step
build wrapper	Config File Provider plugin
build wrapper	secret (convert to credentials binding)
build wrapper	Build a timeout plugin
job property	Lockable Resources plugin
job property	build parameters
job property	build discarder configuration
build trigger	upstream projects trigger
build trigger	SCM pooling
build trigger	timer trigger
build environment	provide configuration files
build environment	use secret text(s) or file(s)
post-build action	JUnit plugin
post-build action	HTML Publisher plugin
post-build action	trigger another project
post-build action	mail notification
post-build action	Do not fail the build if archiving returns nothing
post-build action	archive artifacts only if the build is successful
post-build action	fingerprint all archived projects

Jenkinsfile Customization

After creating a Jenkinsfile from a Freestyle project, you have few options.

You can either copy-paste the Jenkinsfile into a text file.
Or you can download the provided text file and open it in the text editor.
Then, to perform the same tasks performed by the Freestyle project, review the Jenkinsfile and edit as required for the new Pipeline project.

Creating a Pipeline Project in Jenkins & Adding a Jenkinsfile

The next move is to add the Jenkinsfile as the configuration file to a Pipeline or Multibranch Pipeline project after creating a Jenkinsfile from a Freestyle project and editing the Jenkinsfile. Creating the Jenkins pipeline project first. Then, by copying and pasting the Jenkinsfile into the Pipeline editor or storing it in an SCM like GitHub and linking the repository to the Pipeline project, add the Jenkinsfile to your project.

Steps to create a Pipeline project and adding the Jenkinsfile:

Step 1: In Jenkins, create a pipeline project and copy and paste the Jenkinsfile text into the editor of the pipeline.

Step 2: Store the Jenkinsfile in a repository for SCM such as GitHub and:

Connect the repository to the Jenkins Pipeline project

Connect the repository to a Multibranch Pipeline project

Extending the Declarative Pipeline Migration Assistant Plugin

A small number of plugins are currently supported by the Declarative Pipeline Migration Assistant plugin. For a list of plugins supported, see the table above.

Beyond this list, if you want to add support for a particular plugin that is currently not supported, it involves attaching the API dependency to the converter and creating the extension.

Adding the API Dependence of the Converter

The following code snippet is illustrates how to add a dependency on the converter API:

<dependency>
      <groupId>org.jenkins-ci.plugins.to-declarative</groupId>
      <artifactId>declarative-pipeline-migration-assistant-api</artifactId>
      <version></version>
    </dependency>

Creating the Extension

The following code snippet is an illustration of how to create the extension:

@Extension
public class ShellConverter extends SingleTypedConverter<Shell>
    @Override
    public boolean convert( ConverterRequest request, ConverterResult result, Object target )
    {
      Shell shell = (Shell) target;
      /* Do some work to generate a new stage . . . */
      ModelASTStage stage = new ModelASTStage( this );
      // use a utility method to add the stage to the pipeline model
      ModelASTUtils.addStage( request.getModelASTPipelineDef(), stage);

      // true for success, false for failure
      return true;
    }

Example Conversions

The abstract class SingleTypedConverter<T> defines the extension of the conversion.

BuilderConverter: convert Builder
BuildWrapperConverter: convert BuildWrapper
JobPropertyConverter: convert JobProperty
PublisherConverter: convert Publisher
ScmConverter: convert SCM
TriggerConverter: convert BuildTriggers

Example Build Step conversion

The following code snippet is an illustration of how to convert a Shell script Freestyle step using the API:

@Extension
public class ShellConverter extends SingleTypedConverter<Shell>
{
  @Override
  public boolean convert (ConerterRequest request, ConverterRequest result, Object target )
    {
        Shell shell = (Shell) target;
        ModelASTStage stage = new ModelASTStage( this );
        int stageNumber = request.getAndIncrement( SHELL_NUMBER_KEY );  
        stage.setName( "Shell script " + stageNumber );
        ModelASTBranch branch = new ModelASTBranch( this );  
        stage.setBranches( Arrays.asList( branch ) );  
        ModelASTStep step = new ModelASTStep( this );  
        step.setName( "sh" );  
        ModelASTSingleArgument singleArgument = new ModelASTSingleArgument( this );
        singleArgument.setValue( ModelASTValue.fromConstant( shell.getCommand(), this ) ); 
        step.setArgs( singleArgument );
        ModelASTUtils.wrapBranch(result, step, branch); 
        ModelASTUtils.addStage(result.getModelASTPipelineDef(), stage); 
        return true;
    }
}

Example Build Wrapper Conversion

The following example converts the Config File Freestyle wrapper build using the API. This conversion uses a helper method to add a wrapper around all future build step conversions.

// This was to not have the config-file-provider plugin as a required dependency
// But you can use (as you use your plugin)
@OptionalExtension(requirePlugins = { "config-file-provider" })
public class ConfigFileBuildWrapperConverter extends SingleTypedConverter<ConfigFileBuildWrapper>
{
    private Logger LOGGER = LoggerFactory.getLogger( ConfigFileBuildWrapperConverter.class );

    @Override
    public boolean convert(ConverterRequest request, ConverterResult result, Object target)
    {
        ConfigFileBuildWrapper configFileBuildWrapper = (ConfigFileBuildWrapper) target;
        if(configFileBuildWrapper.getManagedFiles() == null || configFileBuildWrapper.getManagedFiles().isEmpty() )
        {
            return true;
        }

        result.addWrappingTreeStep( () -> build( configFileBuildWrapper) ); 
        return true;
    }

    private ModelASTTreeStep build(ConfigFileBuildWrapper configFileBuildWrapper) {
        ModelASTTreeStep configFileProvider = new ModelASTTreeStep( this );

        configFileProvider.setName( "configFileProvider" );
        ModelASTSingleArgument singleArgument = new ModelASTSingleArgument( null );
        configFileProvider.setArgs( singleArgument );

        ManagedFile managedFile = configFileBuildWrapper.getManagedFiles().get( 0 ); 
        StringBuilder gstring = new StringBuilder( "[configFile(fileId:'" );
        gstring.append( managedFile.getFileId());
        gstring.append( "', targetLocation: '" );
        gstring.append( managedFile.getTargetLocation() );
        gstring.append( "')]" );
        singleArgument.setValue( ModelASTValue.fromGString( gstring.toString(), this ) );
        return configFileProvider;
    }
}

Example Publisher Conversion

The following example converts the ArtifactArchiver Freestyle post-build step using the API. This conversion modifies the model to add some build conditions.

@Extension
public class ArtifactArchiverConverter extends SingleTypedConverter<ArtifactArchiver> {
    @Override
    public boolean convert(ConverterRequest request, ConverterResult result, Object target) {
        ArtifactArchiver artifactArchiver = (ArtifactArchiver) target;
        ModelASTBuildCondition buildCondition;
        if(artifactArchiver.isOnlyIfSuccessful()) { 
            buildCondition = ModelASTUtils.buildOrFindBuildCondition( result.getModelASTPipelineDef(), "success" );
        } else {
            buildCondition = ModelASTUtils.buildOrFindBuildCondition( result.getModelASTPipelineDef(), "always" );
        }
        ModelASTStep archiveArtifacts = ModelASTUtils.buildGenericStep(artifactArchiver, this);
        ModelASTUtils.addStep(buildCondition, archiveArtifacts); 
        return true;
    }
}

Depending on which condition the artifact needs to be executed, we add the step to the build condition.

Example SCM Conversion

The following example converts the Git SCM Freestyle stage using the API. This conversion adds a stage to the Pipeline model.

@OptionalExtension(requirePlugins = { "git"})
public class GitScmConverter extends SingleTypedConverter<GitSCM>
{
  @Override
  public boolean convert(ConverterRequest request, ConverterResult result, Object target)
    {
        List<UserRemoteConfig> repoList = ( (GitSCM) target ).getUserRemoteConfigs();
        if(repoList.isEmpty()){
            return true;
        }
        ModelASTStage stage = new ModelASTStage( this ); 
        stage.setName( "Checkout Scm" );
        List<ModelASTStep> steps = new ArrayList<>(); 
        // a step will be created per remote repository
        for( UserRemoteConfig userRemoteConfig : repoList) 
        {
            ModelASTStep git = new ModelASTStep( null ); 
            git.setName( "git" );

            Map<ModelASTKey, ModelASTValue> args = new HashMap<>();
            {
                ModelASTKey url = new ModelASTKey( this ); 
                url.setKey( "url" );
                ModelASTValue urlValue = ModelASTValue.fromConstant( userRemoteConfig.getUrl(), this );
                args.put( url, urlValue );
            } 

            ...

            ModelASTNamedArgumentList stepArgs = new ModelASTNamedArgumentList( null);  
            stepArgs.setArguments( args );
            git.setArgs( stepArgs );
            steps.add( git );
        }

        ModelASTBranch branch = new ModelASTBranch( this ); 
        branch.setSteps(steps);
        stage.setBranches( Arrays.asList( branch ) );
        ModelASTUtils.addStage(result.getModelASTPipelineDef(), stage ); 
        return true;
    }
}

Create a new stage.
Steps will be generated as: – git url: “”, branch: ”,changelog: ”, credentialsId: ”, pool: ”
A step is created based on each remote repository.
Create the Git step.
Add parameters – URL.
In the original code, add more parameters.
Configure args of the step.
Create a branch for the stage.
Use a utility method to add the stage to the Pipeline model.

Example Build Trigger Conversion

Using the API, the following example transforms the cron trigger. The pipeline mode is changed by this conversion to add a trigger property through a utility process.

@Extension
public class TimerTriggerConverter extends SingleTypedConverter<TimerTrigger>
{
    @Override
    public boolean convert(ConverterRequest request, ConverterResult result, Object target)
    {
        TimerTrigger timerTrigger = (TimerTrigger) target;

        String cronValue = timerTrigger.getSpec();
        ModelASTTrigger modelASTTrigger = new ModelASTTrigger( this ); 
        modelASTTrigger.setName( "cron" );
        modelASTTrigger.setArgs( Arrays.asList(ModelASTValue.fromConstant( cronValue, this )) );

        ModelASTUtils.addTrigger( result.getModelASTPipelineDef(), modelASTTrigger );  
        return true;
    }
}

That’s how you can create your very own pipeline projects!

Wrapping Up!

In this blog, we saw the central feature of Jenkins learning how to create a job in Jenkins, Jenkins build job and creating pipeline projects. Through the Jenkins freestyle project, you can build your project, combine any SCM with any build system. We also saw a few extensions of Freestyle projects like the docker container images. The conversion of freestyle projects into Declarative Pipelines offers a more modern and suggested solution.

Let us know what you feel about this blog in the comments below. Try reproducing the steps for Jenkins to create and build a job to learn on the go.

Happy testing!

How To Integrate Cucumber With Jenkins?

Piyusha — Mon, 05 Apr 2021 07:38:40 +0000

Cucumber is an open-source BDD (Behavior-Driven Development) testing tool for automation testing of web applications. It remains to be one of the best frameworks for functional and acceptance testing. This is because of its widespread adoption in the open-source community, enhanced natural language support, and the ability to support new programming languages quickly. Almost all organizations that use Selenium prefer to integrate Selenium with Cucumber since Cucumber makes it easy to read and understand the application flow. To schedule test case executions remotely, we can also go for Cucumber with Jenkins integration and utilize the immense benefits of Jenkins.

Getting Started with Cucumber Jenkins Setup

Cucumber is well-known for agile development of web applications and reporting the results. With an easy-to-understand language, powerful plugins, and a straightforward integration with Selenium, starting Cucumber with Jenkins integration has never been easier.

Before we begin, let’s understand why you need Cucumber with Jenkins integration in the first place. Here’s why- Cucumber acts as a bridge between:

Manual Tester and Developers.
Software Engineer and Business Analyst.
Manual Tester and Automation Tester.

And as you already know, Jenkins is an open-source, accessible, easy to use CI/CD tool. With Jenkins, you can schedule a run on scheduled time or trigger builds after an event, and it also comes with a plethora of plugins. You can refer to our comprehensive Jenkins tutorial to learn more about this tool.

Now, let us see how to integrate Cucumber with Jenkins.

Prerequisites for Cucumber with Jenkins Integration

We need the installation of the below plugins to start Cucumber with Jenkins integration. Let us see an overview and utilities of the below plugins. This section will see how to make use of some Jenkins plugins that work together with Cucumber formatting outputs.

Cucumber Test Results plugin
Cucumber Reports
Cucumber perf plugin

A. Cucumber Test Results plugin

This plugin takes Cucumber JSON output as input and publishes the results in polychrome. The results will appear in GREEN when they have passed and RED when they fail.

Here’s how to install the Cucumber Test Results plugin. You can also refer to our blog on generating TestNG reports in Jenkins for more information.

Login to your Jenkins server as an administrator.
Navigate to “ Manage Jenkins.”
Go ahead and click “ Manage Plugins.”

Go to the “ Available ” tab under the manage plugins page.
Search for “ cucumber ” in the filter at the right top corner.
Go ahead and check the box against the Cucumber reports.
Choose “ Install without restart ” to start the installation.

Click “ Back to dashboard ” in the left menu
Navigate to the job already created. ‘Cucumber’ in our case.
Click “ Configure ” against the job.

Scroll down to Post-Build Actions. Under “ Add post-build action ,” select “ Cucumber reports.”

Click on Advanced. Go to Post-build Actions. Add below report file in the text box as shown. You can Ignore the message at the bottom. It appears since we don’t have any files yet, once we run the build, they will be created.

The next step is to generate Cucumber JSON reports.
Open the cucumber.yml file in your page-object_framework and add the below-highlighted string.
This is to make sure we have reports.json output in addition to the HTML reports. The cucumber plugin will parse these on Jenkins.

Execution

To execute, click the “ Build Now ” button from the left menu for the job. The build should be successful. Instead of watching console output, select “ Cucumber Test Result ” at the end of the build.

Results

Next, go to Feature –> Scenario , where you can see all the steps’ status. As shown below, they are all GREEN.

B. Cucumber Reports

Cucumber Reports plugin can read the JSON output and produce pie charts for feature or scenario results & bar graphs for tags. This is possible if we had options “-f json -out cucumber.json,” where you output the results to JSON format, i.e., in cucumber.yml. This plugin has advanced options to help us make vital decisions on the test results.

Add a post-build step and select the “ Publish cucumber results as a report ” option, as shown below.

Referring to the job, the below menu option will be visible to us as below.

Next, run the job and click the reports. The reports will be in the form of SWF charts, as shown below.

Results

We get results as below.

C. Cucumber perf plugin

This plugin helps us demonstrate the history of how Cucumber results were performing as a trend. This is helpful when you keep running the suite multiple times.

There are additional features to it, like “ worst-performing features ” over time. This mainly helps us focus on those bottlenecks and allows us to debug them further.

Integrating Cucumber with Jenkins

Now, you are all set with the prerequisites for Cucumber with Jenkins integration. In this section, we will see how to integrate Cucumber with Jenkins using these plugins. In case you haven’t installed them yet, let us start from the very beginning-

Access Jenkins URL. The default is port 8080.
Go to “ Manage Jenkins ” –> “ Manage Plugins.”

Install the following plugins
- Cucumber Test Results plugin
- Cucumber Reports
- Cucumber perf plugin

Make sure while installing these plugins, all the dependent plugins are also successfully installed. This is to ensure the plugins work as expected without glitches.

The next step is to navigate to “ Manage ” –> “ Global Tool Configuration.” Under “ JDK ,” set the path for JDK.

Under “ Maven ,” set the path for Maven.

Create a “ New Item ” as a Maven Project as shown below. Enter any arbitrary name.

Once the job is created, click on the “ Configure ” link on the left-hand side panel.

Scroll down the page. Set the full path of pom.xml under “ Root POM ” and “Goal and options” as ‘test’ and proceed to save the configuration.

Once all the above steps are completed, click on the “ Build Now ” button.

The build will be executed, and the corresponding testing.xml file, which is the pom.xml, will get executed.

[INFO] T E S T S
[INFO] ——————————————————-
[INFO] Running TestSuite
New scenario begins
Starting ChromeDriver 2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f) on port 27348
Only local connections are allowed.
(org.apache.http.client.protocol.RequestAddCookies).
Feb 02, 2021 09:49:41 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: OSS
https://avatars3.githubusercontent.com/u/31874017?s=400&u=c41bfae0fa6b9325fb4f209885b51bd02c7d897d&v=4
https://avatars3.githubusercontent.com/u/31874017?s=400&u=c41bfae0fa6b9325fb4f209885b51bd02c7d897d&v=4
Scenario ends
New scenario begins
Starting ChromeDriver 2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f) on port 24866
Only local connections are allowed.
Feb 02, 2021 09:50:47 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: OSS
Scenario ends
2 Scenarios (2 passed)
14 Steps (14 passed)
2m2.677s

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 128.513 s – in TestSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[JENKINS] Recording test results
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 02:04 min
[INFO] Finished at: 2021-02-02T09:51:45+05:30
[INFO] Final Memory: 12M/28M
[INFO] ————————————————————————
Waiting for Jenkins to finish collecting data
[JENKINS] Archiving D:\cucumberFinal\multiple\pom.xml to com/cucumber.example/0.0.1-SNAPSHOT/cucumber.example-0.0.1-SNAPSHOT.pom
channel stopped
Finished: SUCCESS

Hurray, you have successfully completed Cucumber with Jenkins integration! Before we proceed further, this might be the right time to acquaint yourself with the best Cucumber practices.

Running Cucumber Script with Jenkins

Automation scripts built with the Cucumber framework benefit from being easy to understand, easy to communicate, and good maintenance, so it is popular among test automation teams. An essential aspect of integrating DevOps is incorporating automated testing into a continuous integration (CI) process.

Now that you have completed the Cucumber Jenkins setup, in this section, we will see how to configure Jenkins to run an automated Cucumber test script and show the Jenkins UI report. This can be done using the following methods-

Cucumber’s Command Line Mode
CukeTest’s Command Line Mode
Jenkins Continuous Integration Configuration
Jenkins Test Report Plugin Configuration

Cucumber’s Command Line Mode

To illustrate the Cucumber Command Line, we use Cucumber.js. Many who already know this tool know that it means running the “Cucumber-js” command directly in the directory of the automation script. Or if you don’t have Cucumber installed globally, run “node modules/.bin/cucumber-js” (or, if you’re on Windows, the .cmd extension), and the Cucumber installed locally will be used. After execution, the output of the text on the command line is written. You can refer to our Cucumber.js tutorial for more information.

The command line is identical if you are running it as a job under Jenkins. Just add the “format” parameter, indicating the output file in the “JSON” format, to make Jenkins render a visual report. The following is the command line:

node_modules\.bin\cucumber-js --format json:report.json

A report is output to report.json, and then this file is used by the Jenkins job to view the report.

CukeTest’s Command Line Mode

CukeTest is Cucumber.js’s development tool. These editions support command line mode and are available on Windows Desktop, Windows Store, MacOS, etc. It is available for free at CukeTest’s official website. CukeTest is used to build scripts for Cucumber.js and can also be used to execute scripts for Cucumber.js from the command line.

The benefit of using CukeTest is that with each project, you don’t have to install Cucumber.js. CukeTest already has Cucumber.js built-in, which saves your storage space and time. Another advantage is that it can run Windows automation with the library built-in, capture video, etc.

The Windows Store version of CukeTest has a different command line process, so the command line varies slightly from the desktop version. Luckily, you don’t need to recall all the specifics of the command line.

In CukeTest, you can open a run profile when your project is opened to get the command line by clicking the drop-down arrow next to the “ Run Project ” button and then clicking “ Edit Profiles.”

In the “ Run Profile s” dialogue, the corresponding command line is automatically created for this profile in the command line:

Click the “ Copy to clipboard ” option to copy the commands to the clipboard.
Then click the “ Open console window ” option, and paste the command line to this newly opened window.

Below are the command lines for various platforms-

Command line for Windows Store version

chcp 65001 && start cuketest://?profile=Profile1

CukeTest’s command line output is encoded in UFT8. So using the command “chcp 65001” helps change the command line encoding from UFT8 to UTF8.

CukeTest desktop version

chcp 65001 && cuke --run --format html

CukeTest’s command line output is encoded in UFT8. So using the command “chcp 65001” helps change the command line encoding from UFT8 to UTF8.

Jenkins Continuous Integration Configuration

Once we are aware of the steps to run the project in the command line, we can configure Jenkins to run it. Below are the steps to configure the Jenkins job.

For the demo, the source code is kept in https://github.com/CnodejsTest. This project is used to automate test cnodejs.org community site. Alternatively, we can use our Cucumber project to configure and run the job.

Create a new job in Jenkins. Enter any arbitrary name.

In the item bar, select “ Windows Batch Command.” Here we have to copy & paste the corresponding command line from CukeTest to the Jenkins batch command box. In case you have a CukeTest desktop version installed natively, below is your command line.

npm install
chcp 65001 && cuke --run --format html

The ‘ npm install ’ command installs npm dependency packages that are consumed by the automation script.

Perform the below job configuration UI with the commands as entered below.

Save this project configuration by clicking ‘ Save.’ Now click the “ Build Now ” button, and the project will then start executing.

You can observe the execution progress upon build now of the job in the build history.

We can also find the running job log under the Build History. At the exact moment, the report file is also generated in the Jenkins Workspace directory.

The next step is to configure the Jenkins job to show the Jenkins interface test report. This can be done quickly using the Plugin we just saw in the previous section to format and display the test report.

Jenkins Test Report Plugin Configuration

In Jenkins, we can use the Cucumber reports plugin to show the report. We have already seen the introduction to this plugin in the Cucumber Jenkins setup above. Additionally, you can also read the information about this Plugin available at the official Jenkins website:

The following are the steps to configure Jenkins to generate reports.

Install the Jenkins reports plugin Open Jenkins –> Manage Plugin –> Filter “ Cucumber reports ” in Optional Plugin.

We need to restart Jenkins after the installation of this plugin.

Reconfigure the Jenkins task

We need JSON log data files to generate reports because the Cucumber reports Plugin generates reports by parsing the JSON report file created by Cucumber or CukeTest. Therefore, it is necessary that we first configure a running project which will create JSON log data files.

In the run configuration file, we have to select “ JSON ” in the report format.

When we would run this command, the command line execution output is changed to json.

Display the Cucumber reports In the ‘ Add post-build operation ’ action, select the ‘ Cucumber reports ’ option.

Go to the ‘ Advanced ’ option under the ‘ Cucumber reports.’ In the ‘JSON Reports Path,’ enter “reports.”

Click Save to finish the project configuration.

Generating test report To generate the test report, we have to click the “ Build Now ” button. After the job is completed, a test report will be generated automatically. Besides, the test report provides us with a link to each scenario, where each link, when clicked, shows the details of the scenarios. The report also indicates the screenshot captured in each scenario step or hooks along with each step’s execution result.

So till now, what we have seen is how to integrate Cucumber with Jenkins and how to run a Cucumber script on Jenkins along with the generation of graphical reports.

It is also possible to leverage cloud-based platforms’ automation capability to make remote testing more seamless than ever. You can configure your Cucumber scripts to run on LambdaTest’s Selenium Grid cloud and perform testing on 2000+ browsers, devices & operating systems. In the next section, we will learn how you can make the most of Cucumber with Jenkins integration & LambdaTest.

Leveraging Cucumber Jenkins Setup with LambdaTest Cloud Selenium Grid

Below we would see how we can get our Cucumber scripts running on the LambdaTest platform, and further, we can use the same steps as defined above to integrate Jenkins to generate the reports.

Few prerequisites before we start running the scripts:

It is assumed that Java is installed on the machine. Preferably JDK 8 and above.
Also, it is assumed we have node.js installed. If not, you can easily install it from https://nodejs.org/en/.
Your LambdaTest credentials should be set as environment variables.
- If you are a Windows user, set them as set LT_USERNAME=”YOUR_USERNAME”

set LT_ACCESS_KEY=”YOUR ACCESS KEY”

If you are a Mac user, set them as export LT_USERNAME=”YOUR_USERNAME”

export LT_ACCESS_KEY=”YOUR ACCESS KEY”

In case you are unaware of the access key, you can easily retrieve them from your LambdaTest dashboard.

We also have to update .conf.js files inside the conf/ directory with these credentials.

After we have the environment setup aligned, perform the below steps to run the Cucumber scripts.

Make sure we have our repository cloned from Git.

In case it is the first time you are running a script for a repository hosted on Git, you can clone a sample repository provided by LambdaTest, which is NodeJs-Cucumber-Selenium from GitHub.
Navigate to the cloned directory and install the project dependencies using the below command.

$ npm install

Navigate to karma-jasmine-sample and run $ npm run single to run the test in single or $ npm run parallel to run the test in parallel.

Both the commands validate the test cases and execute the test suite across all the defined test groups. We can check the test results by opening the LambdaTest platform and navigating to the Automation dashboard.

The below sample snippet depicts the same.

Conclusion

This blog presents how a Cucumber with Jenkins integration can help run automation scripts successfully. Additionally, this Cucumber with Jenkins integration can also be leveraged to run scripts over the LambdaTest cloud platform. The best part is that project stakeholders can all read the test reports after the Cucumber Jenkins setup and understand the quality status. Jenkins also has many other features, such as setting up timed service, sending mail, and other functions. You can configure to further customize report notification according to project needs, such as sending HTML reports to key stakeholders as an email.

Do leave your comments if you found this blog helpful. If you have any questions or doubts, you can always shoot up your query on LambdaTest Community and get it resolved!

Happy testing!

How To Generate TestNG Reports In Jenkins?

Piyusha — Wed, 24 Mar 2021 10:32:38 +0000

TestNG is an open-source automated testing framework, where ‘NG’ of TestNG is Next Generation. It is similar to JUnit but designed to be better than JUnit, especially when testing integrated classes. With the help of simple annotations, grouping, sequencing & parametrization, TestNG overcomes most of the older system’s limitations and gives the developer the ability to write more versatile and efficient tests.

In our previous blogs, we have seen how to run tests in TestNG using an XML file, Parameterisation in TestNG, TestNG listeners, and much more. In this blog, we will find out how to generate a TestNG report in Jenkins. This will be particularly useful if you are into Selenium test automation.

Why Use TestNG in the First Place?

TestNG is quite popular among developers and testers for test creation. It offers several practical features like grouping, dependence, prioritization, ease of using multiple annotations, etc. If you are not a fan of TestNG, allow me to introduce you to a few of the several benefits offered by it, especially from Selenium perspective-

It supports annotations.
It has a flexible test configuration, ensuring that the code is multithread safe.
Support for data-driven testing (with @DataProvider).
Support for parameters.
No more need for TestSuite and accompanies a powerful execution model.
Supports a variety of tools and plugins, which include Jenkins, Eclipse, Maven, etc.
Embeds BeanShell for further flexibility.
It is designed to cover tests (almost all): Unit tests, Functional tests, End-to-end tests, Integration tests, etc.
It uses more Java and OO concepts and features.

Importance of TestNG Annotations

Primarily, developers use annotation repeatedly to define a piece of code that is incorporated into the software or business logic used in the programming language Java to control the flow of methods. In Test Next Generation (TestNG), annotations play a significant role. When working on TestNG, testers need to consider the operations and uses of each annotation.

Although Annotations differ from project to project depending on their requirements, the execution flow remains intact for every single project. Below are different Annotations present in TestNG and their primary usage:

@BeforeSuite: The annotated method will run before all tests in this suite have run.
@AfterSuite: The annotated method will run after all tests in this suite have run.
@AfterTest: The annotated method will run after all the test methods belonging to the classes inside the tag have run.
@BeforeTest: The annotated method will run before any test method belonging to the classes inside the tag has run.
@AfterGroups: The annotated method is a list of groups that the configuration method will run after. This method will run shortly after invoking the last test method that belongs to any of these groups.
@BeforeGroups: The annotated method is a list of groups that the configuration method will run before. This method will run shortly before invoking the first test method that belongs to any of these groups.
@AfterClass: The annotated method will run after all the current class test methods have been run.
@BeforeClass: The annotated method will run before the first test method in the current class is invoked.
@AfterMethod: The annotated method will run after each test method has been run.
@BeforeMethod: The annotated method will run before each test method has been run.
@test : This annotated method is a part of a test case.

There are several benefits of the available TestNG Annotations-

By looking up annotations, it describes the methods that it is interested in. Method names are also not limited to any pattern or format.
We can pass annotations with additional parameters.
Annotations are strongly typed so that the compiler can flag any errors right away.
Test classes are no longer needed for extensions.

Writing a Basic Test in TestNG

Now that we have essential elements in place let us see the primary steps involved in creating a TestNG test.

Write the business logic of the test.
Insert TestNG annotations in the code.
Include the test information (e.g., the methods names, class names, group names, etc.) in a testng.xml file.
Run TestNG.

Note: TestNG requires JDK 8 or higher. You can refer to our complete TestNG Tutorial to start from the basics.

Below is a straightforward test is written in TestNG:

package example1;

import org.testng.annotations.*;

public class SimpleTest {

 @BeforeClass
 public void setUp() {
   // here would be the code that will be invoked when this test is instantiated
 }

 @Test(groups = { "fast" })
 public void aFastTest() {
   System.out.println("Fast test executed");
 }

 @Test(groups = { "slow" })
 public void aSlowTest() {
    System.out.println("Slow test executed");
 }

}

Using @BeforeClass and @test , i.e., after the test class is built & before any test method is run, the setUp() method will be invoked. We will run the group quickly in this example, so aFastTest() is invoked while a SlowTest() is skipped.

Things to Note:

No need to implement an interface or extend a class.
Even though the above example uses the JUnit conventions, it is the annotations that tell TestNG what our methods can be called any name you like.
A test method could belong to one or multiple groups.

We can either invoke our test using the command line, the ant task (as shown below), or the XML file: once you have compiled your test class into the build directory.

<project default="test">

 <path id="cp">
   <pathelement location="lib/testng-testng-5.13.1.jar"/>
   <pathelement location="build"/>
 </path>

 <taskdef name="testng" classpathref="cp"
          classname="org.testng.TestNGAntTask" />

 <target name="test">
   <testng classpathref="cp" groups="fast">
     <classfileset dir="build" includes="example1/*.class"/>
   </testng>
 </target>

</project>

Use ant to invoke it:

c:> ant

Buildfile: build.xml

Output

The test results can be browsed using:

start test-output\index.html (on Windows)

If you don’t like this process of getting the test results, there’s an easier way to do it. It is also possible to generate the TestNG reports in Jenkins. There is a plugin Jenkins offers to accomplish this task. The plugin is the TestNG Results plugin. This next section will find out how to generate TestNG report in Jenkins with this plugin.

You might also like- Generate PHPUnit Coverage Report In HTML and XML

TestNG Results Plugin

TestNG Results plugin enables you to publish the results of TestNG created using org.testng.reporters.XMLReporter.TestNG The resulting XML file contains more details than the XML file of the JUnit framework. In graph and table reports, this plugin reveals such additional information. The plugin also makes it possible to import the XML TestNG reports in Jenkins from each build.

Now getting to the ‘how’ part of ‘how to generate TestNG report in Jenkins’- data is parsed using the output that org.testng.reporters.XMLReporter generates. The results are shown with a trend graph, and all the specifics of the failed tests are also presented. And that’s how TestNG reports in Jenkins look like!

Let’s find out more about a Jenkins Selenium report.

Features of TestNG Results Plugin

A Jenkins Selenium report generated using TestNG Results plugin offers the following features-

The page layouts are intentionally designed so that they look very similar to Jenkins’s JUnit plugin. Also known for the fact JUnit inspires them.
You will see a summary of passed/fail checks and passed/fail configuration methods on the project/build tab. To see the test failure, it also enlists the failed test cases with a direct connection.

The test methods in the class result summary page are separated from the configuration method.
In the Package result summary, we can see the order in which test methods were run during the regression run.

The pass/failed pattern of previous job results will be seen in different graphs.

Able to hide or expand tables that display package and class results.

Utilities of TestNG Results Plugin

Using this plugin is very straightforward once you have configured your build to run TestNG-based test cases and figured out how to generate TestNG report in Jenkins. Simply trigger Publish TestNG Results in Post-Build Behavior.

This option enables the following properties to be configured:

TestNG XML report pattern- This is a pattern for the file name that can be used to find the TestNG XML report files (for example, */target/testng-results.xml*). The path is either an Ant-style pattern (e.g., fileset) or a list of files and folders separated by the characters’;:,’. It is also vital that TestNG be configured to generate an XML Jenkins Selenium report using _org.testng.reporters. This is to ensure XMLReporter for this plugin functions properly.
The Escape Test Description string- If checked, while displaying test method results, the plugin escapes the description string that is associated with the test method. Unchecking allows us to use HTML tags to format the description, which is enabled by default.
The Escape exception messages- If checked, the plugin escapes the test method’s exception messages. Unchecking allows you to use HTML tags to format the exception message, e.g., embed links in the text. This is enabled by default.
Show Failed Builds- If selected, the plugin includes results from the failed builds in the trend graph. This is disabled by default.
- In case this is a maven build, it is always better to configure the build step with -Dmaven.test.failure.ignore=true option. This results in the build with test failures being marked as Unstable, thus distinguishing it from the build that failed because of issues not related to the tests.
- Also, even when this option is selected, failed builds with no results and aborted builds will not be displayed in graphs.
Marking the build as a failure on a failed configuration- This marks the build as failed if there are any configuration method failures.
Configuring thresholds for marking the build as Unstable/Failed- A Threshold Mode can be configured by the user as a number of tests or a percentage of tests and then configured separately for these values. Even if there are several test failures or label the build as failed, even if there is a single test method failure, this allows for situations where we want the builds to be marked successful.

How to Generate TestNG Reports in Jenkins?

Now let us see how to generate TestNG Report in Jenkins. To be able to do this, you need to have all the required tools in your system.

Prerequisites-

TestNG Project
Jenkins – You can check Jenkins installation here.
TestNG Results Plugin in Jenkins

Once you have all this installed, we can get started with a TestNG Project.

How To Create a TestNG Project?

In order to generate TestNG reports in Jenkins, we need to start from scratch. First, we will create a TestNG project. To create a TestNG project, you will need to follow the steps mentioned below-

Step 1: Open Eclipse and create a Java class.

Step 2: Keep all the library files in a folder.

Step 3: Write a Java program.

The below snippet highlights all the three steps mentioned above.

Step 4: Now convert the Java Program into TestNG.

Alternative Method- Another way you can create a TestNG Project is by creating a new TestNG project all from the beginning.

Step 1: Click File –> New –> Java Project

Step 2: Give it an arbitrary Project Name, say “FirstTestNGProject” and then click Next.

Step 3: Next step is to start importing the TestNG Libraries into the project. Click on the “ Libraries” tab –> “Add Library”

Step 4: Choose “TestNG” on the Add Library dialog, and click Next.

Step 5: Click on Finish.

It should be noticed that TestNG is included on the Libraries list.

Step 6: The JAR files containing the Selenium API will now be added. These files are located in the Java client driver we downloaded from http://docs.seleniumhq.org/download/.

Then, navigate to where the Selenium JAR files are placed.

After adding the external JARs, the screen should appear like this.

Step 7: Click on Finish and confirm that our FirstTestNGProject is visible on Eclipse’s Package Explorer window.

After we have the TestNG project, the next step is to install Jenkins. You can refer to the guided steps for Jenkins installation in our Jenkins Tutorial.

Now that we have TestNG Project and Jenkins platform ready, the next step is to install TestNG Reports Plugin in Jenkins.

Installing TestNG Reports Plugin In Jenkins

The next and the most crucial step to know how to generate TestNG report in Jenkins is to install the TestNG Reports plugin. To install the plugin in Jenkins, follow these steps-

Open Jenkins (localhost:8080).
Click on ‘ Manage Jenkins ‘ → Click on ‘ Manage Plugins ‘

Click on ‘ Available ‘ tab
Select ‘ TestNG Results Plugin ‘
Click on ‘ Install Without Restart ‘

Open the ‘ TestNGProject ’ Job created earlier.
Click on ‘ Configure ’ and scroll down to select ‘ Post Build Actions ’
Click on ‘ Add Post Build Actions ’ dropdown list.

And select ‘ Publish TestNG Results ‘.

In the TestNG XML Report Pattern edit box enter ‘ **/testng-results.xml ‘ and click ‘ Save ‘

We have already created a new project ‘ TestNGProject ‘to run TestNG Tests and generate TestNG Reports after execution using Jenkins.
Click on the ‘ Build Now ‘ button to execute it. This will invoke testng.xml from the batch file.
Right-click on Build Number and click on Console Output to see the result.
Once the execution is completed, there will be a link to view ‘TestNG Results’

There you go- that’s how to generate TestNG Report in Jenkins!

Alternative Method To Generate TestNG Reports in Jenkins

There’s another way to set up the TestNG reports in Jenkins Dashboard.

Login into Jenkins.
Manage Jenkins and Install TestNG Result Plugin.
Please make sure that you restart Jenkins after the plugin installation.

Next, go to Jenkins Home Page → Create New Jenkins Job and in Post-Build Action select → Publish TestNg Result.

Now either give the complete path of your testng-result.xml or type **/testng-results.xml in TestNG XML report pattern to search inside your project workspace.

Execute your Jenkins Job, and Now your Jenkins Selenium report or ‘ TestNG Results ’ would be visible in Your Jenkins Dashboard.

How to Share TestNG Reports in Jenkins?

In the sections above, we saw how to generate TestNG report in Jenkins or Jenkins Selenium report, as some like to call them. Once you have generated those TestNG reports in Jenkins, what would you like to do with them? The ultimate goal of generating a Jenkins Selenium report is to share them!

There is a possibility to send those email notifications directly from Jenkins along with the generated TestNG reports. In this section, we will learn how to email the generated Jenkins Selenium report to team members, leads or even managers.

Let’s see how we can accomplish this-

Start ‘jenkins.war’ file to get started with Jenkins ( Assuming it is already installed ).
Open Jenkins (http://localhost:8080).
We would need a required set of Plugins before we start:
- Email Extension
- Email Extension Template Plugin
- TestNG Results
- Selenium HTML Report
Configuration for Sending Email
- Go to Jenkins –> Manage Jenkins –> Configure System –> Email Notification
- Enter the email id in the “ SMTP Server ” edit field. For e.g. “smtp.gmail.com”

Click on “ Advanced ” and configure as shown in the below image.

Username and password need to be provided for the account from where we want to send the email notification( You can either use your corporate Email or personal based on situations).
It is always better for testing purposes to check the “Test configuration by sending test email” checkbox to validate whether Email was sent successfully or not.

Enter the recipient’s email address and click on “Test Configuration” button.
If we can see the “Email was successfully sent” message, then it can be confirmed that the configuration is correct.
If we see any error message, it could be either because the configuration is incorrect or security access issues. Follow the below steps to enable “Less secure app access.”
Navigate to https://myaccount.google.com/
Click on “ Security “
Go to the “Less secure app access” section and switch it on.

Yay! You are all set to send the email notification for TestNG reports in Jenkins!!

Now it’s time to test this.

Go to Jenkins Dashboard
Create a new Job.

Let us create a new project named “VerifyEmailNotification”.

Give the Project Description and select ‘Execute Batch Command’ from the Build drop-down and enter any invalid commands as shown in the image below, as each unstable build will receive an email. (Then only email alerts will be sent if the build fails).

Select “Email Notification” from “Post-build Actions” and add the email address of the recipient and check “Send email for every unstable build” as shown in the image below.

Click on “Save.”
Now click on “Build Now” to start the execution.
Go to “Console Output” to verify the result.

Validate the recipient’s mail to confirm if the email was sent from Jenkins.

Now that we are confident our email notifications are working, let us configure Jenkins to attach a TestNG Report in the Email.

Go to Jenkins –> Manage Jenkins –> Configure System –> Extended Email Notification
Enter “smtp.gmail.com” in the “SMTP Server” edit field NOTE: Here, Gmail is taken just as an example.
Next, click on “Advanced” and fill in the details as shown below.

Click on “Save.”
Go to Jenkins dashboard and create either a new project, or you can choose an existing project. Here we have an existing project [Flight Reservation (POM)]

Click on Configure on the left-hand side.
Select “Editable Email Notification” from “Post-build Actions”
Click on “Advanced Settings”
Select “Always” from the “Add Trigger” dropdown.
Click on “Advanced” and fill in the details as shown below.

Mention the path of emailable-report.html of TestNG in the Content edit field.
Click on “ Save.”
Next, click on “ Build Now ” to start the execution of the job.
Navigate to “ Console Output ” to check the result.
Validate your recipient mail to see the Test Execution report.

And here you go. Now you are ready to generate TestNG reports in Jenkins & share those Jenkins Selenium reports!

Wrapping Up

I hope this blog has helped you take report generation and sharing one step ahead. Now you not only know how to generate TestNG report in Jenkins, but you have also learned how to share those TestNG reports in Jenkins via Email. It’s time to elevate Jenkins & TestNG’s functioning by sending out the email notification with your very next Jenkins Selenium report. Let me know how it goes in the comments section below.

Happy testing!