Category: Postman

Continuous Delivery – OPA and Postman, Newman and Jenkins #4

Continuous Development – OPA and Postman, Newman and Jenkins #4

In this part of our series, we will take our existing platform composed of Postman, Newman and Oracle Intelligent Advisor and plug it into Jenkins, to allow us to run lights-out tests according to a schedule, and to enable the testing team to receive emails when the tests have completed.

For reference, the other parts of this series are here : Part 1, Part 2, Part 3.  For the record, let’s be absolutely clear about one thing – this is not the only way to automate this process, and there are other tools and platforms that can do the same job. Heck, if you are lucky you will not even have to worry about this because some other team or department already has this thought through and all you have to do is deliver projects. But you might want to understand the mechanics of this process anyway, or if you are unlucky (or lucky, depending on your point of view)  then when the musical chairs are happening and you are the one left standing so you inherit responsibility for this, at least you will have a head start.

So, off to work. Many of you will know Jenkins, the stable, powerful open source automation platform that allows us to pretty much build, deploy or test anything. The biggest bonuses of Jenkins are that it is free to use, easy to install and that it comes with an incredibly large set of plugins, which extend the functionality to include lots of other things.

If you have installed Jenkins, which is about as easy as it can be, you should find yourself looking at the basic setup screen of Jenkins:

If you are pressed for time, just accept the suggested plugins first, and then return to the list of plugins later. You will need to add several, if they are not already present. The most important one is the Node.JS plugin, since we want to use Jenkins to communicate via Newman, and you will remember from the last post in this series that Newman runs on Node.js.


  1. Manage Jenkins, Plugin Manager
  2. Click Available to find more plugins
  3. Filter for nodejs
  4. Select the plugin
  5. Install without a restart or with a restart as you wish depending on whether you have more to install.

Other important plugins include potentially the Email Extension Plugin which you might find useful, and potentially lots of others. You will also need to make sure that you have installed any plugins you want to use with Newman, recall that in the last post in the series you added an HTML report generation to Newman. If you didn’t do that yet, you should do it now. There are a few other settings in Jenkins that you need to setup, notably the System Admin Email Address but most of it should be fairly straightforward and you will find out that you need it when Jenkins complains.

Now you need to tell Jenkins a little more about Newman. In the Global Tool Configuration of Jenkins, via the Manage Jenkins option, scroll down to the Node.js seection and ensure you have added the details about your Node.js installation, and that you want to use Newman with it.

Jenkins Setup NodeJS

  1. Add NodeJS
  2. Give it a name whatever you would like to see in the logs
  3. Install Automatically
  4. Choose the version that you installed (the installer would have told you that)
  5. Add this to ensure that Newman and the HTML Reporter you used with Newman are available

Save all of that and you are just about ready to use Jenkins for real.

Create a new Freestyle Project in Jenkins and give it whatever name you wish. Then in the Build Environment section, ensure that you signal the need for Node.js. In the Build Step, choose Execute Windows batch command (unless you are not running windows, in which case there is a non-Windows equivalent). Add, for example, the following command line to run Newman with Node.js and use the HTML reporter, with your data file from the previous post in this series. The file name ensures that this file is overwritten each time you run this project. If you don’t you will end up with lots of files, each using a timestamp as the name, which can get confusing.

Having set up our Jenkins Project, we are going to add a Post Build Action to ensure that your team (or just you) gets an email with the report from Newman. The screenshot below assumes you have added the Extended Email Plugin mentioned above. Note the items below the image.

Jenkins Post Build Step

  1. Without going into too much detail, the extended email plugin provides all sorts of options for creating email groups and deciding which email (there can be many templates) goes to which person.
  2. This is where you add the Newman HTML report. Note the syntax of the  attachment path is not maybe what you are used to.
  3. The Jenkins build log can also be included with the mail, and can be useful if you have added many build steps and want to see what they did.

At a minimum you might want to set up an email that is sent when there are failures:

You can click the advanced options and begin that process to set up who receives emails in different trigger circumstances.

Save your project. Now, as a final point, you may which to schedule these tests to run when you are not in the office. So you can add Build Triggers from the top of the Project Configuration and use CRON-style syntax to decide when this will run.

Jenkins Setup Build Triggers

Now all you have to do is save your Jenkins Project. Of course, if you don’t want to wait until midnight you can always Build Now. You will recall from the third part of this series that my data file had an iteration in it that was set up to cause the Postman test to fail, because the journey time was too long. So, sure enough, I get an email in my Inbox:

Jenkins Email 5

The HTML report is attached to my email, and I can view the details of the failure, which is indeed the iteration 3 from my data file:

So we are now in a position to test, in an automated fashion, our Web Service from Oracle Intelligent Advisor. Although the example given was using the REST assessor, Postman will happily let you run SOAP XML calls as well, so this would be applicable too. In the next part of the series we will look at using Jenkins to test our HTML-based Interview for the same project.

See you soon!

Continuous Delivery – OPA and Postman, Newman and Jenkins #3

Continuous Development – OPA and Postman, Newman and Jenkins #3

Following on from the two previous episodes, (part one / part two) the stage is almost set to move to automation over and above the simple concept of a Postman Collection. In this chapter we will see how to use data files in Postman to facilitate making use of CSV or JSON data in a dynamic way, and we will get Newman up and running. So let’s start with the first element.

In our Collection, we currently have one request which is sending information to OPA and getting a journey time and plan back in response. Great. But we want to test our journey planner with more than one journey. And we want to be able to easily change the journeys without having to mess with our request. So this is where the ability to load data into a Collection from a text (CSV or JSON) file comes in very handy in deed. In pictures

  • Create a CSV file with your data in it. For the project used in this example we would need to have something like this

Collection Data File

Notice the headers are origin and destination. They will become variables that we need to add to our request.

  • Make sure that the request is updated

So now my request looks like this:

Request Modified

Don’t forget to save your request. Now if you go to Collection Runner and run your Collection, you can use the Select File button and select your file:

Using a Data File in Postman

As soon as you select the file, you will see that the number of iterations changes to take into account all the rows in your file. Now when you want to push a new set of journey tests all you need is to change that file. Awesome!

Armed with all of this, we can now move to the next challenge. Postman is great but not everyone wants to use the graphical user interface all the time. How about the ability to run a Collection from the command line? That would be great AND would open up lots of options to running Windows Scripts and the like. Well, the news is good. Newman is the CLI (Command Line Interface) for Postman. Setting it up if you are not familiar with Node.js can be a bit daunting, but let’s just make a shopping list first:

  1. Install Node.js (here)
  2. Install Newman using the Node Package Manager
    This step should be as easy as opening the Node Command Prompt (which got installed in your Windows Start Menu in the first part) and typing something like the following:

    Installing Newman

  3. Run Newman from the Node.js prompt to use Postman Runner with your chosen Collection and Data File using the ultra simple command line.

So the last step might look a little like the following – notice that in the command line you specify the Collection (which you must first export to a file from inside Postman) and the Data File (wherever it is – I put both files in the same location for ease of use. And once the job is done, Newman reports back with it’s own inimitable, Teletext-style display of the output. A word of advice – before exporting a Collection make sure you have saved all the recent changes!

Exporting Ready for Newman


Once the Collection is exported, you might run it like this. This is just a simple example. It is possible to also load environment variables and more to fine tune it.

Newman Command Line

After a few seconds we get the output from Newman. Note the 4 iterations, and that iteration 3 failed due to an excessively long journey time.

Newman Output in Teletext

Now this is of course a major step forward – running Collections from the command line, feeding in data sets just by using a simple command line switch – these features mean we can really now start to think of automation in our approach to testing this project. But we are not done yet. The output from Newman is pretty ugly and we need to change that. By installing Newman HTML Reporter, we can get a much nicer file. Follow the steps in the link and then change your example command line to something like this:

Note the new command line option -r html (which means, use the HTML reporter) and the –reporter-html-export option (with two dashes) which ensures the output file is stored in the folder of my choice. And the HTML output is so much better:

Newman HTML Reporter

What a nice document. So far so good. We can spin these collections with just a command line. But wait a minute, what about running tests when you are not in the office – sure, you could write yourself some sort of Powershell Script in Windows. But what if you wanted to run a set of tests every day at 3pm, then again on Tuesdays, then once on the first Monday of every second month…it would get quite hard. And what if you wanted to send the HTML Report automatically to all your colleagues on the testing team…but only if there were failures?

And so, we march onward to the next phase. See you shortly!

Continuous Delivery – OPA and Postman, Newman and Jenkins #2

Continuous Development – OPA and Postman, Newman and Jenkins #2

Following on from the previous post in this series, we now have Oracle Policy Automation and Postman playing nicely together. In this chapter we will use  some scripts in our Collection, both to streamline the authentication process and to provide us with some feedback about the execution of our collection.

So, first things first let’s add some scripts to our Postman Collection. In Postman there are two sorts of scripts, which you can think of as “before” and “after” scripts. For example you might run a script to prepare some data, or to authenticate your user with Oracle Policy Automation before you actually run a request. Similarly, after the test has run, you might investigate an output attribute from Oracle Policy Automation, or look at the response time of your request and compare it to others. That sort of thing. Let’s look at some useful examples. And just before we continue, remember that “before” and “after” scripts are available both at the Collection level as well as the individual Request level.

  1. Authenticating before the Collection is run

This is a common need  – to make sure the Collection can actually run without having to manually go and authenticate if your 1800 seconds are up. Of course more sophisticated scripts could store the current expiry and check to see if authentication is needed but this is just a simple example of how to authenticate. For example, you could paste this into your Pre-Request Script or your Pre-Collection Script:

var myVar = "?grant_type=client_credentials&client_id=YOURUSER&client_secret=YOURPASSWORD";
console.log("Updating Token");
url: "http://YOURSERVER/determinations-server/batch/auth" + myVar,
method: 'POST'
}, function (err, res) {
pm.environment.set("OAuth_Token", res.json().access_token);

This requires you also to set up a variable in your Environment in Postman called OAuth_Token, and also to ensure that your Collection knows to use it. Follow these steps.

  1. Create the Environment Variable by clicking the Eye icon and then Edit. Leave the value blank, your script will populate it automatically, Ignore the other variables in the screenshot they are not used by this simple script.
  2. Add the Script to manage the Variable.
  3. Add a reference to the new variable in Authentication.

So in pictures, first the Environment Variable to be added:

Postman Variables 3

And the Collection or Request Pre-Request Script Tab:

Postman Script Before

And in the Authentication Tab (note the double braces):

Postman Variable Usage

Now we can launch the Collection and know that the authentication token will be acquired as well, without having to remember to do it manually.

This is the first step towards our automation platform being truly useful. But before we do that, let’s add a couple more things to our Collection. We will add a test script that actually tests something. In my example Project, one of the outputs is the duration of the trip between the two stations. Let’s say that for whatever purpose, we consider a journey of greater than 20 minutes to be a functional failure. So we want to write an “after” script that looks at that for us. And let’s also say that we want to look at the response time and ensure that it meets our internal guidelines. So we might have a script like this:

function standardDeviation(values, avg) {
var squareDiffs = => Math.pow(value - avg, 2));
return Math.sqrt(average(squareDiffs));

function average(data) {
return data.reduce((sum, value)=>sum + value) / data.length;

if (responseCode.code === 200 || responseCode.code === 201) {
response_array = globals['response_times'] ? JSON.parse(globals['response_times']) : []
postman.setGlobalVariable("response_times", JSON.stringify(response_array))

response_average = average(response_array);
postman.setGlobalVariable('response_average', response_average)

response_std = standardDeviation(response_array, response_average)
postman.setGlobalVariable('response_std', response_std)

This script uses three global variables that you create in the same way as environment variables and one of them is an array, representing the different times you run your request for example. In pictures:

Add an “after” Test Script:

Add the Variables :

Run the Collection or Script and observe the variables are evolving:

And let’s add another test, this time to check the journey duration is not too long (paste this after or before the existing “after script).

As you can see above, you are able to access the output of the request using JSON and then you can pose a condition. In the example above, the test has failed because I chose two stations that are very far apart. This is getting rather interesting. The next part is to use some data that makes our requests a bit more useful and to automate them. See you soon  for the next part!

Kudos to the following pages :

How to Automate OAuth2 in Postman

A Postman script to calculate average response times

Logo by Southpaw Projects LLC