Automate Testing Your API with Postman, Newman and Jenkins (Part 3)

This is Part 3 of a four part series on how to automate testing with your APIs. If you missed the introductory blog, you can find that here, the second blog about Collection Runner can be found here, and the fourth summary blog can be found here.

Adding Postman Tests to Jenkins

Recall in Part 2 that we were able to download our entire Collection of end points and the tests within each end point into a json file.

Here is what an Environment file looks like:

 <code>
 {
 "id": "85aa6050-b035-f1ac-d5f2-0c19db311268",
 "name": "Demo-OpenWeatherMap",
 "values": [
 {
 "enabled": true,
 "key": "APPID",
 "type": "text",
 "value": "YOUR ID HERE"
 },
 {
 "enabled": true,
 "key": "CITYNAME",
 "type": "text",
 "value": "Philadelphia"
 },
 {
 "enabled": true,
 "key": "CITYID",
 "type": "text",
 "value": "4440906"
 }
 ],
 "timestamp": 1512531783947,
 "_postman_variable_scope": "environment",
 "_postman_exported_at": "2017-12-06T03:46:26.268Z",
 "_postman_exported_using": "Postman/5.3.2"
 }
 </code>
 

Recall that an Environment stores key-value variables that are specific to your environment. For example, you may have a Dev, Stage, and Prod environment. Your APPID variable can have a different value depending on which environment you are working in. In the export, they are just that, key value pairs.

A Collection file looks more complicated but don't worry. It's just just more json, albeit with some JavaScript sprinkled in. Recognize the JavaScript? It's the code you wrote in the Tests tab within Postman.

 <code>
{
 "info": {
 "name": "DEMO-TESTS",
 "_postman_id": "15918068-3ba0-bb61-5f65-1d411301f1ae",
 "description": "",
 "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
 },
 "item": [
 {
 "name": "GET weather by RANDOM city id",
 "event": [
 {
 "listen": "test",
 "script": {
 "type": "text/JavaScript",
 "exec": [
 "var jsonData = JSON.parse(responseBody);\r",
 "\r",
 "//Saving the response \"name\" value into an environment variable\r",
 "postman.setEnvironmentVariable(\"CITYNAME\", jsonData.name);\r",
 "\r",
 "pm.test(\"Status code is 200\", function () {\r",
 " pm.response.to.have.status(200);\r",
 "});\r",
 ""
 ]
 }
 },
 {
 "listen": "prerequest",
 "script": {
 "type": "text/JavaScript",
 "exec": [
 "var cityIds = [",
 " 5206379, // Pittsburgh",
 " 4440906, // Philadelphia",
 " 4505716, // Baltimore",
 " 4160812, // Key West",
 " 3882428, // Lost Angeles",
 " 5128638, // New York",
 " 5809844, // Seattle",
 " 1723862, // Boston",
 " 4887398, // Chicago",
 " 5506956, // Last Vegas",
 "];",
 "",
 "// Get a random number between 0-9",
 "var randomCityId = Math.floor((Math.random() * 9) + 1);",
 "",
 "// Set an environment variable to the value of the random city id from array",
 "postman.setEnvironmentVariable(\"CITYID\", cityIds[randomCityId])"
 ]
 }
 }
 ],
 "request": {
 "method": "GET",
 "header": [],
 "body": {},
 "url": {
 "raw": "api.openweathermap.org/data/2.5/weather?id={{CITYID}}&APPID={{APPID}}",
 "host": [
 "api",
 "openweathermap",
 "org"
 ],
 "path": [
 "data",
 "2.5",
 "weather"
 ],
 "query": [
 {
 "key": "id",
 "value": "{{CITYID}}",
 "equals": true
 },
 {
 "key": "APPID",
 "value": "{{APPID}}",
 "equals": true
 }
 ]
 },
 "description": ""
 },
 "response": []
 },
 {
 "name": "GET weather by city name",
 "event": [
 {
 "listen": "test",
 "script": {
 "type": "text/JavaScript",
 "exec": [
 "var jsonData = JSON.parse(responseBody);\r",
 "\r",
 "pm.test(\"City Name matches Request'\", function () {\r",
 " var contains = false;\r",
 " var cityFromResponse = postman.getEnvironmentVariable(\"CITYNAME\");\r",
 " //console.log(cityFromResponse);\r",
 " \r",
 " if(jsonData.name == cityFromResponse) {\r",
 " contains = true;\r",
 " }\r",
 " else\r",
 " {\r",
 " console.log(\"Unexpected city\");\r",
 " }\r",
 " \r",
 " pm.expect(contains).to.be.true;\r",
 "});\r",
 "\r",
 "pm.test(\"Contains object: 'weather'\", function () {\r",
 " var contains = false;\r",
 " for (var key in jsonData) {\r",
 " if (key == \"weather\") {\r",
 " contains = true;\r",
 " }\r",
 " }\r",
 " pm.expect(contains).to.be.true;\r",
 "});\r",
 "\r",
 "pm.test(\"Contains string: 'rain' \", function () {\r",
 " var contains = false;\r",
 " var expected = \"rain\";\r",
 " pm.expect(pm.response.text()).to.include(expected);\r",
 "});\r",
 "\r",
 "pm.test(\"DOES NOT Contain string: 'snow' \", function () {\r",
 " var contains = false;\r",
 " var expected = \"snow\";\r",
 " pm.expect(pm.response.text()).to.not.include(expected);\r",
 "});"
 ]
 }
 }
 ],
 "request": {
 "method": "GET",
 "header": [],
 "body": {},
 "url": {
 "raw": "api.openweathermap.org/data/2.5/weather?q={{CITYNAME}},us&APPID={{APPID}}",
 "host": [
 "api",
 "openweathermap",
 "org"
 ],
 "path": [
 "data",
 "2.5",
 "weather"
 ],
 "query": [
 {
 "key": "q",
 "value": "{{CITYNAME}},us",
 "equals": true
 },
 {
 "key": "APPID",
 "value": "{{APPID}}",
 "equals": true
 }
 ]
 },
 "description": ""
 },
 "response": []
 }
 ]
} 

 </code>
 

Checking your code into a Repo? Why not add your integration tests to the Repo too? That is what we are going to do. Go ahead and add these exported files to source control so that Jenkins can pull them down when it builds your project. We are using Git here. We will add the Postman files to the root of the project.

While I don't recommend you do, you can edit the exported files directly if you wanted to make a quick change or do a "search and replace." Personally, I believe it is best to make all edits within Postman and re-export the files. This will ensure the json is still valid and that you don't break the schema that Newman is expecting.

How do we tell Jenkins to run Newman?

Remember, the goal is to have these integrations tests run on every check-in. But how do we tell Jenkins to do this? We will write a script that calls Newman and pass it the Environment and Collection json files! At a client we demoed this to; they were hosting Jenkins on a Windows server. For this scenario, we will be writing a Powershell script. If your Jenkins server is on Apache, you can do the same thing in Bash. You can download the Powershell file here https://github.com/duyn9uyen/postman-jenkins-demo/blob/master/Jenkins_PostmanDemo.ps1

 <code>
 #Windows Batch Command in Jenkins

 echo "You are in the powershell script now..."
 $SourceFilePath = $env:WORKSPACE
 $FilenamePostfix = "*.postman_collection.json"
 $EnvironmentFile = "Demo-OpenWeatherMap.postman_environment.json"

 # Get all Postman test files
 $JsonFiles = Get-ChildItem -Path $SourceFilePath -name -Filter $FilenamePostfix | Sort-Object -Property CreationTime -Descending

 # Change to directory where we have NodeJs installed. Otherwise, the 'newman' command will not be recognized. 
 # You can install NPM and Newman as a user and copy the files from C:\Users\[username]\AppData\npm into C:\ drive.
 #cd You can find the NPM packages here: C:\Users\[username]\AppData\Roaming\npm\node_modules\newman\bin
 cd C:\npm\node_modules\newman\bin

 # Loop through the json files and execute newman to run the Postman tests
 foreach ($File in $JsonFiles) {
 $collectionfilepath = "$SourceFilePath\$File"
 $environmentfilepath = "$SourceFilePath\$EnvironmentFile"
 node newman run $collectionfilepath -e $environmentfilepath --disable-unicode 

 if($LASTEXITCODE -eq 1) {
 echo "Integration error found!"
 exit 1
 }
 }

 exit $LASTEXITCODE
 </code>
 

• I added an echo so that I know I'm in the Powershell file. You will see this output on the Jenkins server. It's good to know exactly when my script is started vs other outputs from Jenkins.
• SourceFilePath is the Jenkin's environment variable denoting the location of where your project files build on the server.
• FilenamePostfix is the default filename that Postman appends to your file when you export them.
• EnvironmentFile is the name of the environment file to be used. (You can make this smart by making this a parameter that you can pass in.)
• I start off by getting all json files in the location you have the Postman files. It's just the root directory here.
• One tricky thing is to get Jenkins to recognize npm commands. To do this, I had to copy the NPM packages to a location that Jenkins can have access to. That is why I change to the directory to C:\npm\node_modules\newman\bin with the CD command.
• Once I'm in the directory that has all the NPM packages, I can now call Newman! I loop through my json files and run Newman for every Postman Collection file I find. So in theory, you can have however many Postman collections you want. I'm sure there is a limit before things start breaking but I have about 5 collections and about 7-10 endpoints in each collection in a project that I'm working on.
• exit $LASTEXITCODE, is very important. This command returns the result of the last command. So I make sure that on every loop, the Newman command was successful. If any tests fail, I stop the script and exit 1. This will result in a failed build on Jenkins.

Add Script to Jenkins

Go into Jenkins and create a Freestyle Project. Configure this project to pull your repo code from Git. Once you have it building your project correctly, we will add in a Build step. Find the General Tab, Build section and add the following command:

powershell.exe -executionpolicy bypass -command "& '%WORKSPACE%\Jenkins_PostmanDemo.ps1'

• The "-executionpolicy bypass" is needed or else Jenkins won't run the script. This is similar to changing the Execution Policy in Part 2.

If you want an email alert when the build breaks, configure the E-Mail Notification section. You can search the web for details on how to configure Jenkins to do that.

Run The Build and Examine Results

Run your project you just created and see if Newman runs properly. Remember when I echoed "You are in the powershell script now?" Examin the Jenkins output and you can see why I did that.

Here, we have a failure because the city "Philadelphia" did not include "rain" in the weather response.

Summary

We talked about the exporting Postman files, adding these files along with your code to Git. We wrote a Powershell file so that we can give to Jenkins to execute Newman on our Postman collection and integration tests. I showed you where in your Jenkins project to add the Powershell script. We ran the Jenkins project and examined the results of the integration test.

Next, we will take a step back and understand what we did in Parts 1, 2 and 3.

Continue onto Part 4!