Python BDD Testing with Jenkins support

Juan Amezcua 11 Jun 2018

Python BDD Testing with Jenkins support

This tutorial will explain how to run implement BDD with Python and Behave. Also how to run this in Jenkins avoiding some pitfalls and allowing the code to be aware of the environment.

Background

Behave is a behavior driven development framework for Python. It uses Gherkin, a domain specific language that is business readable. This language lets you describe software’s behaviour without detailing how that behaviour is implemented. For those with a Javascript background Gherkin is also used by Cucumber.

Tool support

PyCharm Gherkin plugin

PyCharm has a “Gherkin” plugin that supports editing .feature files.

alt text

It provides .feature files with a specific icon in the project explorer.

alt text

It also provides syntax coloring for them.

alt text

Behave scaffolding

The paid version of PyCharm provides scaffolding of the unimplemented steps in your feature files. A workaround is possible with the Community Edition.

Go to “Edit configurations…”, add a new one for main.py.

alt text

Name it “snippets” and in parameters add the “–snippets” switch.

alt text

When you run this the output will provide for you the scaffolding for the unimplemented steps in your features files.

As an example, we create a test.feature file that has no corresponding test.py file implementing the steps.

Feature: The day must be sunny

  Scenario: Force the sun to shine
    Given a cloudy day
    When we sing to the sun
    Then the sun comes out and shines

Then we run the snippets configuration, after the output corresponding running the .feature files that have a matching .py file, we get the following output:

You can implement step definitions for undefined steps with these snippets:

@given(u'a cloudy day')
def step_impl(context):
 raise NotImplementedError(u'STEP: Given a cloudy day')

@when(u'we sing to the sun')
def step_impl(context):
 raise NotImplementedError(u'STEP: When we sing to the sun')

@then(u'the sun comes out and shines')
def step_impl(context):
 raise NotImplementedError(u'STEP: Then the sun comes out and shines')

Project struture

This is the general layout of the project. In the root we have:

setup.py, the dependencies that we require to be installed and some metadata main.py, launches our tests and if some extra configuration for behave would be needed this is the place to put it features, in this folder we will store the .feature files containing the our BDD expressed requirements features/steps, in this folder we will store our python files. For each .feature file we must have a Python file, i.e., “features/correct_data.feature” -> “features/steps/correct_data.py” resources, if some other resources are needed like data to match some results this is the place to store them

alt text

Feature files

The behavior of our software is going to be described using .feature files. Here is an example:

Feature: The flow returns the expected status codes

Scenario: Run the flow with a valid date Given a date 2018-05-25 When we make a post request to fx-rate-curves Then the response status code is 200 The keywords here are “Feature”, “Scenario”, “Given”, “When”, “Then”, but there are two more “And” and “But”, these are renamed by behave to take the name of their preceding step

Every line begins with a keyword and we have one keyword per line. “Feature” groups one or more “Scenario”s.

Tags

An scenario may be tagged like in the code below with “@some_word”. This can be used to run tagged scenarios. @slow Scenario: Run the flow with a bogus date Given a date bogus When we make a post request to fx-rate-curves Then the response status code is 500 With the command switch tags we can specify that we only want to run the scenarios that:

have one of the tags “tag1”, “tag2”, “tag3” –-tags=tag1,tag2,tag3” have all of the tags “tag1”, “tag2”, “tag3” –-tags=tag1 –-tags=tag2 –-tags=tag3 This feature is compatible with the way main.py is implemented. This is explained below.

Data driven testing

In case we are doing data driven testing we may include tables or chunks of text in the steps.

Tables

They can be specified this way in the .feature files:

Scenario: some scenario
 Given a set of specific users
 | name | department | 
 | Barry | Beer Cans | 
 | Pudey | Silly Walks | 
 | Two-Lumps | Silly Walks |

 When we count the number of people in each department
 Then we will find two people in "Silly Walks"
 But we will find one person in "Beer Cans"

The table data will be stored in the “context.table” variable and will be accessed like this:

@given('a set of specific users')
def step_impl(context):
    for row in context.table:
        model.add_user(name=row['name'], department=row['department'])

Text

It can be specified this way in the .feature files:

Scenario: some scenario
 Given a sample text loaded into the frobulator
 """ 
 Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
 eiusmod tempor incididunt ut labore et dolore magna aliqua.
 """ 
 When we activate the frobulator
 Then we will find it similar to English

Step files

Each .feature file must have a corresponding .py file inside the “features/steps” folder and for every different step sentence in every feature a step must be implemented. Implementations may be reused between scenarios in the same .feature file.

For the previous example the step “Given a date 2018-05-25” is implemented by:

@given("a date {date}")
def date_is_valid(context,date):
    context.date = date

As you can see, in every function we may pass a context variable and any other variables that we have created for the specific step:

context, global variable used to communicate information between steps. We can create more “fields” inside this object, i.e., context.date. Besides “context.table” and “context.text” there is another base filed “context.failed”. With this variable we may test if a step has failed. step variables, we define them in the step and take the value from the feature file, i.e., in the previous example we have a date variable that is going to be dealt with as a string and would take its value from the feature file as 2018-05-25. Other kinds of restrictions may be set on the variable types/content (see behave site documentation) Environmental controls We can also use fixtures just tagging some function with @fixture or specify code that we want to run:

@fixture def selenium_browser_chrome(context): # – HINT: @behave.fixture is similar to @contextlib.contextmanager context.browser = webdriver.Chrome() yield context.browser # – CLEANUP-FIXTURE PART: context.browser.quit()

def before_all(context): use_fixture(selenium_browser_chrome, context) # – HINT: CLEANUP-FIXTURE is performed after after_all() hook is called.

def before_feature(context, feature): model.init(environment=’test’) ´´´ ##Runnning behave The starting point for the behave tests is the main.py file. As you can see below, the only specific piece of code is running behave_main() and passing the exit code out. If the tests succeed will be zero and one if they do not.

from sys import exit
from behave.__main__ import main as behave_main

if __name__ == "__main__":
    exit(behave_main())

Besides the –tags switch mentioned above other switches may be passed. For further information see the behave documentation.

Jenkins integration

Running this kind of tests in Jenkins is easy. We just have to include a file named “Jenkinsfile” in the root of our Github repository and request from Jenkins to run “setup.py” and “main.py” in this order with the python interpreter (the code snippet for this is in the “Writing permissions” section below).

But integration poses three problems:

setup( name = “ndf-fx-rate-curves”, version = “1.0”, packages = find_packages(), include_package_data=True, url = ‘https://github.com/cardano/flow-system-fx-rate-curves’, description = “E2E testing for FxRatesCurvesWebApi NiFi flow. Basic testing for the REST endpoint, when data is stable testing the DB contents should be done”, install_requires = [“behave”, “requests”] )

![alt text](/assets/images/post-images/2018_06_11_python_bdd_005.png "PyCharm Gherkin plugin")

To run this you should setup how to run "setup.py". Pay attention to the "install" text in the "Parameters:" textbox.

![alt text](/assets/images/post-images/2018_06_11_python_bdd_006.png "PyCharm Gherkin plugin")
### Writing permissions
This is solved by adding the "--user" switch. It writes to the user folder.
```groovy
stage('Run End to End test') {
    when { expression { return !fileExists('./.jenkinsTriggered') } }
    steps {
        dir('test/e2e') {
            sh 'python3.6 setup.py install --user'
            sh 'python3.6 main.py' 
        }
    }
}

URL naming

This is solved with two actions:

declaring an environment variable in the Jenkinsfile. With a more general view in mind we could make this environment variable depend on the environment (dev, pre-prod, prod) or even make it specific for branch.

environment {
    BASE_URL_IN_JENKINS = 'http://some_url_in_jenkins'
}

making the code aware of the existence of this variable: from os import environ

BASE_URL_IN_JENKINS_KEY = "BASE_URL_IN_JENKINS"
BASE_URL_IN_DEV = "http://some_url"
BASE_URL = environ.get(BASE_URL_IN_JENKINS_KEY) if BASE_URL_IN_JENKINS_KEY in environ else BASE_URL_IN_DEV

Further information

The behave site extends the information provided here and explains other features: http://behave.readthedocs.io/en/latest/

The Jenkins site has more information regarding Jenkins: https://jenkins.io/doc/