Archive for the ‘Testing’ Category

Continuous Integration for Agile Project Managers (Part 3)

June 15, 2010

In part 1 of this series, I hopefully provided you with an introduction to Continuous Integration (CI) and an overview of the building blocks of the CI process.

With part two, we introduced the concept of a CI life-cycle and how to bind software quality checks to discrete life-cycle ‘phases’.

In this final part, I’ll attempt to explain how you can use CI to run functional tests early in your development process and show how this helps us to promote a software release on its journey from development to production. I’ll then try to condense the three parts of this series into a handy diagram.

Automation, Automation, Automation

It should hopefully be clear by now that a successful CI implementation is largely dependent upon other processes that can be automated. For example

  • checkout or update of source code by a version control tool
  • compilation of project source code by a build tool
  • execution of source code checking by a static analysis tool
  • unit test execution by a unit test harness or runner

Therefore, it seems reasonable that if we can automate a software development task in a way that is deterministic and repeatable then it can be executed during our CI process. The intention is to remove human error and have these tasks run in the background, and in the event of a ‘failure’ to be notified by a CI server. Some more advanced candidates for automation might include:

  • provisioning of a clean, known harness or environment into which we deploy a release for the purposes of functional or acceptance testing
  • starting and stopping services that are required for our functional tests and population of any test data dependencies
  • execution of the functional tests

Functional Testing and CI

OK, so how do you go about adding these additional capabilities? Well, creating an automated suite of functional tests that can be triggered during the CI process is a reasonably non-trivial task.

Let’s take a look at the main areas for consideration.

Test Fixtures

In addition to writing the tests, you also need to carefully consider your test dependencies (also known as ‘fixtures’), such as any test data you might need in a database for example. Some code frameworks, like Ruby On Rails, encourage development teams to consider these test fixtures early-on in a software development project. Outside of such frameworks, it is more common to find ad-hoc processes. The key thing is ensure that your dependencies, such as a database, are in a known state prior to test execution and that the configuration or population of dependencies is entirely automated. Here, we can again apply the concept of a CI life-cycle to this particular scenario. We will need to bind the activity of test data population to a life-cycle phase prior to the execution of functional tests. We may also need to create a subsequent binding after test execution to ensure that we ‘tear-down’ the test data and keep everything in a known, consistent state.

Test Runner

Obviously, we require a tool that can execute functional tests in an automated fashion. Furthermore, there needs to be a way to trigger is execution at an appropriate stage in the CI process. Dependending upon the nature of the application you are developing, there are a number of choices. You can find a useful list of GUI testing tools here. The majority of projects that I have worked on in recent times have been web-based, and we have used Selenium extensively.

Selenium is open source, and allows you to test your app in multiple browsers. It also has bindings for lots of different programming languages, including a basic ‘dialect’ of HTML markup that is accessible to non-programmers (this type of thing is more formally referred to as a Domain Specific Language or DSL). It also comes with a browser plug-in that allows you to record and replay your tests. For the more technically-minded amongst you, I’ve already blogged in some detailed about how we use Selenium on Mike and included an example project that can be checked out from Google Code.

Test Environment

The third piece in our jigsaw is the environment within or against which the functional tests will run. Environments are often overlooked where automation is concerned, but it is very common to experience issues during CI that are a result of an inconsistent or poorly-configured environment. In my opinion, you should consider environments as a ‘first class concept’. They should be treated like any other artefact of the development process – configuration managed, reproducible and ideally, verified or unit tested.

In addition, keeping the development environments in a form which mirrors a ‘cut-down’ copy of production really helps when promoting your application through the various test environments it may encounter on its journey into the live estate. Hopefully this should help you avoid the “Well, it works on my machine….” discussion that frequently happens in dev shops where environments are poorly controlled and managed.

If we have automated the ability to provision or script the creation and tear down of an environment, then should not surprise you that we can harness this within our CI life-cycle. For example, with enterprise Java (JEE) projects you can use tools such as Cargo that allow you to provision, start, deploy-to and stop your application server of choice. Cargo supports a wide variety of server environments, from open source, such as Apache Tomcat, through to Oracle WebLogic.

Wiring it Together

Ok. If you’ve been paying attention at the back throughout this series then hopefully the following diagram should make sense. Let’s recap the steps 1 through 12 below that represent the essential elements of a CI process that can help your agile project to deliver a high-quality solution. Note that at any point past stage 3, the build may be set to fail, resulting in an email (or other e.g. IM) notification to be sent to your team.

Continuous Integration - A Summary of Steps

Continuous Integration - A Summary of Steps

  1. Developers work to transform the requirements or stories into source code using the programming language of choice.
  2. They periodically check-in (commit) their work into a version control system (VCS)
  3. The CI server is polling the VCS for changes. It initiates the build process when it encounters a change. The build is executed using a dedicated tool for the job such as Maven, Ant or Rake etc. Depending upon the language used, the source code may need to be compiled.
  4. Static analysis is performed on the source code, to ensure compliance with coding standards and to avoid common causes of bugs.
  5. Automated unit tests are executed.
  6. The percentage of the production code exercised by the unit tests is measured using a coverage analysis tool.
  7. A binary artefact package is created. At this point we might want to assist derivation and provenence by including some additional metadata with the artefact e.g. a build timestamp, or the source code repository revision that was used to produce it.
  8. Prepare for functional testing by setting up the test fixtures. For example, create the development database schema and populate it with some data.
  9. Prepare for functional testing by provisioning a test environment and deploying the built artefact.
  10. Functional tests are executed. Post-execution, tear down any fixtures or environment established in 8 and 9.
  11. Generate reports to display the relevant metrics for the build. E.g. How many tests passed? What is the number and severity of coding standard violations?
  12. The process is continuous of course! So rinse…and repeat….
Advertisements

Continuous Testing with Selenium and JBehave using Page Objects

May 6, 2010

Since Mike‘s inception we have always sought to automate as much of our testing as possible. For some time now we have been using Selenium for our functional/acceptance tests, and thus far have been very happy with this approach. Initially, we decided to use Selenese-style tests, as we felt this would enable non-programmers to help maintain and extend the coverage of our tests. Someone with a basic grasp of markup and enough ability to make small re-factorings to the HTML code generated by Selenium IDE. However, as the range of features provided by our platform has started to grow we have found that we are doing a lot of ctrl+c and ctrl+v of the Selenese scripts, and generally violating our DRY principles. After some internal debate, we finally settled on a new approach that adopts Behaviour Driven Development (BDD) techniques. This works nicely with our Agile, User Story based approach to development and (as you might expect) our advanced internal practices when using Continuous Integration for our own purposes.

BDD Framework Selection

The Mike codebase is predominately Java, so it seemed sensible to choose a Java-based BDD framework. We could have opted for something like the Ruby rockstars fave Cucumber, which is a now well-established, but in the end decided upon JBehave. It got our vote for a number of reasons:

  • It’s an active project, with regular production releases.
  • It’s ‘maven-ized’ and you can bind a handy JBehave plugin to appropriate life-cycle phases
  • It provides a web testing sub-component that gives you a nice, simple abstraction for Selenium
  • Scenarios can be defined in plain-text, just like Cucumber
  • It integrates into an IDE just like any other xUnit-based testing framework (right Click > Run As > JUnit Test)

The plain-text scenarios were of particular interest, as they allow non-programmers to continue to author test flows for each story. On the downside, it does mean that only developers can provide the implementation of these scenarios. But overall, it provides a good fit for our team profile.

An Example

I’ll walk through an example of a simple JBehave BDD-style scenario, that seeks to test that perennial fave – the Java Petstore web application:


 Scenario: Successful Login

 Given the user opens the home page
 When the user clicks the enter store link
 Then the store front page should be displayed
 When the user clicks the sign in link
 Then the store login page should be displayed
 When the user enters username j2ee
 And the user enters password j2ee
 And the user clicks the login button
 Then the store front page should be displayed for user ABC

This combination of ‘Given’, ‘When’ and ‘Then’ maps very nicely to a test context, event and expected outcome for each of the various pathways through a User Story.

So now that we have our scenario, stored in a text file name ‘login_scenarios’, using JBehave we need to create two additional classes. These are:

  1. a trivial subclass of org.jbehave.scenario.Scenario whose name maps to the associated text file (LoginScenarios.java)
  2. a subclass of org.jbehave.web.selenium.SeleniumSteps (LoginSteps.java) that provides an implementation for each of the ‘Given’, ‘When’ and ‘Then’ statements.

For example:


 @Given("the user opens the home page")
 public void theUserOpensTheHomePage(){
 	homePage = new HomePage(selenium, runner);
 	homePage.open("/jpetstore");
 }

Notice how JBehave uses simple annotations to map the scenario elements to Java methods. You’ll probably also notice the use of a ‘page’ object in the method body, which performs the actual heavy-lifting of the tests. In addition, the methods in the JBehave-provided base class SeleniumSteps can be overridden as required. For example, override createSelenium() if you need to provide a custom instance of Selenium with an alternative configuration.

Page Objects

Within a web application UI there are areas that our tests interact with. Using the Page Object pattern allows us to intuitively model these pages as objects within the test code. This massively reduces the amount of duplicated code and means that if the UI changes, the fixes need only be applied in one place. This means we get our DRY testing mojo back after our experiences with copy ‘n’ pasted Selenese markup. To make things even simpler, its a good idea to create an abstract base (Page.java) class that exposes a series of useful methods to its concrete children. Here’s an example of a class that represents the login page of our demo Java petstore app.


 public class StoreLoginPage extends Page {

	public StoreLoginPage(Selenium selenium, ConditionRunner runner) {
		super(selenium, runner);
	}
	
	@Override
	public void verifyPage() {
		textIsVisible("Please enter your username and password.");
	}
	
	public void verifyPresenceOfInvalidCredentialsErrorMessage(){
		textIsVisible("Invalid username or password. Signon failed.");
	}
	
	public void typeUsername(String username){
		type("//input[@name='username']", username);
	}
	
	public void typePassword(String password){
		type("//input[@name='password']", password);
	}
	
	public StoreFrontPage clickLoginButton(){
		clickButton("//input[@value='Login']");
		waitForPageToLoad();
		return new StoreFrontPage(selenium, runner);
	}
 }

Once these page classes are wired into our SeleniumSteps subclasses, we can use the action (open, click, type etc) or verification style methods to drive the tests.

Maven Magic

As mentioned earlier, we selected JBehave in part because of its Maven integration. There is plug-in you can configure to execute the tests during the required phase:


<project>
[...]
  <plugins>
  [...]
    <plugin>
	<groupId>org.jbehave</groupId>
	<artifactId>jbehave-maven-plugin</artifactId>
	<version>2.5.1</version>
	<executions>
		<execution>
			<id>run-scenarios-found</id>
			<phase>integration-test</phase>
			<configuration>
				<scenarioClassNames>
					<scenarioClassName>
						com.mikeci.jpetstore.test.scenario.LoginScenarios
        				</scenarioClassName>
				</scenarioClassNames>
				<classLoaderInjected>false</classLoaderInjected>
				<skip>false</skip>
				<ignoreFailure>false</ignoreFailure>
				<batch>true</batch>
				<scope>test</scope>
			</configuration>
			<goals>
				<goal>run-scenarios</goal>
			</goals>
		</execution>
	</executions>
	<dependencies>
		<dependency>
			<groupId>org.jbehave.web</groupId>
			<artifactId>jbehave-web-selenium</artifactId>
			<version>2.1.4</version>
		</dependency>
	</dependencies>
    </plugin>
  [...]
  </plugins>
[...]
</project>

Obviously, there are some additional services that we need when these scenarios are executed – the app must be deployed and available along with a running selenium server. In the petstore example, the Maven cargo and selenium plugins are used for these purposes. Have a look at the full unexpurgated pom.xml to see how they are configured.

Running The Example

Prerequisites are a Subversion client, Maven 2.2.x, Java 1.6.x and Firefox installed. Localhost ports 8080 and 4444 need to be unoccupied too.

Check out the project:


~$ svn co https://mikesamples.googlecode.com/svn/trunk/maven/jpetstore-maven-multi-module/ jpetstore
~$ cd jpetstore

Run the Maven build:


~$ mvn clean install -Pmikeci,run-functional-tests,run-on-desktop

Et voila, this should build the JPetstore web app, deploy it to Jetty, start Selenium and run two JBehave scenarios against it. You could use this example to bootstrap your own JBehave/Selenium implementation and get started with BDD in no time at all.