Cascade Testing

Cascade is a black-box testing framework. It composes states and transitions in units of code that are linked together. The cascade engine can then use these relationships to generate tests. And that's just the start...

Introducing Cascade

Cascade makes managing black box tests (also known as acceptance tests or end to end tests) easy.

The defining characteristic of these kinds of tests is the inability to set the subject system (the black box) into an arbitrary starting state. This means that we have to walk the subject through different states to arrive at the test point, and then we can test something.

This makes these kinds of tests very expensive. Expensive in terms of time to run, expensive in terms of effort to support them. They are also expensive in terms of technical risk, as they have to deal with asynchronous multi-threaded behaviours. The expense is often regarded as too heavy a price to bear, so practitioners choose to focus on other kinds of tests. This is the reason for the famous Test Pyramid. It is a pyramid shape because of the expense of these tests.

But there is a reason we wanted these tests in the first place! They treat the underlying system as a black box. This is encapsulation! Allowing the implementation to change while keeping the tests untouched, allow us to use the tests to validate our refactoring! Since they are defined at a much coarser level than unit tests, they are much more closely aligned with user requirements. This means they are potentially understandable by the User and so they are valuable to the User.

And besides, if they are expensive, maybe its because the tools we are using to run them are insufficient. How about better tools?

Introducing Cascade...

So what does Cascade do?

Cascade models the state machine of the subject system.

This means that the subject must be a state machine. 'Oho!' You exclaim; 'My application is not a state machine'. To which I will reply: 'you need to make it a state machine.' To which the attentive reader will comment; 'this is not a black box.' YES... Its true. Cascade cannot test any arbitrary subject system. The subject system needs to follow certain rules, which primarily make the subject system deterministic. Your Testers will be able to tell you that they are already doing this if you have any automated acceptance tests or end to end tests. Things like random numbers, dates, asynchronous jobs etc are all really bad for automated tests, since they affect the state that the test is trying to validate. These all need to be adjusted to make the system deterministic.

Okay, so lets assume your system is now a state machine. (See the cookbook section on how to achieve this.) This means that it has a finite set of states (web pages most often) and there are transitions from state to state (html links). Cascade lets you define Steps, which execute a transition (a When) and validate a state (a Then). Here is an example:

@Step(OpenLandingPage.class)
public class SuccessfulLogin {

    @Supplies
    private String username = "anne";

    @Supplies
    private String password = "other";

    @Demands
    private WebDriver webDriver;

    @When
    public void when() {
        enterText(webDriver, "[test-field-username]", username);
        enterText(webDriver, "[test-field-password]", password);
        click(webDriver, "[test-cta-signin]");
        waitForPage(webDriver);
    }

    @Then
    public void then(Throwable f) {
        assertNull(f);
        assertElementPresent(webDriver, "[test-form-challenge]");
    }
 }
An Example Step File

When you define a step, you also indicate the steps that precede it. Cascade can then generate a journey by generating a sequence of steps. The journey is then a single test. Once you have written the step files, you have modelled the state machine.

That's the core idea. But Cascade's advantages don't stop there. You have now modelled the state machine which is a really powerful concept. You can now use that model to do things:

  • Generate hundreds of tests from very few step files. (Perhaps not that valuable, but cool anyway.)
  • Run a minimal set of journeys so that each step runs at least once. (This is really the point of Cascade.)
  • Run only certain journeys that contain specific steps.
  • Generate test reports that graph the state machine.

Generating Tests

Cascade generates tests by permuting step files. Cascade does something a bit clever when it links these step files together.

A step file contains a When method which implements a Transition and a Then method which asserts a State.

But a step is also more than just the transition of state from one state to another. When you arrive at a web page, the web page is populated with data. The data affects the journey as much as the transitions between states. So through inheritance Cascade lets you defined multiple steps that occupy the same position in the state machine. I call these Scenarios.

A Scenario is an implementation of a step in the state machine that contributes data in addition to executing a transition and validating a State. This is done with a Given method.

The idiomatic way to define different scenarios in Cascade is to include them in the same file. This makes it easy to find all scenarios related to a particular state (webpage). This allows for easy maintenance. Here is an example.

@Step(SomePage.class)
public interface Notice {

    public class ShowOneNotice implements Notice {

        @Supplies
        private List notices = new ArrayList();

        @Demands
        public WebDriver webDriver;

        @Given
        public void given() {
            notices.add("Lorem ipsum dolor sit amet, consectetuer adipiscing elit.");
        }

        @When
        public void when() {
            click(webDriver, "[goto-notice-page]");
            waitForPage(webDriver);
        }

        @Then
        public void then() {
            assertElementDisplayed(webDriver, "[test-notice-page]");

            assertTextEquals(webDriver, "[test-text-notice-0]", notices.get(0));
            assertElementNotDisplayed(webDriver, "[test-text-notice-1]");
        }
    }

    public class ShowTwoNotices implements Notice {

        @Supplies
        private List notices = new ArrayList();

        @Demands
        private WebDriver webDriver;

        @Given
        public void given() {
            notices.add("Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do ...");
            notices.add("Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris...");
        }

        @When
        public void when() {
            click(webDriver, "[goto-notice-page]");
            waitForPage(webDriver);
        }

        @Then
        public void then() {
            assertElementDisplayed(webDriver, "[test-notice-page]");

            assertTextEquals(webDriver, "[test-text-notice-0]", notices.get(0));
            assertTextEquals(webDriver, "[test-text-notice-1]", notices.get(1));
        }
    }
}
Linking Step Files Together

So when Cascade executes, it creates journeys from all the Scenarios by permuting them. Potentially, by adding a single scenario, you have doubled the number of tests you can generate. Scary stuff!

Also, slightly besides the point for the moment, but notice how the Then method is referring to the data model when doing the assertions? This is idiomatic Cascade. This kind of thing is necessary when introducing cycles (loops) into the state machine. If you don't like it, then don't introduce cycles into your tests. (They aren't really that useful anyway.)

Given, When and Then methods are all optional.

If you want you can model each transition independently of the data. It gets a bit messy, and the idea of actions resulting in conclusions is a natural concept, so its nicer I think when they are included together. Its up to you. The graphing code won't render a pure transition differently from a pure state step. Something for later...

Completeness

So now you've defined 20 scenarios and you have thousands of tests running through every permutation you can think of... and your tests take hours to run... Is that really useful? No. Not really.

Cascade can generate a minimal set of tests that guarantees that it is complete in some sense.

Cascade defines the notion of Completeness that guarantees that the generated journeys include at least one of every step, depending on the Completeness level.

Here are the list of Completeness levels:

  • STATE_COMPLETE - the tests generated will include each State at least once. (A state is a vertex in the state machine - a web page for example.)
  • TRANSITION_COMPLETE - the tests generated will include each Transition at least once. (An edge in the state machine. - a link or form submission.)
  • SCENARIO_COMPLETE - the tests will include every Scenario at least once. (All steps are included which implies all data items.)
  • UNRESTRICTED - no attempt at minimalisation is attempted.

Naturally, the whole point of defining Completeness is to generate a set that is minimal while preserving the Completeness property.

Filtering

Okay, you've changed the logic on a particular web page. How about generating only the tests the run through that bit of logic? Or in this particular example, lets setup a standing order for later for a user that has only a current account? This is taken from the online banking example btw.

@FilterTests
Predicate filter = and(
    withStep(com.github.robindevilliers.onlinebankingexample.steps.Portfolio.CurrentAccountOnly.class),
    withStep(com.github.robindevilliers.onlinebankingexample.steps.SetupStandingOrder.SetupStandingOrderForLater.class)
);
Specifying a Filter

This snippet is included in the control file, which might look something like this.

@RunWith(CascadeRunner.class)
@Scan("com.github.robindevilliers.onlinebankingexample.steps")
@CompletenessLevel(Completeness.SCENARIO_COMPLETE)
public class OnlineBankingTests {

    @BeforeAll
    public static void before(List journeys){
    }

    @Setup
    public void setup(Journey journey) {
    }

    @Teardown
    public static void tearDown(Journey journey){
    }

    @AfterAll
    public static void after(List journeys){
    }
}
The Control File

This file is where we tell JUnit to run Cascade and do all the other stuff necessary to control the test suite as a whole. We also tell Cascade where to find step files here. More on this later.

The Control File tells JUnit to run Cascade and tells Cascade what to run.

Test Reports

And finally, since we have modelled the state machine, we can generate more detailed test reports.

Because we have modelled the state machine, and we have an idea of how often we run a particular Scenario, we can determine the Order of a Scenario. We can use the Order of a Scenario to emphasize its name based on its rarity in the test set.

The Order of a scenario is a measure of the Scenario's rarity within a given set of generated journeys. The lower the number, the more rare the Scenario is in the test set, and the more significant it is as a part of any test it is a part of.

We can drill down to each journey and see how each journey was constructed. We can see each Scenario's Order.

We can generate graphs showing the entire state machine.

And we can view the values of all variables in scope at any particular point in the journey, including screenshots. (See the cookbook on how to do this.)

And that concludes the introduction to Cascade.

So when should you use Cascade? Read the next chapter on Using Cascade!

Importing The Library

If you are ready to use Cascade, you can download the library off the Central Repository.

Here is the bit of XML you need for Maven.

<!-- https://mvnrepository.com/artifact/solutions.fluidity/cascade -->
<dependency>
	<groupId>solutions.fluidity</groupId>
	<artifactId>cascade</artifactId>
	<version>1.0</version>
</dependency>
A Maven Pom Fragment

And the Groovy for Gradle, if Gradle is your thing.

// https://mvnrepository.com/artifact/solutions.fluidity/cascade
compile group: 'solutions.fluidity', name: 'cascade', version: '1.0'
A Gradle Dependency

Downloading The Sources

And you can checkout the sources, if you wish, off my github page.