In this section, I am going to go over specific working problems that you will face using Cascade. This section is very general, and will end up containing a great deal of random stuff. Specifically at this point, I would like to declare that my knowledge is getting better all the time (I hope), and I'm sure some of these solutions can be improved upon. At any rate, this section will always be a Work In Progress.
- Constructing The State Machine
- Customizing Cascade
Developers run different tests in different environments.
For example, you may have a two stage pipeline in CI. The first stage attempts to validate that the codebase is of sufficient quality to share amongst the development team. And the second phase attempts to validate that the code base is of sufficient quality to be considered a release candidate.
You may have a rule that says that developers must not commit code into CI unless the first stage passes. Timely results in this instance can become very important.
So how is this kind of thing normally done? Well usually, a certain group of tests run if some environment variable has a particular value, for example 'dev'. And another group of tests would run if the environment variable has the value of 'release'.
We will do pretty much the same thing.
So we configure two control files and we add the @Profile annotation to each. Each control file would be in its own file.
The default name is CASCADE_PROFILE but this can be overridden. If you set this as an environment variable, or hand it to the java jvm as a jvm argument, and its value matches the value supplied to the @Profile annotation, then the control file will execute its tests.
Running Tests Concurrently
So the tests are expensive to run? The first thing that will occur to most developers is to run the tests in parallel.
In Cascade, you would achieve this with the @Parallelize annotation. You add it to the control file and cascade will now run each test in a thread pool. The thread pool has so many threads as the value supplied to the annotation.
Running something like Selenium, the number of concurrent browsers it can drive at the same time is quite limited. However if you have even two tests running concurrently they can reduce your total test time by half.
Running in parallel isn't always possible though. If the subject system doesn't support the idea of partitioning data, then you won't have a state machine if you run concurrent tests, since your system will suffer from race conditions driven by the tests themselves.
You've decided that you want to run tests in parallel, but you need to make sure that your subject system remains a state machine when you do so. How can you guarantee this?
The key point is that you must be able to consider the subject system to logically be two separate systems. The data affected by the tests themselves must not affect each other. The two logical processes must not share common state.
An example where this is not possible would be if you are testing a poker game. Anyone who joins the game would affect the pot, and so each test would add a new player and would affect the other tests.
You could make this system compatible with Cascade by having two poker tables. Now the two players have no impact on each other, and consequently the tests driving each player are independent. You have effectively partitioned the state.
The method of partitioning state is very heavily driven by the subject system.
In the case of our Online Banking Example, the problem itself is very separable. Each user's online banking account shares no state with any other user. Even if a transaction were to take place between two accounts, the common practice is to have the funds move into a clearing system which is effectively a queue.
But does our code support parallelism? We might modify state after each test using the same user. In the case of our tests, that is the case. Our user is 'anne'. We would solve this problem by having each test use a different username. The example application places no restrictions on the username used. So now all we have to do is modify the login step to use a different username for each test. We can generate a random suffix for each username, but if you want the numbers to be contiguous then I'll outline the potential solution now.
The solution I'm presenting makes use of a static field shared between all tests. You would add this to the control file.
Then, in the Login step, you would Demand the counter field in all the scenarios that define the username.
Using partitioned data affects the cleanup process for each test. In the Online Banking Example, there is no cleanup, since I am using the same username each time. If I were to partition the data, I would look at introducing a remove-user end point and I would call it in the control file's Teardown method.
The Online Banking Example modifies surprisingly little state. More commonly, tests can modify a great deal of the state in the subject system. A common pattern is to drop the database after each test. You can't do this when you parallelize your tests. Cleaning up after tests can be very complex.
Saving Test Reports
Continuous Integration is a great concept for delivering quality code. Developers are enriching the CI concept all the time. Its very common practise to save the test reports after each test run. This isn't really optional because when the test run fails, developers need to be able to investigate the reasons why. The build server that supported the test run may no longer be available or even exist.
So Cascade generates a report in a form that can be saved. The reports are saved by default to ./build/reports/tests/cascade. When each test executes, a new directory is added to this location based on the system time. This is within the build folder so that it is cleaned up by default. This works for Gradle, but won't work for other build systems.
You can override the report output directory if you want. I do so in the Online Banking Example. I've included the code snippet here.
And so you would configure your CI server to save the contents of that directory.
Writing A Renderer
Cascade allows you to render the Expected Data Model within the test reports. This can be a very nice feature if you Supply and Demand all fields and you use those fields in all the assertions you make. If you do this, then Cascade can persist the Expected Data Model between each transition, so that the reports include a representation of the state at all points in the journey. It's great to have great reporting. I've already included a screenshot of this on the landing page, but here it is again.
But the fields shared between steps can be complex data structures and can render poorly, reducing the value of this facility. Cascade will use Jackson to render the field into JSON.
At this point, you would write your own renderer for the field. In the screenshot you can see the output of four renderers. There is one each for the notices, personal details, payment details and screenshot fields. The first three present nicely in html, while the last is an image.
So lets take a close look at an example. Lets write a StateRenderingStrategy.
This Renderer is a stock renderer that is included with Cascade and it renders a List of Strings as a bulleted list in html. You can see an example of its use in the notices field visible in the screenshot.
So, the interface defines two methods.
The first is the accept method which is used by Cascade when no specific renderer has been defined for a field. There is a global set of available renderers and each is passed the un-rendered field and based on the return value of this method, this renderer will render the field or not.
As you can see from the code in the example, this renderer will render a List of Strings.
As you can see from the example, the code is straightforward.
Taking screenshots is one of those things that developers love to do. At times the test run will fail and the error given will be that the 'expected element is not present'. The developer will look at the code and insist that the element is present! He will then look at the screenshot, if its present, or checkout the codebase and run the failing test, and discover that the preceding transition didn't work, and the page he expected to be on is not the page he is on. Screenshots can be very useful.
And they are visually impressive too.
I've included an example of 'taking a screenshot' in the screenshot you can see above in the section on how to write a Renderer. So I won't repeat that here. Also, taking a screenshot is a Selenium specific thing, so I haven't included one in the Cascade library itself.
So let me recreate that here. While doing so we also have an example of having a Handler work with a Renderer. So first let me show you the Handler.
So what is a Handler again? A handler is a class that runs at some point during each execution of a scenario. They either run before the When method, after the When method or after the Then method. You can see how to configure one by looking at the documentation on @StepHandler.
So we configure the TakeScreenShot Handler after every When method and that way we will have a screenshot of every page in the journey immediately after the transition.
There are some points of interest in this Handler. The first is that we have a field called screenshot which is a String. The Handler Supplies this field and specifies a Renderer while it does so. This will reference our ScreenshotStateRendering which I will describe in a moment. Then we have the handler method. This asks the web driver for a screenshot as a byte array. The byte array is persisted in a file stored in our reports directory. And we pass the file name into the screenshot field.
And then we move onto the Renderer.
The Renderer is only ever configured explicitly on the field using the @Supplies annotation. So the accepts method is never actually used, so it just returns true.
The render method is actually quite busy however. Let me let you in on a little secret. The html reports use Bootstrap, so feel free to use Bootstrap or JQuery in your Renderers. This implementation could have mererly written out the img tag in html and that would have been that. That would have included the images at their actual size in the scope pages. What is nicer however, is to instead have an image thumbnail and when you click on that, a modal dialog loads that displays the image at a larger size. And that's pretty much what this Renderer does.
So as you can see, the screenshot rendering example is actually quite nice. It incorporates a number of different elements of Cascade, including Handlers, Renderers, the use of static and normal fields, configuring a Renderer on a field and a bit of Bootstrap as well.
Constructing The State Machine
I've of half a mind not to include this section. The reason for this is that this section isn't really Cascade specific. Developers and and Testers run into these problems all the time and have been dealing with these problems for a while. I've opted to included some general guide lines and go light on code examples.
Exposing Test Endpoints
I'm anticipating much angst and gnashing of teeth by suggesting that production components should contain test end points (that are disabled in production). This is a sensitive issue.
The problems most often cited are:
The test endpoints are often very powerful. And they normally don't implement the authorisation logic that protects other endpoints.
The principle of encapsulation promotes the idea that different components share a contract and their implementations are independent of one another. This allows a system to change and adapt while keeping the impact of changes low. The same argument applies to understanding the system in the first place. Encapsulation makes complication simple.
Superfluous code deployed to production
Just in general, having code deployed into production that no one uses is considered bad form.
These are really great arguments but they don't pan out. I'll explain why now, but first I wish to make clear that every effort should be made to respect these reasons. I'm not advocating ignoring them.
Automated tests require a deterministic subject. Automated tests require that the subject starts in state A and, after performing some action, we end up at state B. State B is exhibited as assertions in the tests. These assertions don't adjust to correct but unpredictable behaviour as a human tester would.
Certain aspects of systems present problems when trying to make the subject system deterministic.
There are many business problems that assume some starting state in a database. The production system offers no way to set that starting state. It could be the amount a minimum bet costs in a poker game, the transactions in your bank account, the VAT rate for your accounting application. The lists are endless. During development, the product owner asks for functionality to be developed based on how badly he needs the functionality, and how much that functionality costs to develop. The functionality that is often not developed is the provisioning of starting state.
Some systems are naturally based on random numbers. Gambling applications are the obvious example.
Many business processes are based on date. The date affects data and how that data is presented. If dates are displayed to the nearest second, they you can have a race condition that becomes a significant issue for automated testing.
These are the kinds of processes that are defined by the business analyst saying something like 'If the user hasn't done something within 5 days...'. There are two aspects to this. The first is that jobs will trigger which will change state. The test cannot know that the job triggered, so it cannot know what any affected values should be. The other issue, is that in order to test a job that only triggers 5 days later, we cannot have a test that will wait for 5 days.
A multi process environment naturally infers a multi threaded environment. For instance, your test submits a web page and then wants to assert that the following page has content on it. But the server takes a while to respond. It might have to do some database persistence or something. The test makes the assertion immediately, but the following page hasn't been served yet. The result is an error.
So the point I'm trying to make, is that you cannot have automated tests without having a deterministic subject and if your system is not deterministic, you are faced with a choice: Either you don't have tests, or you make your system deterministic. In my view, the risk posed by invalid code naturally has to be more significant than any of the reasons I've cited, including security.
This is the one reason that I absolutely cannot argue with. Security is an extremely important issue and test endpoints tend to have godlike powers when it comes to affecting the state of the system. In a moment I will present some advice on how to expose test endpoints in test and not in production in a safe manner.
This isn't a valid argument to make. To understand why, lets go back to why we should have an encapsulated component in the first place.
The heart of encapsulation is the idea of a contract. Once the contract has been defined, a component can be thought of in terms of the contract, rather than its implementation. This makes things simple. Alternative components can replace the component if they conform to the same contract. If a contract changes, you can identify which components are affected by which components use the contract.
Practically speaking what this means is that developers can more easily understand complex systems by thinking in terms of simple components. By changing a contract, it is easy to identify which components need to be updated to the new contract because they explicitly use the contract. So contracts tell developers how big a change is by the degree to which it changes contracts, and it tells developers which changes need to occur at the same time. This affects what is released into production at a given point in time.
So this is fine as far as peer components is concerned, but does it sound like tests are peer components? No. They don't run in production. They don't form part of the working system. So if they are forced to conform to the contract, they aren't contributing to the goals of making the system easy to understand and to manage changes as components are upgraded into production.
It's easy to see why we would want tests to be encapsulated however, since encapsulation suggests easy-to-manage code. But if I'm faced with the proposition of having inferior tests for this reason, then I think the encapsulation argument fails.
Superfluous code deployed to production
Code that is not used, should be removed. However that is not the case with test endpoints. They clearly are used. Also, I should add that any codebase is dependent on third party libraries. These libraries offer many utility functions and many of those functions are never used by the subject system. And yet that code is routinely deployed to production.
So security is clearly the big worry. So lets discuss some ways to mitigate the security risks.
The first obvious solution is to have the codebase disable the test endpoints in production. You might write something like this.
This works great until someone sets the ENVIRONMENT property to prod in lowercase.
There are a number of ways to improve on this. The first is to test in order to enable the endpoints rather than disable the endpoints. Then if there is anything wrong in the configuration, the development environment breaks rather than production. This point might seem cheap and a little obvious, but I've personally seen this.
The second way to improve on this is to re-use the property that is loading the configuration file that contains configuration for the particular environment. The idea here is to fail fast if the property is not configured correctly as the wrong config file would be loaded.
Another way to protect yourself against this risk is to mitigate the risk by assuming that at some point this endpoint will be enabled in production, and let the infrastructure in production prevent its use. This can be easily done using a layer seven firewall. The important thing here, is that the test endpoints should have their own root. What I mean by this is that REST endpoints are addressed using URL syntax. Group all test end points under their own root, so that the layer seven firewall can have a simple rule for preventing their use.
The method I personally prefer is to build in the capability into your security model. That way you can cover the test endpoints under your usual authorisation system. The method is to have a Admin user that is defined in the database. A privilege is defined that enables use of the test endpoints and the Admin user has this privilege only in development. The problems with this solution is that not all subject applications have a database, or implement an authorisation model. The code to set this up requires some commitment from the product owner.
Externalising Non-Deterministic State
So lets suppose we have a system that exhibits all of these properties that make it non-deterministic.
Well, you could argue that maybe you haven't committed to the microservices concept enough. How about externalising the capabilities that are the source of these problems. Something more like the next diagram.
We can then use stubs in our tests to replace the dependent microservices. This idea may offer a solution to the problem.
But there are some issues we probably need to be honest about with this architecture. The first is that we have replaced method invocations with network calls. Depending on the actual problem you are working on, you may have significant issues with performance as a result. You can combat the performance problems by introducing caching. For example you could cache the date. But if you do introduce caching, you need to develop a way to disable the cache in testing.
The second issue is that you still face the problem of actually testing the code you have externalised. So this solution may work for the first microservice you wish to test, but eventually you face the same problem.
What I can say is that, performance problems aside, the microservices architecture presented here has quite a few other advantages. You can scale the subject system independently. Previously you may have had the problem of having a scheduler within the subject system that prevented you from running up multiple instances of the subject system, since it was stateful. Now it is no longer stateful. You can setup the scheduler independently using some active-passive pattern, while your subject system dynamically expands or contracts across computing hardware on demand.
You can use off-the-shelf software to support some of the functionality. Schedulers and TimeMachines are very common problems.
Synchronizing The Test Process
In the example code included with the Cascade sources, you will see a method call being made called waitForPage in all the When methods. What this method is doing is pausing Cascade until the next page has loaded.
This method uses specific Selenium code to test for the presence of the body tag in the current page that the browser has loaded.
But if you are not running Angular, you are back to square one. What I've done in those instances is wrap all my event callbacks in a wrapper function that sets a global flag while the subject function is busy. Something like the function I've included here. The busy function returns a function that wraps the management of the busy flag.
You would then interrogate the busy flag to determine if there is a event handler that is busy processing. I've included an example of how to do this using Selenium below.
The obvious downside is that every event handler needs to be wrapped. Web developers might argue that this pollutes their clean production code with a 'test artefact'. My view is that if it promotes testing and makes testing more robust and efficient, then I'm ok with that. It also sets a contract between the web developer and the test automation engineer, which means they get in each other's ways a lot less.
In this section I'm going to elaborate on some of the subtleties of modeling the state machine using Cascade.
Steps Following States Or Scenarios
Cascade is very flexible. I'm going to present two use cases to illustrate this.
We start by asking the patient's age and then their gender. This is our first use case. Age is something that belongs to everybody, and so is gender. So we can model these as steps and setup the relationship between them.
I've included examples of the step files here. We have the EnterAge step and the EnterGender step. There is nothing too surprising here. The point of interest is the @Step annotation. As you can see, the relationship simple. The EnterGender step is directly related to the EnterAge step.
But okay, what happens next?
We are on to our second use case. We ask the patient for their pregnancy status. Clearly this isn't something that relates to the male gender.
I've included an example here in the EnterPregnancyStatus step. If you take a look at it, you can see that the @Step annotation is now referencing the Female scenario rather than the EnterGender step.
Okay, so our journeys have split depending on gender. What about afterwards? The registration process then goes on to ask the patient whether they suffer from asthma. This is not gender specific. So somehow we need to route our journeys back again to the common step. The @Step annotation will list the Male scenario and the EnterPregnancyStatus step in this case.
It specifically will not specify the EnterGender step as we will then end up with two journeys for the girls. And since the subject system can only support one journey, the other journey will always error out.
So the point is that you can create relationships by using steps or scenarios.
So lets consider the example in the previous section, but with one alteration. In that example we asked the patient if they are pregnant immediately after asking whether they are female. This is convenient. What happens if the question is asked much further down the line? To illustrate, lets swap the positions of the pregnancy question and the asthmatic question.
Well doesn't that just throw the cat amongst the pigeons? What do we do now?
The solution is to use the @OnlyRunWith annotation.
How this annotation works, is that when the journey is first generated, Cascade will look through all the scenarios that are included and check if they have this annotation. If they do, the annotation will specify a condition which is then tested. If the condition fails, the journey is invalid and is not added to the final set of journeys.
So if you don't want a journey to test the pregnancy status if the patient's gender is male, then include a filter.
Sometimes examples of these can get quite involved. See the examples in the Online Banking Example in the Cascade sources. There I've setup a problem involving a user having different bank accounts. One step sets up a current account and a mortgage account, and a later step views the mortgage page. Other journeys have users who don't have mortgage accounts but having savings accounts instead.
Many applications will contain cycles. What I mean by this is that there will be a sequence of steps that link back into itself creating a cycle or loop of step relationships. Just consider the workflow below.
What we have here is a typical website with a home page. The user can do many different things, but they return to the home page after doing each action.
There are some special annotations that are very useful in these cases.
The first thing to keep in mind when modeling the state machine is that Cascade will only include each step once in any particular journey. This means that the length of a test is inherently limited. In the workflow example you can see that the Back step is included three times and the HomePage step is a central point in the state machine's structure. Cascade needs to know that it must treat the HomePage step and Back steps a little differently.
But first let analyse why we have these steps in the first place. Each step in Cascade may potentially have a When method, modelling a transition, and a Then method, which models a state. In the example given, the Login step would have a login action and the Then method would be absent. That logic would be placed in the HomePage. The reason for this is that immediately after logging in, you would want to assert that you are on the HomePage. You could include the Then method in the Login step, but it would be redundant. In this case there are many ways to get to the home page. Either the user can login, or as the result of a form post while modifying personal details, or clicking on the link while viewing data. After each transition, the state is the same, being on the home page. So the normal structure of steps is to have transition steps and state steps. The ability to include both in a step file is really an optimisation, since very often two steps will be exclusively linked.
So when modeling the state machine, it is necessary to have terminators. But in the case of cycles, these terminators have special characteristics. They should eventually terminate, rather than immediately terminate. So the way to do this is to mark the HomePage step with the @SoftTerminator annotation. This annotation tells Cascade that it should include the current step as many times as necessary, but when it runs out of steps, it should terminate here. There is an alternative annotation, the @ReEntrantTerminator annotation, which allows you to explicitly specify how many times the step is allowed in any journey.
But we still have a problem. Cascade will include a step only once by default in any journey or test. What this means in the above case, is that the Back step will limit the length of the test, so we will end up with three tests, rather than the one test we wanted. We need to tell Cascade that the Back step should be repeatable and we do that with the @Repeatable annotation.
I've tried to keep Cascade as extensible as possible. My first interest was in allowing for the possibility that the journey generation algorithms might manifest some fatal error. I've been over the algorithm quite a bit and I believe it is good, but allowing the users of Cascade to adjust that algorithm or even write an entire new way of generating tests, helps me sleep at night. That way, if there are any problems, then your development pipeline isn't at risk. Fix the issue yourselves and send me an email, or submit a merge request on github. I would be very appreciative.
Moving on from that (hopefully) remote possibility. There are some aspects of Cascade that could be extended on in their own right. The two most probable possibilities in my view is that you may want to generate alternative reports or specify new ways for finding step files. A report mechanism that outputs results to the console might be useful.
So lets go through a trivial example of specifying an alternative reporter. There is one already, and that is the reporter that disables reporting. It is basically an empty implementation. You specify it with the @Factory annotation.
Cascade uses the type information of any classes you provide to the @Factory annotation in order to replace the relevant component at runtime.
Explicitly Specifying Scenarios
Okay, so in the previous example we disabled reporting. This example is perhaps just a little too trivial.
Lets write a Scanner that explicitly specifies the scenarios to be used. This could be much more useful.
We configure this scanner in the usual way by using the @Factory annotation. This scanner will generate one journey.
And that's it. We have reached the end of the cookbook. I hope you find some of this stuff useful.