What are the common elements for writing and running automated UI-driven functional tests?
I am a software developer, not a QA tester, so my interest is in writing some form of scripts to perform tests through a user interface; I am not into tools that record a user's UI interaction.
The question arose recently, while I sat through an introduction to the custom-built testing framework of a project I am working on. (first, a clarification: by "framework" for this article I mean the custom code that a development team needs to create in order for the automated testing system to interact in a meaningful way with the production code. I do not intend it to refer to the testing platform itself. I'll call a tool like jUnit or Selenium a tool or a platform, and the set of tests, test doubles, and other artefacts of testing a specific system I will call the Framework.)
My guide for the learning session had a tendency to do deep-dives into the low-level details. After several minutes, I pulled us into a higher-level discussion, so that I could map the home-brewed implementation to concepts I had already used on several previous projects.
I would suggest that there are at least 5 common elements that any functional testing framework will require of the devs who write tests and expand the framework. Since my Preaching class (see my bio!) taught me to work in memory hooks, I'll spell the acronym "STAMP":
#1 - Scripting
To create effective tests, we need some way to tell the computer what steps to take, where to go, what data or other resources to marshal, what ones to stub or mock.
In past projects, I have used Selenium to define the sequence of steps that the test would execute. In that respect, it was very similar to the coding I was doing for unit tests.
The home-built proprietary system I am learning also needs a way to define the sequence of steps a test must take. It uses CSV files (which we affectionately call the RECIPE CSVs). These are reusable across different tests, and define the sequence of screens that will be stepped through. If a test wants to skip a given screen, it has ways of indicating that.
#2 Test Cases
The more heavyweight a test script, the harder it is to maintain, and the less focused it is on a single purpose.
The challenge, especially with UI tests, is that they are much slower than code-centred unit tests. To get to screen X, you may need to spin up the whole system (or at least the front end) and step through screens A through N.
With so much effort, it's a shame to test only one thing, so there is sometimes pressure or a desire to test many things along the way.
I recommend keeping the test's purpose very focused, and to make other tests for other small sub-sets of behaviour.
To do that, you need the ability to easily create, distinguish, and run different tests for different test cases.
It seems obvious, but one whole category of file types in the system I was learning made little sense until I realized that they were essentially defining suites of individual tests. It was in these test lists that the tests were given a unique identifier, which would then be used in various other files to distinguish one test's data or configuration variations from another's.
In our case, these are also comma-separated files (LIST files) that the test platform knows how to decipher.
JUnit has its own definition of Test Case Classes and Test Suites. TestNG has its own testng.xml
#3 Asserting
In my daughter's education, her teachers are as concerned with the journey as the destination - the process of her thinking is as much a concern in their pedagogy as the "right answer". They don't like grading, and they especially don't like to use the language of failure.
When it comes to automated test suites, however, the "right answer" matters. So we need a way to identify to our test scripts what is expected, so that the test can catch and flag deviations, or fails.
We need the ability to assert what is our expected state, or output, or behaviour.
The most popular automated-testing platforms use variations of exactly that word. "Assert" shows up in jUnit, TestNG, or Ruby's MiniTest. Selenium WebDriver gives language like findElement, which can then be used with Assertions or Verify commands.
It turns out that the home-brewed system I was learning was geared specifically toward testing the impacts on an underlying data model and view caused by the steps of the test. All that was required was to add the verification key-word to the sequence of steps, along with an expression of data expectation, and the platform would cross-reference its state and data.
As I use it more, I appreciate its power and how targeted it is to the application domain. It is, however, much more complex to understand and use than the various flavours of Asserts.
#4 Mapping
As a developer, I am interested in writing code to define the tests (the Scripting discussed above). To do so effectively, I need some way to hook my test code into the elements, widgets, frames, whatever is on my UI.
In past projects, I have used Selenium with a couple different strategies. One is User Interface Mapping, in which a sub-system's element locators are stored in a common place. The test scripts then do not hard-code details of how to locate the element they want. Instead, they refer to the common mapping name. Then if/when the element's id or path changes, the mapping is changed once, rather than in dozens of scattered tests. For more information see here.
I have also used Selenium with a Page Object Model design pattern. It uses an Object-Oriented Page object to allow test code to interact with a given UI page. Any changes made to the actual page then only impact the Page object, not the dozens of tests. For more reading, see here.
In the proprietary home-grown tool, another kind of CSV file (our 3rd different kind, called DATA files) defines this mapping. Each column identifies a UI element of the page, and maps between the UI element's ID and the underlying data model's representation. It has also been extended, so that optional action columns can be associated with each element column, to give per-test influence over any UI-to-Server interactions when the elements are accessed by the tests.
I find this approach clever and potentially quite powerful, but at the cost of human readability. With practice, I am getting better at creating and maintaining them, but the need to scroll horizontally is a minor but persistent annoyance.
#5 Persistence
The final element a good code-driven UI testing framework needs is the ability to define data for the test to use.
Sometimes the data needs to be entered as input into the UI; sometimes the data needs to be in place for lookups or for the UI to act upon.
As with all other cases in the proprietary system I am learning, this is done through CSV files. One set of CSVs (our 4th different kind, call them LOAD CSVs) is used in populating an empty database schema with some minimum data, such as country codes, account types, process flows, etc. These define the base data available to all tests.
The DATA CSVs are then reused in the framework, to define the data details that a specific test is expected to enter on the screen that the same DATA file maps.
So the DATA CSVs therefore serve the double-duty of mapping the UI and specifying values used by a specific test. Each new row uses the test case's unique identifier, and the values relevant to the test's scenario.
I like how easy this makes it to see what data is being tested on a particular screen. At a glance I can see if a screen is well tested (or at least that it has lots of tests inserting data).
However I dislike that it is impossible to reuse data with this framework. If three tests all need to enter the same data on screen A and then variations on screen B, the data for screen A needs to be copied three times in the CSV.
There are other solutions to Persistence as well. I remember one project that used HypersonicDB to create an entirely in-memory database, and a sophisticated set of classes to populate the database with core data, and to allow tests to easily build up test-specific additional values.
On another project, the team's solution had been to define a fairly heavyweight core dataset that lived in a live SQL Server instance. It was maintained through a rather top-down process, and was pushed out regularly to individual workstations and servers for writing and running automated tests. Any changes like new data had to be proven on the dev's computer, then go up the ladder to ensure the appropriate data was added to the next release of the test-bed.
Links:
Selenium
Hypersonic
jUnit
TestNG
I am a software developer, not a QA tester, so my interest is in writing some form of scripts to perform tests through a user interface; I am not into tools that record a user's UI interaction.
The question arose recently, while I sat through an introduction to the custom-built testing framework of a project I am working on. (first, a clarification: by "framework" for this article I mean the custom code that a development team needs to create in order for the automated testing system to interact in a meaningful way with the production code. I do not intend it to refer to the testing platform itself. I'll call a tool like jUnit or Selenium a tool or a platform, and the set of tests, test doubles, and other artefacts of testing a specific system I will call the Framework.)
My guide for the learning session had a tendency to do deep-dives into the low-level details. After several minutes, I pulled us into a higher-level discussion, so that I could map the home-brewed implementation to concepts I had already used on several previous projects.
I would suggest that there are at least 5 common elements that any functional testing framework will require of the devs who write tests and expand the framework. Since my Preaching class (see my bio!) taught me to work in memory hooks, I'll spell the acronym "STAMP":
- S - Scripting: some way to define a sequence of actions
- T - Test Cases: the ability to define and differentiate any number of different test scenarios
- A - Asserting: an ability to compare an expected state or condition against the actual state or condition
- M - Mapping: an association between the screen elements and the test framework
- P - Persistence: a means of defining and injecting data with which the UI will interact, since UI state and functional behaviour will often depend on some notion of persistence
#1 - Scripting
To create effective tests, we need some way to tell the computer what steps to take, where to go, what data or other resources to marshal, what ones to stub or mock.
In past projects, I have used Selenium to define the sequence of steps that the test would execute. In that respect, it was very similar to the coding I was doing for unit tests.
The home-built proprietary system I am learning also needs a way to define the sequence of steps a test must take. It uses CSV files (which we affectionately call the RECIPE CSVs). These are reusable across different tests, and define the sequence of screens that will be stepped through. If a test wants to skip a given screen, it has ways of indicating that.
#2 Test Cases
The more heavyweight a test script, the harder it is to maintain, and the less focused it is on a single purpose.
The challenge, especially with UI tests, is that they are much slower than code-centred unit tests. To get to screen X, you may need to spin up the whole system (or at least the front end) and step through screens A through N.
With so much effort, it's a shame to test only one thing, so there is sometimes pressure or a desire to test many things along the way.
I recommend keeping the test's purpose very focused, and to make other tests for other small sub-sets of behaviour.
To do that, you need the ability to easily create, distinguish, and run different tests for different test cases.
It seems obvious, but one whole category of file types in the system I was learning made little sense until I realized that they were essentially defining suites of individual tests. It was in these test lists that the tests were given a unique identifier, which would then be used in various other files to distinguish one test's data or configuration variations from another's.
In our case, these are also comma-separated files (LIST files) that the test platform knows how to decipher.
JUnit has its own definition of Test Case Classes and Test Suites. TestNG has its own testng.xml
In my daughter's education, her teachers are as concerned with the journey as the destination - the process of her thinking is as much a concern in their pedagogy as the "right answer". They don't like grading, and they especially don't like to use the language of failure.
When it comes to automated test suites, however, the "right answer" matters. So we need a way to identify to our test scripts what is expected, so that the test can catch and flag deviations, or fails.
We need the ability to assert what is our expected state, or output, or behaviour.
The most popular automated-testing platforms use variations of exactly that word. "Assert" shows up in jUnit, TestNG, or Ruby's MiniTest. Selenium WebDriver gives language like findElement, which can then be used with Assertions or Verify commands.
It turns out that the home-brewed system I was learning was geared specifically toward testing the impacts on an underlying data model and view caused by the steps of the test. All that was required was to add the verification key-word to the sequence of steps, along with an expression of data expectation, and the platform would cross-reference its state and data.
As I use it more, I appreciate its power and how targeted it is to the application domain. It is, however, much more complex to understand and use than the various flavours of Asserts.
As a developer, I am interested in writing code to define the tests (the Scripting discussed above). To do so effectively, I need some way to hook my test code into the elements, widgets, frames, whatever is on my UI.
In past projects, I have used Selenium with a couple different strategies. One is User Interface Mapping, in which a sub-system's element locators are stored in a common place. The test scripts then do not hard-code details of how to locate the element they want. Instead, they refer to the common mapping name. Then if/when the element's id or path changes, the mapping is changed once, rather than in dozens of scattered tests. For more information see here.
I have also used Selenium with a Page Object Model design pattern. It uses an Object-Oriented Page object to allow test code to interact with a given UI page. Any changes made to the actual page then only impact the Page object, not the dozens of tests. For more reading, see here.
In the proprietary home-grown tool, another kind of CSV file (our 3rd different kind, called DATA files) defines this mapping. Each column identifies a UI element of the page, and maps between the UI element's ID and the underlying data model's representation. It has also been extended, so that optional action columns can be associated with each element column, to give per-test influence over any UI-to-Server interactions when the elements are accessed by the tests.
I find this approach clever and potentially quite powerful, but at the cost of human readability. With practice, I am getting better at creating and maintaining them, but the need to scroll horizontally is a minor but persistent annoyance.
The final element a good code-driven UI testing framework needs is the ability to define data for the test to use.
Sometimes the data needs to be entered as input into the UI; sometimes the data needs to be in place for lookups or for the UI to act upon.
As with all other cases in the proprietary system I am learning, this is done through CSV files. One set of CSVs (our 4th different kind, call them LOAD CSVs) is used in populating an empty database schema with some minimum data, such as country codes, account types, process flows, etc. These define the base data available to all tests.
The DATA CSVs are then reused in the framework, to define the data details that a specific test is expected to enter on the screen that the same DATA file maps.
So the DATA CSVs therefore serve the double-duty of mapping the UI and specifying values used by a specific test. Each new row uses the test case's unique identifier, and the values relevant to the test's scenario.
I like how easy this makes it to see what data is being tested on a particular screen. At a glance I can see if a screen is well tested (or at least that it has lots of tests inserting data).
However I dislike that it is impossible to reuse data with this framework. If three tests all need to enter the same data on screen A and then variations on screen B, the data for screen A needs to be copied three times in the CSV.
There are other solutions to Persistence as well. I remember one project that used HypersonicDB to create an entirely in-memory database, and a sophisticated set of classes to populate the database with core data, and to allow tests to easily build up test-specific additional values.
On another project, the team's solution had been to define a fairly heavyweight core dataset that lived in a live SQL Server instance. It was maintained through a rather top-down process, and was pushed out regularly to individual workstations and servers for writing and running automated tests. Any changes like new data had to be proven on the dev's computer, then go up the ladder to ensure the appropriate data was added to the next release of the test-bed.
Selenium
Hypersonic
jUnit
TestNG