-----
All,
We have recently done some research on automated UI testing to perform our smoke tests (smoke test: press the main buttons in the application to see whether smoke comes out or not). Since you are in the same business I thought this would be interesting for you as well. We have browsed through an extensive list of UI test tools and looked at various criteria:
- It should be easy to create scripts: testers should be able to create the scripts (no programming experience required) and it should not take days to create a simple test script;
- Low maintenance: eg small modifications in the UI (such as the location of a button) should not lead to modifications of the script, a single change in the UI should not lead to a change in multiple scripts.
- It should be possible to run the scripts scheduled on a separate test machine;
- It should be possible to automate a Java application;
- It should be possible to generate a report of what has failed and succeeded;
- Low cost;
- Easy to create scripts: creating a script is a matter of drag and dropping actions and filling in parameters. Scripts and images can be re-used so the more scripts you create the easier it becomes. It is also possible to do “record and replay”, but this is less maintanable;
- Low maintenance: due to the fact that a swing application doesn’t use native code to create buttons we have too use images to search for a button (our scripts mainly consist of “click-on-image” actions). When the position of a visual component changes it can still be found by the search algorithm, so nothing has to be changed. Also, images are reused, so if the visual appearance of a component changes, you only have to change one image and all your scripts will be updated. Also since each script is built up in one or more actions it is very easy to remove an action or to add an action somewhere in the script to modify it according to a UI change (unlike record-and-replay scripts which become blackboxes);
- Run on separate test machine: The test scripts can be compiled into executables. We have already integrated this with our build server. As soon as the nightly build is finished the executables are started. Also, since the test machine has different hardware we ran into timing issues. VistaRunner provides many options to deal with this. You can add fixed delays between steps, but it also has actions like “wait for window to (dis)appear” or “wait for image to appear”. Of course, screen settings influence the image searching and should be the same, though we are still hoping for OCR capabilities;
- Test a java application: unfortunately we can’t use most of their actions which are based on windows controls but due to the image based actions we are still able to do everything we want;
- Reporting: it has some build in reporting functions, but it also does extensive logging. We have built a small program which starts the executables, parses the log and gives an overview of the result just like JUnit;
- Low cost: 1 license is less than 100 dollar
As a downside, it is a MS Windows application so you can’t use it on any platform (which is not a problem for us). A big plus, their support is at least as enthusiastic as Jide’s. On our request they have made several improvements on the image searching which is now very fast and even more easy to use. Btw, we are not stakeholders, we just think it is good product which could be interesting for you as well;-)
Regards,
Christiaan des Bouvrie
Cost Engineering Software
-----
Of course, we're curious if other people (incl. Jide) do automated UI testing and what product they use for it.
I've posted it on this forum since it is publicly accesable
Greetings,
Walter Laan
after tried it for a few minutes.