Testing and Mobile Web testing
In the last three decades, the world of software testing has been developed and advanced to the point where no one has any skepticism about its importance, especially today in the DEV/OPS era, where an application’s change is exposed almost immediately in a production environment. There are many methodologies and tools developed to support this segment, and today you can even find tools that are open source. Still, not all the tools are equal and not all the tools are designed for testing mobile web applications. Thus choosing the right testing tools is a crucial step for the entire project’s success.
The world of mobile web development is characterized by its richness of development frameworks e.g. JQM, ExtJS, Angular etc. So it is very important that your chosen testing tool will be part of this world, moreover it is very important that this testing tool will be able to execute your tests on both environments, the mobile emulator, and the real device.
What is CATJS?
CATJS is an open source automation framework for testing mobile web applications. CATJS executes tests on the real device as well on emulators. CATJS supports popular frameworks like JQM, ExtJS or Angular. For more information about CATJS and for installation instructions please visit http://catjsteam.github.io/ and/or https://github.com/ransnir/catjs-jqm-seed.
The AUT (Application Under Test)
The mobile web application I chose to test is called MyReminder, a Mobile Web Application written in Javascript (using JQuery and JQM). The application is supposed to manage your daily events with a very simple web based GUI. In this article I will focus on the Login Screen, and I will explain how to build very efficient tests in a very short time. (a link to the MyReminder app)
Building The Tests
Let’s analyze the login process (Figure 1).
The user is asked to provide his username and password. If he tries to login-in with an empty username, the login will fail and a message, “Missing Username”, will appear in the message bar. If he fills out the username, but leaves the password empty, the login will fail with “Invalid Username or wrong password” in the message bar. The user will get same response if he will fill out a username and password that do not exist in the system. When using correct credentials, the system will move to the next screen that shows the user’s event list. Now let’s think about testing, sounds simple doesn’t it? We need 3 different tests, one for each scenario.
Test 1: Login attempt with an empty username and password.
Test 2: Login with a username but with an empty password (we should use a name that exists in the system)
Test 3: Login with a username and password that do not exist in the system.
Test 4: Login with a username and password that exist in the system.
As a matter of fact, I can think of about at least 3 other test in addition, but we will stick with these 4 tests for now.
Firstly I would like to start with a quick guide about the CATJS test structure. A CATJS test called a scrap.
It is a comment inside the HTML/JS file (you can see an example in figure 2).
Let’s look into the scrap structure. We have the scrap body @[scrap ….]@ Which is surrounded by a comment <!– … –>. Inside the scrap body we have the @@name property, each scrap needs to have its own unique name, immediately after that we have the @@embed property. The @@embed property is a directive to CATJS that the scrap is embedded inside an HTML page, so it needs to be wrapped with the script’s tag. Of course, if the scrap is embedded within a JavaScript section the value needs to be false.
Immediately after that we can find the test commands. This time I won’t go through each of the available commands, however you can find the complete list in the links I have mentioned above. Let’s look at the @@jqm setText command. The first argument is the object id and the second argument is the text value that we want to set inside the object (in this case, the text field). In addition to the regular test commands, we have a special command for test validation, the assert command: @@assert ok( JavaScript verification statement, message); The first argument to the assert command is a JavaScript statement, if this statement is true the assert succeeds.
Now after we are done with the short CATJS scrap introduction let’s start designing our test cases. In the below figure you can see a flowchart diagram with the login flow. On the left side you can see a box for each planed test. Our first test is quite simple. In the login screen, we just need to click on the login button without filling up the username or the password. Then we need to verify that the correct message appears.
<!-- @[scrap @@name test_one @@embed true @@jqm setText("#username", ""); @@jqm setText("#password", ""); @@jqm clickButton(“#loginbtn”); @@assert ok($('#message').text()=="Missing username.”,”test one passed”); ]@—>
Let’s see what we have in the above scrap.
On lines 4 and 5 we set the username and the password to an empty string, we use the function setText with the object id as a selector. Immediately after that you can find the clickButton command which will performs a click on the login button. To complete the test the last line is an @@assert command, a verification point that tests if the desired message appears in the message bar.
Bases on the above test we can easily create the other tests.
<!-- @[scrap @@name test_two @@embed true @@jqm setText("#username", "jodoe"); @@jqm setText("#password", ""); @@jqm clickButton(“#loginbtn”); @@log test the login screen with empty password @@assert ok($('#message').text()=="invalid username or wrong password.”,”test two passed”); ]@—>
This time I fill the username field. In addition I also used the @@log command (line 8) which prints a message into the run log.
<!-- @[scrap @@name test_three @@embed true @@jqm setText("#username", "johndoe"); @@jqm setText("#password", "nopass"); @@jqm clickButton("#loginbtn"); @@assert ok($('#message').text()=="invalid username or wrong password.”,”test three passed”); ]@—>
The test populates the username and the password, however the user is not registered in the system thus the login should fail again.
<!-- @[scrap @@name test_four @@embed true @@jqm setText("#username", "admin"); @@jqm setText("#password", "iamadmin"); @@jqm clickButton("#loginbtn"); @@assert ok($.mobile.activePage.attr("id") =="reminders","login/logout success"); ]@—>
In the fourth test, I use the credentials of an existing user, thus the verification point needs to test if the application moved onto the next screen.
Running The Tests
Now all we need is to run the tests. To do so we need to build the CATJS project by running the
following command: catcli -init (see the figure below). Then follow the instructions and complete the configuration.
Now lets move into the newly created folder called cat-project and within this folder lets run the following command catcli —task t@autotest. This command scans the project source code, pickup all the scraps and automatically creates a CATJS scenario.
Let’s build and start the local test server with the following command: catcli -cbs
Within a web browser enter this address: http://localhost:8089.
The test will start running immediately and on the side you can see the test execution progress.
Technology Through My Eyes 
Seems like CatJS is more for Developer rather than Testers. As we are embedding test scripts with in actual code. Is there any way that actual code remain untouched and we can write scripts in separate files.
you are correct , CatJS come to change the existing dev life cycle and provide the ability to test early in the process and get fast feedback ( QA becoming lean and focusing on customers / UX / Security , etc ) .
you can remove the annotation once going to production ( aka @scraps ) or have it in a separate file ,
we know it sound dirty to write annotation / test code side by side your code ,but we found it very productive once code shifts between teams / Dev which can read the annotation and understand the flow and not just the code .