Thursday, July 25, 2013

Life of a Tester....tool evaluation pt 2: TestComplete


TestComplete Evaluation

Released by SmartBear
Installation of TestComplete was easy, smooth, and very professional looking.





I installed and opened TestComplete. The start page directed me without a lot of clutter.


I was able to navigate through TestComplete without any training from documentation or tutorials. The two main areas are the Project Explorer and the Workspace.

 
The following Project Explorer snapshot displays the test types that I have evaluated in TestComplete. Under the Advanced tab is the script area where the code level test cases are held. The KeywordTests is the GUI point and click record and playback test cases. NOTE: there is also an area to house our  Manual test cases.




The KeywordTests displays the recorded Item, Operation conducted by me, Value given, Description, and if there is an Auto-wait timeout for the line item. I was able to double click on each of these fields which then displayed a popup to alter the chosen item.


A cool feature I found is in the Workspace area. After I recorded a test, Script or KeywordTest, a window pane appeared at the bottom of the right hand side of the screen, displaying captured screens. I was able to double click them to enlarge and see exactly where I had clicked during the test.

 


Web testing, including cross browser
TestComplete is able to effortlessly record and execute tests in all browsers. After recording the test, I was able to go into the Workspace window, click on the Navigate line of the Keyword test and the browser selection window is displayed. This is really useful as I am able to record a test in one browser and execute the test in the same, or a different, browser.
 


Desktop testing
I was able to utilize TestComplete to record and execute tests for desktop applications.

Scripting
TestComplete has separate area where low level code scripts are located. I recorded a script through movements on a mouse and TestComplete compiled the steps into script format. (a script can be written from scratch, also, if desired.) TestComplete allows for scripts to be written in C#, C++, Delphi, Jscript, and VBscript.
 

Documentation and training; (somewhat) short learning curve
I downloaded the free trial, and was contacted by an Account Manager via email. He introduced himself as well as sent a link to a tutorial on getting started. I have been receiving tips via email since. The SmartBear website, as well as the testing community, has a plethora of video tutorials, hints, documentation, and Q&A boards.


Maintenance, longevity, reliability and support
Although I do not know the level of support from SmartBear, the testing community is very active on TestComplete questions, issues, tips, and tricks.

Error recovery
The logs are displayed in the Project Explorer window pane. I was able to double click on a log item and retrieve the line item details.



Reporting
TestComplete comes with a tool named ReportGenerator.exe, which creates a zip file of all the memory dump files (.dmp) given by TestComplete and constructs an html page listing all of the events.


Cost
A TestComplete node locked license is $1999 and a Floating User License is $4499.

Customizations (future proofness)
SmartBear seems to be doing routine updates to keep up with technology.

Established in market
TestComplete is very competitive in the test tool market.

Integration with Rally
Unknown.

Distributed Testing
The documentation states that TestComplete can execute distributed tests, but I did not evaluate this section.

Ability to run unit tests outside of environment
Unknown.


Monday, July 22, 2013

Life of a QA Tester....tool evaluation: Sahi Pro


Sahi Pro Evaluation
Released by Tyto Software
Installation of Sahi Pro was easy. I did find it a bit strange that it did not automatically install in the Program Files folder. The program is kicked off by a .bat file which launches the Sahi dashboard.



After choosing the browser that will be used, a front Sahi Pro page displays instructed me how to bring up the Controller.




Although the controller is laid out fairly well, I did find it a bit “clunky”.  I also had some issues running a script that had both a function that had been created through the GUI and another that was created from scratch.

Web testing, including cross browser
I was able to test in IE, Firefox, and Chrome.

Desktop testing
Sahi Pro does not test desktop applications.

Scripting
The language is based on JavaScript.

Documentation and training; (somewhat) short learning curve
The Sahi Pro website contains many pages of training materials.

Maintenance, longevity, reliability and support
Unknown.

Error recovery
I was able to see the logs of the tests run and upon clicking on a log.



After clicking on a test name in the Suite Name column, I was brought to a page where I could see where the script had failed (if applicable). I can then double click on the error, highlighted in red below, and will be taken to the place in the script where the test had failed.


Reporting
Sahi Pro stores its reports in a H2 database by default. Custom reporting is possible, but it may be necessary to store it into a central MySQL or another database. There is a MySQL .jar that can be utilized to create and customize reports. Also, this process requires a manual update, MySQL does not do this automatically.
Cost
$495/user/year

Customizations (future proofness)
Sahi Pro does not have this capability.

Established in market
Sahi Pro is not widely used .

Integration with Rally
Sahi Pro does not have this capability.

Distributed Testing
Sahi Pro can be distributed and run tests on multiple machines as long as the Sahi Pro application is installed on each machine.

Ability to run unit tests outside of environment
Sahi Pro does not have this capability.

Conclusion: We have decided that Sahi Pro is not a strong contender in our search of automated test tools and has been withdrawn from the choices.

7/25/13: Edited title- added "Sahi Pro".