“Success in automated functional testing is less about getting it right and more about avoiding mistakes”. Selection of the right functional testing tool is one of the crucial decisions you make for automated functional testing. If it goes wrong, you pay hard. Because every functional testing tool has some limitations you might overlook some hidden problems.
In selecting a tool, you have a set of selection constraints, which can be classified into hard and soft constraints. Hard constraints are fixed and must be satisfied in any case. These constraints include executing environments, programming languages, platforms, and framework used to build the application. The Soft constraints include cost, maintainability, usability, schedule, level of programming and technical skills required. These constraints can be compromised or partially met.
While selecting the tool, you need to balance both the constraints based on the project requirements. However, it’s challenging to satisfy all the constraints with one functional testing tool. To help you with that, I’ve discussed some of the important tool selection criteria and manual testing challenges. Also, I’ve listed the most popular functional testing tools.
Challenges in Software Testing
There are many challenges in software testing for which QA teams look for testing tools to automate the functional testing. Because human error occurs more frequently with manual testing, automation maximizes efficiency and streamlines projects for success. Manual testing is also time-consuming, taking up more of your team’s attention during work hours. It is most practical when test cases run only a few times without frequent repetition.
We’ve helped one of our clients to automate their manual testing process. The company helps organizations to improve sales execution and streamline productivity by providing sales enablement software. They were facing both business and technical challenges while developing the software.
- The product had about 7000 manual tests which were increasing cost because of long testing cycles.
- Software modules were going into production with an insufficient level of regression testing. It created an unacceptable level of risk.
- Frequent releases caused versioning issues that ultimately affected wide range of customers.
- Frequent releases caused versioning issues
- Testing was depended on inconsistent existing test data because creating new test data was too time-consuming.
- Browser compatibility issues against different browsers like Internet Explorer (8,9,10,11), Firefox, Chrome, Safari & Opera and platforms (Windows & MAC).
- Complex enterprise systems that include multiple integrated technologies.
- Inefficiencies among testing resources with varying levels of technical/coding skills.
- Complex enterprise systems that include multiple integrated technologies.
We collaborated with our client’s development team at various levels to address these challenges. We took whole ownership of test automation – strategy, framework set, script development/maintenance, continuous integration, execution environment, and delivery.
We identified the right automation tools, framework & solution for various technologies. We came up with a cost-effective solution with open source tools.
- Achieved almost 50% cost reduction of the overall QA process
- Reduced the testing cycle time from 10 days to just 2 days
- The automated functional testing tool has shortened the regression cycle testing cycle and enabled public service release of product builds.
Criteria for selecting a functional testing tool
While selecting the functional testing tool, it’s important to create a list of requirements for better the evaluation. Tool selection is often based on criteria that are less than optimal. So what are the right ones? Here are some key criteria which will help you to identify the best suited functional testing tool.
Programming Language and Development Environment
There are two approaches to work with automation tools in terms of programming language. Either use a powerful high-level language that is easy to learn, such as Ruby or write the same programming language used by the programmers.
In Continuous Integration run, programmers need to fix the bug when the test is written in the same language as the production code when commit gets fail. Better yet, programmers can save time if the tool could run as a plug-in inside the developer’s integrated development environment (IDE).
If the tool runs outside of the IDE and uses a different programming language, it is unlikely that the programmers will learn the new tool or do the work to support the tool when it reports “failures.”
Supported Platforms and Tagging
Make sure the tool supports all the platforms and levels the team supports (web, mobile web, iOS native, Android native). If it doesn’t support most common platforms then the team need to cover that risk in another way which will increase the cost. For such use cases, the tool can create a “page object” expressed as functions. The automated checks call those functions. It allows you to rerun the same test against a different page object because page objects are created at runtime.
Tagging is one way of tracking which tests run in which browsers. With tagging, you can instruct the tool to “run all tests for the edge browser full size,”. You can separate different tests like only back-end tests, only front-end tests, only tests that hit certain API, etc.
If the supports all the platforms, then the tool will need to track and run the subset that is supported. The ability to tag tests by feature can also be a powerful way to reduce risk while speeding up the delivery process.
Test reports document test results and their analysis. Therefore, select the functional testing tool whose dashboards and charts have a customizable degree of detail with respect to the intended audience (managers, customer, and developers).
Licensing and Support Costs
Consider the license cost as well as the costs that may pop up in the future (compatibility add-ons, support fees, etc.). The tool should do everything you need it to do without being a slow drain on the testing budget.
CI, DevOps Support
To avoid the automation delay, QA team run automation tool with CI process. Running the tests under CI creates a new requirement: The tests will need to be versioned with the code. That means the tool needs to run from the command line and produce output the CI system can interpret. For such requirements, your tool must be compatible with other CI tools.
Who writes the automation scripts in your company? Developers or test engineers?. You need to select the functional testing tools based on your team’s culture. In many, companies, developers write the automation scripts as they are more adept with programming. It saves the time of QA team while automation testing.
While selecting the functional testing tool, ask this question to yourself, “What are the critical bugs and where do they appear?”. You can get this answer with a bug tracker. That kind of research might find the majority of bugs in the business logic, the database layer, or the graphical user interface (GUI). For example, if the more important bugs are in the GUI, then test automation of the business logic through unit tests won’t add much value. It certainly won’t be the first place to start.
After selecting the tool, ask this question to know whether the tool can catch the most important bugs found both in test and production. If the answer is “probably not” or worse, then restart the tool selection process.
Functional Testing Tools
Here are the top functional testing tools that can help you meet your testing objectives perfectly. The list includes both open-source and commercial functional testing tools.
Unified Functional Testing (UFT) is a well-known commercial testing tool for functional testing. It provides a comprehensive feature set for API, web services, and GUI testing of desktop, web, and mobile applications across platforms. The tool has advanced image-based object recognition feature, reusable test components, and automated documentation. It’s a commercial tool so that automation testing is expensive due to licensing.
UFT is got the highest views and followers on IT Central Station in functional testing tools category. Here’s the review of UFT by Paul Grossman posted on IT Central Station.
“Our current project features more than a 1000 manual test cases, which took several days and resources to execute. Now, the suite executes in six hours and less than two when run on multiple machines.
With Jenkins connected, or the new cloud-based StormRunner Functional Testing, the tests can be launched anytime, even at the end of the day, and be ready for analysis the next morning.”
2. Tricentis Tosca
Simplifying end-to-end functional testing, Tosca automates tests across all layers of modern enterprise architectures. The layers are from API to web, mobile and custom/packaged app UIs to BI and data warehouses. It is designed to embrace and extend your existing tests while enabling continuous testing within your CI/CD process.
3. Sauce Labs
Sauce Labs provides the world’s largest secure and highly scalable automation cloud for testing desktop, mobile web, native, and hybrid applications. Sauce Labs helps companies accelerate software development cycles, improve application quality, and deploy with confidence across 450+ browser/OS/device/platform combinations.
Its GUI object recognition capability can automatically detect and update UI objects. It reduces the effort to maintain test scripts when the AUT is changed.
Selenium is possibly the most popular open-source automated testing suite for Web applications. Selenium is composed of several components with each performing a specific role in aiding the test automation.
Selenium supports multiple system environments (Windows, Mac, Linux) and browsers (Chrome, Firefox, IE, and Headless browsers). It also provides a test domain-specific language (Selenese) to write tests in programming languages, including Java, PHP, Groovy, Perl, C#, Python, Ruby, and Scala.
6. Worksoft Certify
Worksoft is a leading global provider of automation software for high-velocity business process testing and discovery. Enterprises worldwide use Worksoft intelligent automation to innovate faster, lower technology risk, reduce costs, improve quality, and deeply understand their real end-to-end business processes. Global 5000 companies across all industries choose Worksoft for high-speed process discovery and functional testing of digital, web, cloud, mobile, big data, and dozens of enterprise applications, including SAP, Oracle, and Salesforce.com.
7. Micro Focus LeanFT
LeanFT is a powerful yet lightweight functional test automation solution, that supports a wide range of AUT technologies. Targeted to technical test automation engineers and developers/testers in Agile teams, LeanFT is fully embedded in standard IDEs and integrates naturally with the Dev and QA ecosystems.
“The most important feature is the ability to handle complex algorithms, such as fuzzy logic techniques which is the first step towards artificial intelligence in our field.
By strengthening our understanding of our problems, LeanFT allows us to be able to define problems. From an executive level to the lowest level, every company needs to be able to understand the infrastructure of every single aspect of challenges that businesses face today. It has helped us be able evolve to change.”
8. Parasoft SOAtest
Parasoft SOAtest is widely recognized as the leading enterprise-grade solution for API testing and API integrity. Thoroughly test composite applications with robust support for REST and web services, plus an industry-leading 120+ protocols/message types.
9. Cross Browser Testing Tool
Ranorex is a quite comprehensive commercial automation tool for web, mobile, and desktop testing. The tool features advanced capabilities for GUI recognition, reusable test scripts, and record/playback.
The tool supports Selenium integration for web application testing. Testers can distribute the execution of their tests across platforms and browsers using Selenium Grid. Ranorex offers a low-pricing model for businesses.
To help you compare the best functional testing tools, IT Central Station has ranked Top 10 functional testing tools based on hundreds of real user reviews.
Here is the color coding for different criteria used in Bar of each functional testing tool.
The total ranking of a product, represented by the bar length, is based on a weighted aggregate score. The score is calculated as follows: The product with the highest count in each area gets the highest available score. (20 points for Reviews; 16 points for Views, Comparisons, and Followers.) Every other product gets assigned points based on its total in proportion to the #1 product in that area. For example, if a product has 80% of the number of reviews compared to the product with the most reviews then the product’s score for reviews would be 20% (weighting factor) * 80% = 16. For Average Rating, the maximum score is 32 points awarded linearly based on our rating scale of 1-10. If a product has fewer than ten reviews, the point contribution for Average Rating is reduced (one-third reduction in points for products with 5-9 reviews; two-thirds reduction for products with fewer than five reviews). Reviews that are more than 24 months old, as well as those written by resellers, are completely excluded from the ranking algorithm.
You can download the complete report from here: Top 10 Functional testing tools comparison by It Central Station.
SoapUI is a cross-platform functional automation testing tool. It has been designed to help test APIs such as SOAP and REST interfaces to ensure interoperability of different applications. So it is mainly used for WebAPI testing. The tests you write in SoapUI are very manageable. It provides drag and drop feature which makes test creation simple and easy.
14. Marathon Testing
Marathon is a cross-platform test automation framework for Java/Swin, Java/FX, and Web applications. It supports test automation through scripts. You can create readable scripts in ruby using its inbuilt script recorder. It provides a powerful integrated environment for managing test automation projects. There are two products Marathon and MarathonITE.
You can use Marathon for small applications(< 10 screens) and MarathonITE for complex/large applications. MarathonITE provides some advanced features like object map maintenance, extract-method refactoring, and data-driven testing.
15. Katalon Studio
Katalon Studio is a free Functional Test Automation IDE, designed for efficient and effective testing. It supports web, mobile and API testing and works on Windows, Linux, and Mac. It takes benefits of top two testing frameworks Selenium and Appium for integrated software automation. It’s easy-to-use and does not require great programming knowledge to write automated test cases.
In specific it addresses the challenge of lacking the integration among open-source test automation frameworks.
Appium is the most popular and open source tool for mobile app testing(both iOS and Android), mobile web, and hybrid applications. Appium supports multiple programming languages that have Selenium client libraries like- Java, PHP, Ruby, Python, C#, etc. The good part is you don’t need to modify the application to make it testable.
Watir is an open-source tool for automating web browsers. Though, watir is a family of Ruby libraries it supports any technology on which your app is built. It opens the browser as a normal way how the user opens it. Unlike Selenium, Watir does not have an official IDE for recording and editing test scripts. To use Watir Webdriver one needs to write test scripts by hand using Ruby.
18. Telerik Test Studio
Telerik Test Studio is an automated functional testing solution. Telerik Test Studio uses C# or VB.Net language whereas several other testing solutions impose their own proprietary language. It allows test scripts to be shared with other teammates via source control.
The platform offers a graphical interface, making it easier for developers to interact with developed applications. Testers can either create tests using record/playback functionality or the Telerik Test Studio API.
19. QA Wizard Pro
QA Wizard Pro automates the functional and regression testing of web, Windows, and Java applications. A tester can use the same set of functional test scripts for automation testing. It supports HTML 5, Ajax, .Net, Active X, Oracle Forms, Infragistics controls, QT, etc. It also supports cross-browser testing using the same set of scripts.
CUITe (Coded UI Test enhanced) is the automation testing tool which is included in Visio Studio 2010. Automation teams can develop sophisticated tests, using coded UI with the layered framework. It supports both webs as well as Windows projects. Coded UI tool supports descriptive programming. It allows software testers to automate scenarios based on object properties.
Rapise comes with unique Learn and Go testing method which is much more time efficient than traditional record-and-playback processes. Rapise provides powerful built-in support for all the environments like Java, Google Web Toolkit, Web applications including AJAX, etc. With Rapise you can use the same environment to test all these technologies. Also, it supports Chrome, Firefox and Internet Explorer.
TestDrive is an Automated Software Quality (ASQ) solution that addresses the requirements to achieve rapid automation. It is designed to test browser and legacy applications, as well as GUIs, including Java(tm), Ajax, Flex(r) and Silverlight(r). Its unique self-healing script technology allows users to run existing scripts over updated versions of the application. These scripts will update to incorporate changes according to preset criteria.
TestMaker 6 is the latest version of PushToTest’s next generation Ajax and Flex Web application testing framework. It repurposes a single test script to be a functional test (smoke tests, regression tests). It supports Web 1.0 application testing and Rich Internet Application (RIA, using Ajax, Flex/Flash) testing.
Maveryx Test Automation Framework is a functional and regression tool. It provides powerful and easy-to-use features for beginners (scriptless) and experts to automate a wide range of applications. Maveryx unique technology inspects the application’s UI at runtime as an expert human tester would, recognizing objects even if they have changed since test creation.
TestCraft is a codeless Selenium test automation platform. The revolutionary AI technology and unique visual modeling allow for faster test creation and execution while eliminating test maintenance overhead. Testers create fully automated test scenarios without coding. Customers find bugs faster, release more frequently, integrate with CI/CD and improve the overall quality of their digital products.
The Leapwork Automation Platform lets both technical specialists and non-developers design automation flows, without ever typing or reading a single line of code – and then execute cases with the click of a button.
Functionize uses AI and machine learning to create tests quickly and match tests to application development changes. Its test cloud allows testing team to perform functional testing within a single platform. It eliminates the need for QA teams to build and maintain their own testing infrastructure.
30. QARA Test
QARA (Quality Assurance with Rapid Automation) a product by The Digital Group is an integrated platform that supports both Manual and Automated Test Case Management. Its functionalities include – declarative, no-scripting, no-coding functional test automation design, management, planning, and execution along with built-in reporting capabilities. QARA also supports testing with AngularJS, BrowseEmAll, and BrowserStack giving users the ease of handling tests using just one platform.
As you can see, many of these automation tools have unique features to offer in addressing the growing challenges of automated testing. So take a look at the actual problems the team is trying to solve. Then find a tool that addresses those risks, works for the skillset the team has and integrates with the work process and technology stack – and give it a try.