When it comes to test automation tool, choosing the right provider is a critical component for success. It is not a decision to take lightly: it requires the entire team to ensure the solution is fit for purpose. First it’s important to compare all the solutions available on the market and rule out the solutions that don’t fit the teams vision. Creating a small pilot project with the three solutions that look the best fit may take a little time but gives valuable hands on experience in your environment easing the decision making process.
Is this selection exercise worth the effort? You wouldn’t buy a new car without a test drive so if your goal is to have a successful test automation team we would recommend putting the solution in the hands of the people who will use it to make sure it has the required features, makes their jobs easier, their time more effective and with the right tool it can even be fun.
Test automation: do I really need it?
According to The World Quality Report 2018-2019 by Capgemini, test automation is still under-exploited. Only 18% of functional test cases are generated with test automation tools. Among the companies which haven’t adopted test automation yet, there’s still a significant percentage of users who find it hard to realize the benefits.
Automation is recognized to be the best way to increase test coverage, to earlier detect defects and reuse test cases, to reduce test cycle time, support agile teams in continuous delivery and to reduce the overall cost of testing.
Is this true for each and every company? Obviously not. So, how can you tell if test automation is the right solution for you? There are few cases where test automation clearly makes sense: when you have many repetitive tests, frequent regression testing iterations, continuous delivery pipelines or when you need to simulate a large number of users.
If you have less than 2 releases per year, if the business process you want to test are few and the changes you make have a low impact on other processes, then manual testing should enough for you.
What to automate?
However, let’s say your team wants to release more functionality, more frequently. The danger is trying to automate everything and having too many unnecessary automated tests exponentially increasing the time required for maintaining and running them. Angie Jones, a famous Developer from the US specializing in automation strategies, has shared with us the “3 lessons from the test automation school of hard knocks”. Lesson #2: Less is more. Even Elon Musk recently admitted having abused an excessive level of automation.
How to determine if a test should be automated? Tests that only test functionality like a batch process that runs once a year wouldn’t make sense it automate. It would be more cost-effective to manually run that, rather than invest the time to develop and maintain an automated script.
ROI holding the power of balance
What matters in the end, in all QA activities, is the Return on Investment. Purchasing a test automation tool is a long-term investment that should reduce risk and increase the speed of your IT delivery.
So, before taking the decision you need to know what your ROI is and to help you we’ve created a ROI calculator which can guide you.
Test automation: prerequisite
Even if your calculated ROI is significantly high hold your horses: a great test automation tool itself won’t solve all your QA problems.
There are two prerequisites, too often underestimated, that you will need to adopt a test automation tool:
- Resources availability: do you have an approved budget for the tool? Do you have testers willing to spend some time trying the tool in a pilot project? Once you’ve made the choice, do you have people available to create and set up the test cases?That’s a one-time execution activity which, when implemented properly, will be the basis for a long-term successful QA strategy. It will also be easier to maintain the scripts if you create them properly.
- Commitment and company culture: if you don’t involve everyone in your team, the tool itself will be a waste of money. CTO, testers, and developers all need to embrace the change have an automation mindset Get the right people involved and excited about the new tool which, with the right approach, can revolutionize their daily activities.
Set your business goals
Most technology initiatives fail as they start on the wrong foot by deciding on the tool before deducing how it will be useful, test automation is not an exception to those initiatives unfortunately. The first objective is to understand what it is that we are trying to achieve by implementing this tool, where are the pain points for the end-user, business or project team that this tool will help circumvent.
As mentioned at the beginning identifying the right automation tool is not an easy task: there are so many tools to choose from, with different features, each of them having strengths and weaknesses. Not every tool suits your project needs, though.
First exercise: settle your own requirements.
Most tools have pros and cons. It all depends on your company, your IT systems and implementation methodology. Try to understand your needs first, write them down and choose what features are a “must have” or just a “nice to have”.
Here is a list of considerations to start drawing your boundaries:
- Project type (web/fat client/mobile)
- The scope of the project (CI CD/test coverage/…)
- Test automation and development skills within your team
- Choose which and how many test cases you want to automate
Key criteria to choose the best automation tool
The test automation market is getting bigger and bigger. Every day new players are entering the market, offering tools with thousands of functionalities. Tool scouting can be quite overwhelming and frustrating. Let’s prioritize: what are the main criteria to analyze?
These are 12 key points to consider before taking a decision:
- Ease of developing and maintaining the script. Make sure you choose a tool that let you create scripts and maintain them easily. That will save you a lot of time, especially when requirements change frequently, and you have to update scripts quite often.
- Ease of use for a non-technical users. A tool with an easy UI where no script is required for text execution is ideal for team members with no coding expertise. Functional experts are good examples of users who know the business processes inside out but are not coders.
- Which testing types does it support? Do you need to run unit tests, functional, regression, load, or maybe all of them? Choose the solution fitting your needs but be careful: don’t go for one just because it’s supporting all the testing types. Take only what you need: usually tools concentrating on just a few of them are more powerful and have more functionalities.
- Cross-browser testing. Do you have a browser-based system? On how many browsers do you want to test the application? That highly depends on the end-users and the browser they’re using to run the system.
- Technical support. There’s nothing more frustrating than a tool which doesn’t come with a good customer service. A trial or a pilot project can be the right time to understand how quick, friendly and expert the assistance is.
- Input test data. Sometimes you need to run very complex tests, so you need to upload your business data to cover many users and cases. How easy it is to provide it? A tool supporting test data input from various data files such as Excel, XML, txt, would be a big relief. Even better if you can connect it to different data sources.
- Test Report: reports are very important. At first glance you should be able to understand what the overall situation is, how many test cases are successful and the ones which failed. In case of failure you need a comprehensive report about it, telling you exactly on which step the script failed.
- Straight team conversations: how easy it is communicating within the team when some test fails? Prefer tools that support team conversations or consider to integrate with one of them, like Slack. Here is a nice article about ChatBots that can help you with this task.
- Integrations. The IT systems are becoming more and more complex, involving many highly integrated tools. It is very important to consider a solution which offers integrations or even an open API which let you build your own, in case your system is very complex.
- GDPR Compliant test data. According to The World Quality Report 2018-2019 the main pain point for companies using test automation is test data availability and compliance. Testers are often struggling to have relevant, valid and GDPR compliant data available for test cases. It is ideal to choose a tool including test data management functionalities or which integrates with one of them.
- Cloud-based. A cloud-based solution will prevent you to download heavy programs. You can access the latest version online and have your data safe, accessible any time on the cloud.
To help you comparing the different tools we have created a matrix summarizing all these criteria. You can download it and assign a score based on your needs.
Qualibrate ranking: let’s tick the boxes
Qualibrate is a SaaS solution that is quick to deploy, simple to use and easy to maintain. Qualibrate empowers your team to deliver faster.
All you need to do is record your Business Process which will become your unique source of truth. With this single centralized recording you can easily create regression tests, Business Process Documentation and even Interactive Training for end-users. Qualibrate also supports you accelerating manual testing expediting customers through User Acceptance Testing.
Let’s analyze Qualibrate based on the key criteria we’ve described above:
- Ease of developing and maintaining the script. With Qualibrate you can automate a Business Process in as little as 15 minutes (watch the demo here). All the steps recorded automatically have clear understandable descriptions, screenshots, are automatically parameterised and a default data set is created for you. Simple.
- Ease of use for a non-technical user. Qualibrate requires no coding skills to create automated scripts. That doesn’t mean that you can’t customize your script. With Qualibrate you are really flexible: it can be used both by technical and non-technical users.
- Which testing types does it support? Qualibrate supports unit, integration, functional and regression tests.
- Cross-browser testing. Qualibrate supports multiple browsers.
- Technical support. We are a team of QA experts also delivering services. We know the daily challenges of quality assurance, that is why we offer our expertise to help you start and succeed with test automation.
- Input test data. Qualibrate is a data-driven platform. The values you enter during recordings are captured for that step and act as an input parameter. You can easily view and modify these parameter values directly in the web platform.
You can also choose to provide a different source for the test data that can act as an input. Here are the options.
- Excel: Using the import functionality for data management you can import data from an excel file and create iterations that will run repeatedly to create the test data.
- API: Qualibrate has an open API (Application Programming Interface) that allows the test data to be populated for any test case.
- As part of the ongoing CD/CI(Continuous Delivery/Continuous Integration) method of working, Qualibrate can be integrated with tools like Attunity and other release orchestration tools. These tools can slice and dice data pulled from production and pass it on to Qualibrate using the API.
- Object recognition and mapping. The Qualibrate recorder automatically recognizes objects and picks the proper method to map them. You don’t need any technical knowledge to know which user action has to be used for which object, as this is determined by Qualibrate during recording. Once the recording is done, you have the option to change the user action and test data. You can choose the method you prefer by using ID, name, class, link and xpath.
- Test Report: dashboards within Qualibrate are easy and quick to understand.
- Team conversations: you can chat with your colleagues within Qualibrate for an easy collaboration. No more emails, just things done, quickly.
- Integrations. We have an open API that you can use to customize your own integration.
Using Qualibrate’s CLI (Command Line Interface), Qualibrate can be integrated with any tool. You can use it to invoke execution of test scenarios on any machine connected to Qualibrate.
- GDPR Compliant test data. Thanks to Gold Client integration we are able to provide scrambled and masked business data for SAP systems. Read about it here.
- Cloud-based. Qualibrate is cloud-based: no need to download it, just log in and access the latest version online. Your data will be safely stored on our online repository.
If you want to have extra information about Qualibrate don’t hesitate to contact us or to ask for a demo: you will be able to see how easy it is to use Qualibrate.
We hope this guide will save you time when looking for the right tool. What do you think about the key criteria we chose? Can you prioritize them? Write your ranking in the comments below.