Automating Test Automation

512 0

During my 8 years in the QA & Testing Industry, first as a Test Automation engineer then a Technical Lead, Test Automation Engineers have a saying:

If it is done manually, then automate it

I feel this should apply to the manual creation of automated test scripts as well. Building automated tests can be tedious and repetitive at times when using a tool like ALM/UFT/BPT. At least in my experience. The constant drag drop of components representing the sequence of test steps is not particularly exciting stuff. Not to mention that is an activity that does not bring any value if you come to think about the reason why we create tests in the first place.

The ultimate goal of test automation is to increase efficiency on testing, and to find defects as early as possible in the software delivery lifecycle. All in all, composing an automated test is an activity that has no added value in itself.

I have experienced a variety of ways that automation tooling vendors have tried to automate test creation. Below I will discuss some pros and cons that I have found with them.

 

Cool style during 90s
Figure 1: Supposedly cool style during the 90s

Record/Replay in QTP/UFT

My first client in the automation field was using Mercucy QuickTest Professional (QTP) v9.5. By this point the functionality of WinRunner had been amalgamated in to QTP. The option to record user actions into an automatically generated script was my first exposure to automatic generation of automated scripts.

I am sure during the late 90s this must have looked amazing. However, this era also thought the picture on the left was cool, so I do question the judgement.

Pros

Quick to generate tests

 

Cons

  • Will record every action the user does on the AUT. This includes any unwanted mistakes performed by the user during recording
  • Absolute nightmare to maintain if technical changes happen
  • Complex interactions with the AUT can’t be recorded
  • Needs to be highly processed afterwards to add in data management, as the values used during the recording will be hard coded in to the test. To make the solution reusable these need to be parameterised.
  • Limited reusability. QTP Recordings could be placed in to Actions but there was not much flexibility with this system.

Thankfully, the project at the time had moved beyond this, and adopted a revolutionary concept at the time called Business Process Testing (BPT). It delivered an additional “business layer” between Quality Center and QTP in combination with VB scripting capabilities delivered through Mercury’s “descriptive programming” framework.

UI Screen Scrapers and Scanners

My next exposure to tooling, designed for helping automated test automation, was things like SAP TAO, and ITAS (a proprietary solution created by IOVIO). These tools were designed to aid with the creation of Business Process Tests within the QC/BPT/QTP framework. Creating BPTs can be a slow process as the user needs to add in components or iterations of components for every action they want the test to perform in the AUT.

When it came to filling in large forms within the AUT this can be a very tedious and frustrating process. To get around this issue, Screen Scanners were created. They can create Business Process Components that represent the screen in question, containing a parameter for each field on the screen (Text Boxes, Check Boxes, Lists etc…) but not controls that will move the user away from the screen, like Buttons.

The Components created for each screen would contain a Getter (for extracting information from fields), a Setter (for inputting information in to fields) and a Verifier (for checking the information contained within fields).

As this scanned component contains the ability to interact with every field on the screen, the automation user does not need to individually add components/iterations for each of the fields they want to interact with. They just add the screen component once and they are done.

Test automation: AUT Screens and Components
Figure 2: An example of an AUT Screen and the Components created for it

Pros:

  • Efficient. Avoids the constant addition of multiple Input components to fill in the fields on the screen. Without this scanned component, the Automation User would need to add 11 components, or iterations each time they had to include the screen shown in Figure 1 in to their test.
  • Maintainable. If the technical details for fields on this screen change after a patch, then there is 1 central place to make this change. This change can either be done by the project Technical Lead or can be done by an Automation Engineer by re-scanning the screen. Since every script accessing this screen should contain the same component, it means that all scripts will be updated by the one change.

Cons:

  • Even with Scanned components, the Automation User will still need to manually create the Test by adding components for Button interaction, Data Manipulation, and Screen Components. This takes time, especially for longer tests.
  • If there is a difference on the screen between two environments (SIT, PRE-Production), then two different sets of Screen Components would need to be maintained, on top of the two different versions of the Test.

Process Recorders

After screen recorders the next item on my journey was a revisit to the action recorder scene, where things had progressed extraordinarily far from the WinRunner days. SAP, capitalising on the popularity of SAP TAO, integrated a process recorder into SAP TAO called the Process Flow Analyzer (PFA). Creation of PFAs would also tie in to their SAP Solution Manager Business Process Blueprint, and allow for automatic impact assessments using the Business Process Change Analysis (BPCA) capabilities.

This tool was built to analyse the potential impact of introducing changes into a SAP systems and identifying any automated process tests that would require maintenance as a result of the change. In theory, this concept is very powerful, but was very difficult to setup and to operate, and sometimes results were not very accurate.

SAP TAO’s PFA recording feature was fairly straight forward. It allowed a non-technical user to start the recording and then follow their normal process steps within the SAP system. While some might complain about various issues with TAO, like BPCA not really working, or mandatory ALM integration, I am personally not a huge fan of the UI, which looks like someone went crazy with the VB6 UI Designer.

Pros

  • A recorder can automatically create the Test based on the User’s interactions with the AUT, instead of the User having to constantly move between the AUT and Test Repository. This can lead to a User creating a test within half the time it would usually take them using a different approach.
  • Using a recorder also means that the test creation can be assigned instead to Functional Experts, who know the processes being Tested, rather than being created by a specialist Automation team, who have limited knowledge of the process. This means that a smaller Automation team is required, as they will instead be focussed on finalisation of Tests and Maintenance, rather than the Creation step.
  • When using TAO for recording, the output from the PFA would automatically be converted in to a sequence of steps in the BPT format and uploaded to ALM.
  • The Data Table is automatically added, and the Test parameterised, which reduces the workload on the Automation Team.

Cons

  • Recorders will typically record every action that the user performs on the AUT. It means that if the User makes a mistake during the recording, but then corrects it afterwards, this mistake will be entered in to the recording. While PFA does allow the user to change the last recorded Action, the user has limited ability to fix anything recorded previously. This mistake will either require a new recording being created from scratch, or the Automation Team will need to remove the specific incorrect steps post-creation.
  • Complex interactions with the AUT, repetitive interactions with the AUT, or data manipulation steps cannot be recorded. Such as checking on the status of a Job within the AUT and moving on after the Job has Completed. This kind of interaction would need to be added post-recording by the Automation Team, usually by Custom Component creation by the Technical Lead.
  • The Automation Team will also still need to process the recording to parameterise the Data Input. Then create the Data Sets for use in future executions of the recorded test.
  • If something technical changes within a screen, the Technical Lead will still need to make this change within the code of the recordings during their maintenance. SAP tried to minimalize this Con with the BPCA. However, unless the project was fully utilising this ability, then this stayed a major annoyance with the approach.

The Future?

The IT world is a constantly evolving and improving area, as companies look to be the next big thing in their chosen field. Here I am going to discuss my next interest in the area of automating test automation.

Qualibrate aims to bring test automation to the next level. Not only: it automatically generates test documentation and training material. The recording process is smooth and the UI is very easy: with Qualibrate anyone can automate a test. It is also very easy to maintain each test.

Thanks for reading this blog, and I hope you found it informative and useful. I would like to add that I have not experienced during my career some of the other big automation software like TOSCA and TestComplete. If you have experience with these and would like to share it for other readers, please comment below.

Leave a Reply