QA Test Execution Maturity Model

Background

Definition: Testing a ticket is comprised of two processes, namely, designing test cases followed by executing them.

Disclaimer

This page focuses on defining the growth stages of a QA's capability to perform the second process (i.e. executing test cases) via a Test Execution Maturity Model.

Definition: A Test Execution Maturity Model (TEMM) is a roadmap for a feature QA in their journey of acquiring the skills and knowledge required to execute test cases reliably.

Goal: The goal of our TEMM is for STO QA to be able to execute test cases using a well-defined process that is repeatable, sharable and auditable. In the final phase, the QA should have sufficient skills and knowledge to explore testing more advanced topics, like performance, reliability, and security.

Our TEMM consists of four levels:

• Initial Phase – Level 1: QA can conduct manual testing
  • Keywords: UI, manual
• Reusable Phase – Level 2:  QA can apply a systematic way to conduct testing
  • Keywords: engineering, reusable, auditable, systematic
• Automated Phase – Level 3: QA can seamlessly convert daily feature testing artefacts into automated test suites
  • Keywords: tooling, automation, test-development
• Empowerment Phase – Level 4: QA can test more technical aspects/features and help developers reduce their number of bugs
  • Keywords: performance, reliability, tech-aspect, prevent

Diagram

Note

Before proceeding, we have to acknowledge that testing a backend system from its admin portal has inherent limitations. As long as this is the case within STO, we should be aware of this issue and actively find ways to mitigate our blindspots while testing.

Initial Phase – Level 1

Goal

• Able to conduct manual functional testing, given a test case design

Signal

• Start to design test cases based on UI operations
• Start to use UI operations to sign off a ticke

Metrics

• Understands the UI business logic
• Most of the tickets including UI and API tickets are tested by UI testing

Advantage

• None or minimal IT knowledge is required, no technical barrier
• Only need to learn business knowledge
• More convenient for performing end-to-end (E2E) testing

Disadvantage

• Prone to be blocked by unstable UIs
• Only able to test functional UI features unable to cover non-UI aspects
• Technical tickets cannot be adequately covered
• Difficult to leverage a systematic engineering way to conduct testing
• Test results are not auditable or trackable
• Test steps cannot be reused 

Reusable Phase – Level 2

Goal

• Able to utilise a popular tool that supports a reusable, high-efficient, auditable way to conduct testing (e.g. Postman)

Signal

• Start to get rid of using UI to conduct testing
• Start to design test cases based on the system’s complexity
• Start to share some test artefacts among the team
• Start to sign off a ticket with clear and proven automated generated test proof
• Start to reuse the testing effort from the previous testing
• Every effort considers maximum reusability

Metrics

• No more over limited API tickets signed off by UI
• Become an advanced user of Postman 
• Tickets signed off with clear, proven, automated generated test proof
• 90% of tickets signed off with an output of reusable test artefacts

Advantage

• Postman is a very popular tool that can help share the testing artefacts among the team/the organisation
• No longer blocked by unstable UIs
• Able to cover some edge cases as no longer restricted by UI

Disadvantage

• Need to spend time to do research on a tool and learn how to apply it in daily work
• A popular tool usually is a trade-off of easy-to-use VS. flexible 
• Some edge cases can’t be covered by a general tool
• Still not fully automated

Automated – Level 3

Goal

• Automation testing is an ability, not a responsibility; any QA in the team can be trained to be good enough to write code to test code
• Can quickly convert the testing effort to be part of automated regression test suites after sign off a ticket

Signals

• Start to conduct testing by writing code to test 
• Start to feel that it is more efficient to write code to test, feels using a tool/UI to test is in low productivities
• Start to see more and more feature tickets can be converted to AT effort earlier
• Start to have more time to focus on big features because more and more small tickets can be signed off by AT

Metrics

• Automation regression testing is something that a feature QA needs it, feature QA builds it, feature QA maintains it
• AT QA will focus on empowering engineering productivities of building more advanced tools for performance, security, code coverage, reliable testings, etc.
• Release should be more faster
• 50% tickets signed off/partially signed off by AT

Advantage

• Maximise the automated reuse of our feature testing artefacts: test execution by writing code to test code, the test execution effort can be seamlessly converted to AT suites earlier
• Integrate test execution with SDLC, export our AT_as_Service to developers; improve release efficiency 
• No more required a dedicated AT engineer help convert case; save efforts from sharing knowledge with AT engineer

Disadvantage

• Learning curve may be steep. Need to allocate time to develop these skills

Empowerment – Level 4

Goal

• Start to have time and skills to conduct more specific testing, like performance, reliability, and security testing

Signals

• Start to spend a big part of the daily effort in specific testing because most of feature testing should be covered by AT
• Start to find ‘naughty’ bugs like performance or security flaws
• Start to help developers reduce bugs (not just finding bugs) by conducting more insightful testing earlier during the development phase

Metrics

• Reduce performance, reliability, and security live bugs which can improve tech product’s overall quality
• AT QA provides advanced tools for performance, security, code coverage, reliable testings, etc.

Advantage

• Cover more system tech characteristics, e.g. performance, reliability, stress, security which are critical to tech products

Disadvantage

• Learning curve is steep and requires engineering experience

Conclusion

This TEMM is defined with a stackable concept in mind, to gear QAs towards quality engineering excellence

• Initial Phase: has some intuition about the business knowledge. E2E testing still relies on it 
• Reuse Phase: with some business knowledge, start to look for a reusable and engineering way to conduct testing. Daily feature experiencing/ad-hoc testing still rely on it
• Automated Phase: able to find a more robust and fully automated way to conduct testing
• Empowerment Phase: the scripting/skills accumulated from the Automated phase can empower the QA to perform more advanced testing