Fundamental Test Process

Agenda
What is testing? Why is testing necessary?
Main testing principles
Key phases of Fundamental Test Process according to ISTQB standard

What is testing?
Why testing is necessary

What is Testing?
When we are testing something
we are checking whether it is OK

Why testing is necessary
People make mistakes

Why testing is necessary
We should assume our work contains mistakes

Why testing is necessary
Software is part of our life

Why testing is necessary
Some of the problems might be trivial, but others can be costly and damaging - with loss of money, time or business reputation - and even may result in injury or death

Why testing is necessary
Not all software systems carry the same level of risk and not all problems have the same impact when they occur

Typical Objective of Testing
For any given project, the objectives of testing may include:
To evaluate work products such as requirements, user stories, design, and code
To verify whether all specified requirements have been fulfilled
To validate whether the test object is complete and works as the users and other stakeholders expect
To build confidence in the level of quality of the test object
To prevent defects
To find failures and defects
To provide sufficient information to stakeholders
To reduce the level of risk of inadequate software quality
To comply with contractual, legal, or regulatory requirements or standards, and/or to verify the test object’s compliance with such requirements or standards

Causes of software defects
Human action that produces incorrect result
A flaw in a component or system that can cause the component or system to fail to perform its required function
Deviation from expected result

Defects, Root Causes and Effects

Cost of defect
The cost of finding and fixing defects rises considerably across the life cycle

Testing and Quality
Testing helps us to measure the quality of software in terms of:
the number of defects found,
the tests run,
and the system covered by the tests.

Software Quality – ISTQB definition
Quality: the degree to which a system, component, or process meets:
- specified requirements
- customer or user needs or expectations

Software Testing - ISTQB definition
Testing is the process consisting of:
all lifecycle activities,
both static and dynamic,
concerned with planning, preparation and evaluation of software products and related work products
to determine that software products satisfies specified requirements,
to demonstrate that they are fit for purpose,
and to detect defects.
Description of testing as a process
Objectives of the test process

Quality assurance- ISTQB definition
Quality assurance: Part of quality management focused on providing confidence that quality requirements will be fulfilled

Quality Assurance/ Quality Control

Testing Principles

Principle 1
Testing reduces
the probability of undiscovered defects remaining in the software but, even if no defects are found, testing
is not a proof of correctness

Principle 2
Exhaustive testing is impossible
Instead of exhaustive testing, use risks and priorities to focus testing efforts

Principle 3
Early testing saves
time and money
Find defects early on when they are cheap to find and fix

Principle 4
Defects cluster together
Review defects and failures in order to improve processes

Principle 5
Beware of the pesticide paradox
Updated or NEW test cases potentially find more defects
Test cases need to be regularly revised, new and different tests need to be written

Principle 6
European Headquarters – Lviv, Ukraine
Testing is context dependent
The higher the possibility of losses, the more we need to invest in testing

Principle 7
Absence-of-errors is a fallacy
The fact that no defects are outstanding is not a good reason to ship the software

Fundamental Test Process

Fundamental Test Process
A fundamental test process consists of the following main groups of activities:

Fundamental Test Process
Determining the scope, objectives, and risks of testing
Defining the overall approach of testing
Integrating and coordinating the test activities into the software lifecycle activities
Making decisions about what to test, the people and other resources required to perform the various test activities, and how test activities will be carried out
Scheduling of test analysis, design, implementation, execution, and evaluation activities, either on particular dates (e.g., in sequential development) or in the context of each iteration (e.g., in iterative development)
Selecting metrics for test monitoring and control
Budgeting for the test activities
Determining the level of detail and structure for test documentation (e.g., by providing templates or example documents)
Test planning

Fundamental Test Process
Gather information and provide feedback and visibility about test activities
Guiding or corrective actions taken as a result of information and metrics gathered and (possibly) reported.
Alternative version:
Measure / analyse results
Monitor / document progress
Provide information on testing
Initiate corrective actions
Test monitoring and control

Fundamental Test Process
Analyzing the test basis appropriate to the test level being considered
Evaluating the test basis and test items to identify defects of various types
Identifying features and sets of features to be tested
Defining and prioritizing test conditions for each feature based on analysis of the test basis
Capturing bi-directional traceability between each element of the test basis and the associated test conditions
Test analysis

Fundamental Test Process
Designing and prioritizing test cases and sets of test cases
Identifying necessary test data to support test conditions and test cases
Designing the test environment and identifying any required infrastructure and tools
Capturing bi-directional traceability between the test basis, test conditions, test cases, and test procedures
Test design

Fundamental Test Process
Developing and prioritizing test procedures, and, potentially, creating automated test scripts
Creating test suites from the test procedures and (if any) automated test scripts
Arranging the test suites within a test execution schedule in a way that results in efficient test execution
Building the test environment (including, potentially, test harnesses, service virtualization, simulators, and other infrastructure items) and verifying that everything needed has been set up correctly
Preparing test data and ensuring it is properly loaded in the test environment
Verifying and updating bi-directional traceability between the test basis, test conditions, test cases, test procedures, and test suites
Test implementation

Fundamental Test Process
Recording the IDs and versions of the test items or test object, test tool(s), and testware
Executing tests either manually or by using test execution tools
Comparing actual with expected results
Analyzing anomalies to establish their likely causes (e.g. failures may occur due to defects in the code, but false positives also may occur)
Reporting defects based on the failure observed
Logging the outcome of the test execution (e.g., pass, fail, blocked)
Repeating test activities either as a result of action taken for an anomaly, or as part of the planned testing (e.g., execution of a corrected test, confirmation testing, and/or regression testing)
Verifying and updating bi-directional traceability between the test basis, test conditions, test cases, test procedures, and test results
Test execution

Fundamental Test Process
Checking whether all defect reports are closed, entering change requests or product backlog items for any defects that remain unresolved at the end of test execution
Creating a test summary report to be communicated to stakeholders
Finalizing and archiving the test environment, the test data, the test infrastructure, and other testware for later reuse
Handing over the testware to the maintenance teams, other project teams, and/or other stakeholders who could benefit from its use
Analyzing lessons learned from the completed test activities to determine changes needed for future iterations, releases, and projects
Using the information gathered to improve test process maturity
Test completion

Revision History