Understanding the things we need to test

When designing a strategy or approach for testing it’s easiest to start by looking at what it is we have to test. By this I don’t mean the product and features but the commonalities of the things under test.

All projects / products basically come down to having the same things that we need to test. Regardless of the specifics of architecture and feature set we can look at this breakdown to start designing our approach to testing.

Ideas

Everything starts with ideas, even potential bugs and problems. By pushing left and working with the business to challenge ideas and question them we can help bake in quality early.

Questioning ideas
Risk storming
Market research

Requirements / Stories / Artefacts

Ideas from the business are communicated into asks of engineering teams through some kind of documentation. Reviewing these to ensure there’s shared understanding and clarity can prevent assumptions being made and bugs being introduced.

Triforce / 3Amigos
Design / Architectural review
Risk analysis

Code

The lowest level of the final product that we test. Testing code as we write and deploy it (or at the time of other deployments) ensures that it works as expected. These low level tests are fast to run, supporting regression in a pipeline, but focus on engineering integrity and not business needs.

Engineering guidelines (linting and code standards)
Unit tests
TDD
PR reviews
Code review tooling

Features / Services

Code coming together to create logic or behaviour that needs testing. This might be with an engineering focus and can be at the business level (testing that services meet the business’ asks as given). This checks not only that how we’ve built something is correct but what we’ve built is correct too.

Requirements / Story testing (Functional tests)
Unit tests
BDD
Exploratory testing

Integrations

Services and applications coming together to share data or other interactions. This testing is for the parts of your overall product coming together (both internally and external 3rd party systems). Testing these separately to fill end to end testing is faster and cheaper to run in a pipeline and also more easily supports debugging.

Requirements / Story testing (Functional tests)
Integration tests
Contract testing
API testing
Exploratory testing

Workflows

The whole architecture coming together to perform desired outcomes. Rather than integrations that focus on point to point communication, this is where we test the whole. This testing is expensive and slow (as we need to have all services running and step through them all) so isn’t our best option for pipeline or regression tests.

Requirements / Story testing (Functional tests)
End to End tests
Manual AND automated testing
Exploratory testing

Business Needs

Theres always the possibility that we’ve assumed what the business wants, or that we need to confirm that our product will work in the real world. We need to test to confirm that what we’ve built (or are building) actually meets what the business actually wants, even if they’ve asked for something else. The earlier we do this testing, the shorter our feedback loops for finding out if we’re off course.

User testing / UAT
Bug bashes
A-B testing
Market research
Hallway testing
Demos
Non functional testing

Deployments

Moving code around might cause problems. What if we miss something, or move the wrong thing in and that causes breakages in our logic or behaviour? What if something we have no control over (like a 3rd party system) changes and breaks

Something? How do we check that our movement of code is safe and continues to be safe?

Pipelines / CI/CD testing
Observability (monitoring and logging)
Regression testing
Health checks

These are all things that need to be covered by testing. Some types of tests might cover more than one of these areas, which is fine so long as they give the appropriate context that’s needed. Teams need to ensure that they have a plan to cover all of these things, so that gaps are not left (ideally communicated in an approach with Who, Where, When and How).

I think talking about the WHAT of testing in this way is more useful than the HOW (like the testing pyramid does) because it gives more context. Many testers and engineers might know of the testing pyramid and can say types of tests but not understand the meaning of why we use them. By describing the meaning we can help them strategise their testing in better ways.