Start with the risks, not the method

Back in 2022 I wrote a test approach for a new team and it was rubbish.

I’d been talking for years about asking the WHAT of something before jumping to the HOW… it’s how I’d said to define acceptance criteria in Triforce, it’s the whole shape of my 5 Ws and an H approach, it’s what I say about exploratory testing when I tell people to look for risks first and then work out how to test them. But when I sat down to write a real approach for a real team I produced this:

• Helping drive out ACs in Triforce.
• Using exploratory testing and [these tools] to uncover information.
• Capturing test notes in [this tool].
• Raising bugs and managing them via Jira.

Every line of that is true but none of it is useful. What I’d actually written was “I will do testing at you” in a long document. It said nothing about the project, nothing about what might go wrong on it and nothing that anyone could use to understand what we’d derisk on the project. I could have handed that same page to any team in any company and it would have been equally accurate… and equally worthless.

I saw that this wasn’t something useful at all so I rewrote it starting from the risks instead, the second version was a completely different document. That’s the bit worth writing about, so let’s talk about why method-first documents go generic and what changes when you flip the order.

Why we default to the method (the HOW)

Because it’s EASY!

Method (the HOW) is comfortable and easy to knock out; we know the tools, we know the techniques, we can write that section without leaving our desk and it feels like progress because words are appearing on a page. Risk is the uncomfortable part because it speaks to uncertainty, it needs you to go and understand what’s happening, talk to people who know things you don’t and admit where you have gaps.

There’s also a template problem. If you search for a test approach template you’ll get a dozen documents with the same headings waiting to be filled in, which quietly tells you that the headings are the work. But a generic boilerplate template with your project name at the top is the same as taking a generic Dungeons and Dragons character sheet from the starter set and just focusing on the mechanics of how they play, technically valid but it tells you nothing about what you’d actually be doing in roleplay.

Here’s how you can tell if this was a useless test approach… could you take this document into a different project, not change anything and nobody would notice? If the answer is yes then all you’ve done is redocumented the basics of testing, not created an approach or strategy.

Starting with the risks (the WHAT)

In order to drive out something more meaningful I used the same approach that I use in exploratory testing, defining the risks and then looking for how we solve that risk. Starting by looking for and documenting the uncertainty and problems widens our net and thinking beyond starting with a limited set of tools.

1. What could go wrong here? Specific to this product, this architecture, this team, this deadline rather than “bugs might exist”.
2. How much do we care? What’s the risk to the business framed as likelihood and impact, weighed against what this organisation is actually willing to live with (is this good enough).
3. So what will we do about it? What are the techniques, the tool, the person, the timing.

It’s the same kind of process that something like risk storming uses. When I’m doing this, I don’t limit this to just testing or functionality, I try to be holistic in my thinking and I’m not afraid to point out things that other people in the team need to own or do.

The benefits of starting with the risks

More people can join in. Nobody outside engineering or testing probably has an opinion about whether you use Postman but I’m sure everybody has an opinion about whether it’s acceptable for the checkout to fall over during a launch. Framing your work as a risk puts it in the same language that the rest of the organisation already uses for decisions, which is how you get a meaningful decision rather than being told to go and test more.

You start shifting left early. Risks are available before code exists, defects aren’t. If you can name the risk at the beginning of the project then you can design ways to fix for it earlier (which is much cheaper); this is something I’ve written before in not waiting to be given a scope before you start testing.

You see the gaps. This is the one that’s really useful for a strategy or approach! Looking at the risks without filtering them through what you already know how to do means that you’re more likely to identify things that you have no idea how to test for. That’s not a failure, that’s the most valuable line in the document… it’s the gaps that you wouldn’t have covered otherwise. Maybe that’s a training need, a tooling spike or a hire into the project, which is made visible early enough to do something about. This beats out a method first approach document which probably wouldn’t surface this, because you only ever wrote down the things you were already planning to do.

You can surface new things. This approach allows us to actually address new things that we might not have heard of before or have tested. You might not have a tool or technique to hand for writing about AI and AI guardrails, but I bet you can raise the risks.

It sets context for AI. I’ve spoken a lot about AI and how it needs context to be meaningful, for example giving a basic prompt of “how do I test this” will give you a shallow and maybe meaningless set of tests that isn’t useful for the project. But if there’s meaningful risks associated to the prompt or a meaningful approach built from risks, then that sets the context of what you want to look for / what quality you want to build in. Setting the risks you want to work on (rather than just a method to follow) is a crucial guardrail and interface into AI.

The bit I actually got wrong

Looking back at 2022 the mistake wasn’t that I’d forgotten the model of WHAT then HOW, I knew the model, I’d written the model down and told other people to use the model. What happened was that I was busy and using a template was faster, so I optimised for producing a document instead of producing an understanding.

Being senior doesn’t protect you from that, it just means you assume that fewer people are going to check your work. Which is probably the real argument for writing your process down somewhere public, so that you can actually get engagement on what you’re planning to do.

Thanks for reading. If this resonated (or if you think that I’ve got it wrong) there’s more to read on the blog… or get in touch via LinkedIn. I’ll talk quality to anyone who wants to listen!