Why did testing miss this… Because You didn’t set a risk appetite

We’ve all been on the receiving end of that one question sometime in our career… when a production incident or an escaped defect happens some manager says “why wasn’t this found in testing?” The assumption here being that testing was a safety net meant to catch everything and it failed, it’s implied that the failure is yours.

It isn’t a failure with testing, it’s a missing executive decision… what is our appetite for risk? Because as soon as this gets answered, we set a floor for quality that must be cleared and we can govern and build from.

This isn’t a testing capability problem (we need better testing), it’s usually that nobody has decided how much quality the business actually wanted and that decision rolls down for you to make. If you’re the one leading that team, it means a business risk that you’re accountable for is quietly being decided by whoever happens to be closest to the code that day.

As a thought exercise, think about your organisation and whether it’s actually set and written down any non-functional expectations:

• How secure should the code be? What level of vulnerability will you allow?
• How accessible should the designs and code be?
• How usable should workflows be, how easy to discover and troubleshoot for an average (or not average) user?
• How maintainable should code be, how inheritable and readable should it be to people?
• How fast should responses take and for how much data over how many requests?
• How expensive should it be to develop, run and maintain the product or features?

My guess is that one or many of these things haven’t been written down at an organisational or team level for your company, that’s my experience of working in and talking to other quality practitioners and engineers. Especially as we moved to Agile development and optimised more for speed, things like defining quality frequently got left behind. This means that there’s a gap that needs to be filled… how good should things be?

Plus this matters more now than it used to; a clearly set quality floor is the thing that lets you safely hand quality decisions to AI without drowning in slop… more on that later.

The business risk

If the exec team don’t set an appetite for risk (and therefore set a floor for quality) then they’re not owning the associated business risks:

• Legal non compliance (security/accessibility/regulatory) that results in an inability to trade in certain markets and fines.
• Costs and overhead that result from rework that comes from failure.
• Reputation and brand damage that results in loss of sales and customer churn.
• Trade offs for short term gain that impact business goals in the future.

These are substantial business risks associated to quality, they have an impact on finance and the ability for the organisation to be able to operate in markets. Without that exec decision the decision about risk acceptance rolls down into teams and individual contributors to make. That means, for every piece of work decisions are made by employees who don’t have the accountability for the risks associated.

Adding to that, a junior employee probably doesn’t know wider business strategy, legal compliance or what trade offs on cost and time can be made, but gets implicitly asked to make a decision impacting them every day. They end up having to make assumptions in a vacuum of information about risk appetite which can lead to the wrong thing and being asked “why didn’t you find this?”

The problem with quality advocacy

If a decision about how good something needs to be isn’t happening from the exec, it usually falls to quality professionals to advocate for it from the bottom. We end up having to try and educate and evangelise for a bar for quality that doesn’t exist… which puts us at odds with the business and our coworkers who see quality as “gold plating” and frequently tell us that our bar is too high.

Bottom up advocacy for quality, even when it comes linked to business risk, frequently fails because of other commitments; how often does testing get squashed into smaller and smaller timeframes because you have to hit a deadline of release? The top down mandate for speed or delivery usually gets made in absence of a similar mandate for quality, so teams (and managers) only have one thing to lean on for what to do… go faster and any discussions against that are derailing their goal.

You can try to reduce this by trying to advocate for the specific quality that you think will work for the organisation at that time, but the risk is that talking about any quality in a vacuum of an agreed risk appetite is that everyone thinks you’re evangelising for too much quality. I’ve worked in a number of teams where this is the case, people know what good could look like but are choosing to make the trade off on quality because they have a deadline. They didn’t want education or reports on failing quality signals, they wanted to release in a timeframe that met with management expectations.

I recommend: if you want to advocate for quality, be a trusted advisor and explain what trade offs you’re making on quality in terms of business risks “if we don’t test for these non functional requirements there’s the risk of legal non compliance, can we accept that?” Turn these trade offs into a risk acceptance decision point rather than a hypothetical discussion on speed vs. quality.

What do we do? Get the decision on risk appetite made

It’s the right thing to talk about quality trade offs as business risk acceptance, but doing it for every piece of work ticket by ticket is exhausting… and doesn’t really scale. You end up having the same conversation on every release and hoping the right person is in the room that day. What you actually want is the decision made once, by the people who own the risk, so the line already exists before the deadline panic kicks in.

So how do you get an exec to set the line? Here’s what’s worked for me.

Talk about risk, not quality

If you take one thing from this post, take this: stop asking for quality and start asking people to accept a named business risk.

Saying “We need more testing” is the kind of thing that sounds like evangelism, or too high of a bar for quality (in fact for most execs they assume quality is happening so will think you’re asking for polish and perfection). Whereas “There’s a possible legal compliance risk which could result in fines and not being able to trade in the market… are we happy accepting that?” is a decision someone actually has to own. Asking for a decision based on the things the execs already understand (business risks) helps them to get what’s needed and make an informed call.

Adopt existing processes

It also helps massively if you don’t try to invent a new process and instead just use one they already trust. Most organisations have already set a risk appetite somewhere (usually a security risk register). Someone will have already stood in front of leadership and got them to agree how much risk is fine and how much isn’t using a framework, so just adopt that. I was advised once before that leadership and execs don’t want to learn new things, they want to make decisions and move on… so rather than building a new shiny framework with new language, piggyback on what they’re already used to seeing and using.

Keep your ask small

Don’t ask for everything, it can be tempting to go through all the non functional quality attributes and say “how fast, how accessible, how good does this need to be?” but that won’t work because you’re asking too much all at once (including asking for understanding). Instead it’s better to wrap things up together so that you can make one smaller ask, how much risk appetite do you want to accept:

• High – We care about customer outages and legal non compliance that could result in fines and customer churn.
• Medium – We care about core feature behaviour including customer data errors that cause support issues and costs through rework.
• Low – We start to care about user journey experience and cosmetic optimisations that make our customers happy and want to stay with us.

From that one more understandable ask, we can drive an understanding and translate the testing and fixes needed to meet that risk appetite. This becomes a floor, for quality, a decision on what the minimum bar the organisation has agreed to meet; everything below it becomes a conscious choice rather than implicit or assumed.

Share the trade offs

You’re asking for a decision so need to let the decision makers know what the costs and benefits are for making that decision. It’s good to set that as if we pick a high risk appetite then we get X but we lose out on Y to allow for informed decision making to take place.

• High – We can release to market faster and protects us from legal non compliance but we might ship slower later or have to pay to rework things in the future, some customers might get upset.
• Medium – Slower to release and more costly but we please more customers (higher sales and reduced churn) and can support the product more cheaply in the future.
• Low – Much slower to the market and way more costly impacting new feature delivery, lower risk of reputational damage and minimal rework costs in the future.

We can always say that any trade off is just for now and can be adjusted as needed based on changing risk appetite.

A caveat: I know that in some organisations that testers won’t have access to the exec team or can easily influence at that level. It becomes our role to meaningfully escalate and influence the people who can get this decision made by helping them understand the risks and what’s needed.

What happens next is governance

If we get an organisational risk appetite set we can actually start to track quality risks against the agreed quality floor, using it to inform what our testing and defects we manage. We get to say “this is outside of the agreed risk appetite, so we need to test for it” and enter into discussions with the team backed by a mandate from the exec. Plus it gives us a view on the things we can ignore because a decision was made that supports a compromise of not doing this now to allow something to get shipped.

Discussions around quality and testing feel less like theoretical evangelism and trying to win people over… because we can point to an agreed floor for quality that the exec don’t want to dip below.

Stuff the team doesn’t want testing or bug fixing on that’s below the quality floor (outside of risk appetite) gets raised as exception back to exec for a steer on the trade off; asking are we happy to accept this risk in order to go faster?

Things within the risk appetite can be traded off for team speed or becomes a decision on building above that floor to making something even better. We can state that trade off which makes us as quality professionals seem pragmatic and part of the team’s goals of going fast.

I’ve found it good to report on any bugs / issues found as being inside and outside of risk appetite, as this shows whether testing has been pragmatic and focused and if we’re working to optimise for agreed quality as well as speed. As I’ve said before when it comes to quality engineering, the aim is to work to a state of good enough so that we can help teams to meet their goals of shipping quickly, working within risk appetite helps this to happen safely.

Reviewing and moving the risk appetite

The risk appetite being set gives us a floor and that floor might need to move based on data… maybe we over optimised on speed and it turns out customers are now unhappy. The organisation needs to track key quality signals and use this to inform changes needed to the quality bar.

• Sales numbers and what is quality stopping sales
• Customer churn, are people leaving due to quality
• Change failure rate (as a part of DORA metrics) for releases
• Slowdown of release cadence because quality makes shipping hard
• Legal and regulatory non compliance measures
• Incident and high severity issue numbers indicating quality failures

If these numbers start to show a problem and the business risk appetite changes, then a new decision needs to be made and the floor for quality has to change with it.

Conclusion

A frequent failing in how organisations handle quality is not giving the decision of how good something needs to be to the right owners; then the absence of a meaningful business decision cascades down resulting in testers filling the gap. This can mean the wrong trade offs happen and too much (or too little) quality happens, reducing the team seeing us as trusted advisors which hurts our ability to advocate for the right level of quality.

Organisations need to take the business decision of “how much risk related to quality are we willing to accept” and give it to the right people, the ones who have the authority to make that decision. That allows the whole organisation to operate with certainty for how much quality needs to be baked in.

With a quality floor and risk appetite in place, the next time someone asks “why wasn’t this found in testing?” there’s a real answer that isn’t a shrug or an apology… it’s because we agreed we didn’t need to look for it and made the trade off to deliver faster. That’s not a failure of the process, that’s a decision made by the people who own the business risk.

Thanks for reading. If this resonated (or if you think that I’ve got it wrong) there’s more to read on the blog… or get in touch via LinkedIn. I’ll talk quality to anyone who wants to listen!