Why aren’t we Talking about Shift Right in Quality Engineering?

With there being more buzz and discussions around Quality Engineering, I’m seeing more people discussing shift left as part of what testers do. For anyone who doesn’t know, shift left is a term we give to testing earlier by reviewing ideas, stories and designs before we build anything.

But something that’s conspicuously missing in all these discussions, posts and talks is shift right. Why aren’t we talking about our post deployment testing so much as a part of Quality Engineering?

SPOILER ALERT: BECAUSE IT’S HARD.

What is Shift Right?

If shift left is testing earlier then shift right is testing later. The terms come from trying to help people understand where and when we can test relative to a “testing phase” on a timeline. We’re basically saying that testing can happen in more places than a test phase and can happen everywhere.

Shifting right testing encompasses all the things we can be doing at (and post) deployment into live environments. This includes things like:

• User testing, manually or tools based to track user behaviours and sentiment for what we have developed. Telling us how people use our product and which features they use (and how frequently they use them).
• A/B testing, deploying different options of a feature to learn which works best for users. Telling us which options for UX, workflow or front end design performs better with our customers.
• System monitoring, for health, faults and resource usage to understand how what we’ve built is performing. Telling us where undesirable (or desirable) system performance is and allowing us to track patterns and usage to predict future usage.
• Looking at reviews, comments online and net promotor scores to see what people are saying about the product. Telling us the things customers think and feel about the product that they might not tell us in person.
• DORA metrics and escaped defect numbers, to learn about the deployments we’ve made and how well they went. Telling us how our DevOps processes and ability to make and respond to changes in our product are going.
• Logging, to help with debugging and understanding how applications are being used (and how much things are used).

All these techniques and areas of testing are things that tell us how our product is used by real clients and on real systems. The aim is to create a feedback loop of actual usage so that we can reduce the abstraction of risk and look at the realities of usage.

Why it’s Important

Quick question: What’s one of the main points of Agile?

The point of Agile is quick and small deliveries allowing us to put things in front of customers and LEARN FROM THEM in order to improve our products. We shortcut requirements gathering and some design work because we know we’re going to put something in front of people to see what they like and then respond to that feedback.

Without shift right testing we’re not learning from production usage or our customers, which defeats the whole point of Agile development!

• A live deployment reduces the uncertainty of a test environment. We see how our software performs on representative hardware / cloud infrastructure as this may differ from what we have tested.
• Actual user data removes the uncertainty and guess work on thinking about what our customers actually like. There’s too many variations on opinions and thought processes to be able to guess and test for them all up front, so we need them to tell us what they think.
• The breadth of testing we get in production from real world usage hunts out more edge cases and scenario variations than we could ever feasibly test prior to deployment.

Frequently in testing we hear people talking about how Quality “is a measure of customer satisfaction” with a product or that only the end user can define and evaluate quality. But then we forget to actually push for the testing that gives us this insight!

It’s also worth remembering that a lot of what we do earlier in Agile development (especially in the quality space) hinges on the idea that we’ll learn from our users and respond to that with changes. User Stories are lightweight requirements that make assumptions on what good looks like in terms of a user need, assuming we’ll go out and validate that later. We cut down the time taken to design and test things in our Agile sprints assuming that we’ll test in production and make fixes as needed. In fact our Definitions of Done might be more lightweight in terms of what good enough quality is because we plan to test in production with users.

Our whole SDLC and development processes assume there’s a feedback loop that comes from our customers via shift right.

When I spoke to Nicola Sedgwick to learn about her Quality Radar she told me that the reason a whole quadrant of the radar is given over to observability & shift right is because teams are typically bad at implementing it. Not many teams know the art of the possible in this space, know where to get started or how to implement things; this tells us that it’s important that we talk about it as a part of Quality Engineering in order to start advocating for it and getting other engineers thinking about it.

Why we don’t Discuss it

I spoiled this up front, we tend to forget about (or don’t talk about) shift right because it’s hard.

• It’s a hard sell, taking up team effort and adding costs. Organisations and management may not want to implement it because it’ll be seen as slowing velocity and adding expenses.
• It’s relatively unknown, the ideas surrounding the implementation of observability and shift right aren’t widely known about by engineering teams and testers.
• Not knowing the art of the possible. Do a lot of Quality Engineers know about shifting right or do they think of it as something that we should be doing and advocating for?
• We don’t test in production!!!! Common testing wisdom (which arguably is outdated) stops us talking about how we test post-deployment. Any production testing is seen as risky and as something we shouldn’t do, even though that’s the point of Agile.
• Thinking that it’s not for us testers to talk about. At many organisations, post-deployment stuff is handed over to a DevOps or support team, which don’t usually include testers. Do we think it’s hard for us to break into influencing this?

It’s likely that where teams and organisations aren’t implementing a lot of shift right we just don’t have a lot of practical examples of implementation to share.

What can we do (What am I doing)?

We need to get more stuck in to advocating for shift right as a QE community to other Quality Engineer and to the wider software engineering community. This means both sharing the art of the possible as well as running and providing more practical workshops and information.

By talking in the community more about Shifting Right we can set the expectation that Quality Engineering also means testing in production to learn about quality there. This means as a community learning that we can (and should) test / explore / learn in production and not immediately responding with “no, we never do that” to the idea, based on outdated knowledge.

By advocating for shift right using tools like the Quality Radar we can show others the art of the possible and help them to understand the gaps in their processes, tooling and knowledge. Even starting to raise the topic with others can lead to an increase in knowledge and get more people talking about shift right, which in turn will create a pull for content online and at conferences. By sharing the value of shift right testing (including that learning from customer usage is the whole point of Agile delivery) to others we can sell the idea and make it less hard to get people to adopt this additional testing.

By sharing wins we can help others see how they can get started in this space. I’ve talked about my use of a Quality Radar in teams to show them what’s possible for their shift right testing. This (hopefully) will lead others to do the same and create more advocacy and understanding of what’s a complicated issue to solve.

Finally, by getting used to mentioning it whenever we talk about Quality Engineering we can create the wider understanding that shifting right falls within our area of expertise. Blogs, talks and conversations that reference shift left testing and testing earlier should (in my option) by default also make reference to shift right and testing post-deployment. Even if it’s just an additional throw away comment or reference to shift right, it’ll start to build an expectation that testing happens everywhere.

A note on shift left

This is an edit to the blog post based on feedback received.

Shift right testing should be used alongside shifting left! We’re not advocating to not do one instead of the other, instead I advocate for using both appropriately based on your context and needs.

Shifting left shortens feedback loops and de risks building the wrong thing; but it isn’t exhaustive, can be rooted in assumptions and can slow a team down. We can add shift right testing into our software development lifecycle to help fill in the gaps, but this is a much longer feedback loop.

The aim is to be pragmatic. We should not try to exhaustively shift left and ruin team velocity and time to market and instead set a view of what’s good enough for us to get started on building and releasing a product.

Thanks for taking the time to read! If you found this helpful and would like learn more, be sure to check out my other posts on the blog. You can also connect with me on LinkedIn for additional content, updates and discussions; I’d love to hear your thoughts and continue the conversation.