In dungeons and dragons (or any role playing game) we play as a group of people that all bring different skills to the table. But sometimes an adventuring party will have gaps in the skills they need and have to bring in a mercenary to fill the gaps, something that we might do in testing by using AI agents.
Let’s look at an example together… a group made up of only bards on an adventure, might struggle to face all of the challenges thrown at them.
Whilst these charismatic wordsmiths can easily win people over, their gaps in magic knowledge and lower fighting skills makes their adventure harder. The bards didn’t ignore these obstacles (I’m pretty sure that in dungeons and dragons people expect to fight dragons), but they probably didn’t consider all the challenges they would face when they were putting their party together.
Engineering teams can do the same thing, being stacked full of people with the same skills, which means they can’t solve for certain challenges when they come up.
How does this happen?
I’ve worked with teams of testers (like I’m sure we all have) whose skills as a team were awesome in one specific area, but weren’t generalists; they excelled at checking that requirements had been met but couldn’t advocate for quality, check NFRs, report with quality narratives or add tests to pipelines. The testers themselves weren’t the problem, they were doing exactly what they were brought in to do, but a lack of diverse skillsets in the team meant we couldn’t cover the breadth of quality that was needed.
culture fit vs. Culture add
In our dungeons and dragons game a bard might think, “yes I have awesome skills and I need more of that” because they haven’t played as other classes and can’t see what else is on offer. After all, they’ve solved every problem so far through charisma, careful negotiation and seducing absolutely everyone (yes… trying to seduce everything that moves is a well known dungeons and dragons meme). They tell their friends to play a bard too and we end up with a party of all bards, a monoculture of the same specialism.
We can find that a hiring manager might only look for a limited set of skills… probably informed by what they knew as testing. Someone whose full experience of testing is limited to functional testing might consider that as everything that testing can be and doesn’t look at what else could be needed. The idea of culture fit, bringing in people that match what we do, rather than looking for culture add, filling the gaps we have with new skills is very prevalent in hiring.
I’ve seen this play out in interviews. I’ve been a candidate where I could see testing culture add gaps and advised the recruiter to look for skills that they probably didn’t even know that they needed.
Developers as testers
This is similar to the culture fit issue we saw. In engineering teams most of the successes have come through engineering so we hire for more of that, sometimes neglecting the idea of new challenges that come our way. That’s when we end up with teams of all engineers or developers who have to pick up the testing themselves.
As I’ve talked about before, testing is not a solved problem in engineering, with teams not knowing the art of the possible in what testing they can do (or what’s needed). People just don’t know that other skills might be needed to solve problems, similarly to a dungeons and dragons player not knowing what other spells or skills might be available to other types of player class.
Having one tester
In dungeons and dragons most players will tend to focus on one class, rather than multi classing. This is a very valid way to build a character, but it means that someone is a specialist rather than a generalist with a wide array of skills (in fact the game prohibits you from being proficient at every class).
Plenty of teams only have a single testing specialist, if they’ve specialised in an area rather than being a generalist then we get the same problem. Someone who doesn’t have the skills to deal with (or awareness of) a wide array of testing challenges leaves us with gaps.
The usual answer to this problem is to grow our testers by multi classing them into the skills of other classes, which builds lasting skill… but this takes time and sometimes the dragon is already at the door / eating our maidens / burning down our village.
Can an AI agent be the mercenary?
So what do we do when we realise our team is all bards? Well it’s okay (it happens to most of us) because we’ve always had ways of filling those gaps, we can hire for what’s missing, grow our own people by multi classing them and now there’s a new option… bringing in a mercenary in the form of an AI agent to give us the culture add we need.
In D&D a mercenary is someone we bring into the party for a short time to cover a skill we lack, which gets us to solve a specific challenge by paying some gold to get the job done now rather than having to level up and multi class ourselves.
Let’s expand on our previous example, the adventuring party of bards have gone into town and hired a wizard and a barbarian as a short term mercenary.
With these different skills in their party, the adventure is a success with all of the hurdles overcome and treasure / experience points for everyone! If you’re reading this and you play dungeons and dragons tell your dungeon master that I said you could level up!
We can think of an AI agent as a mercenary that we can hire into our team on demand for a cost of a subscription / tokens. We can use it to fill in the gaps that our testing and engineering teams have and bring in skills that we otherwise don’t have.
That’s what my QE skills do… the three-amigos skill reviews a story for testability to bring in shift left testing, exploratory-testing turns acceptance criteria into scored risks and test charters and the Playwright skill writes automated end to end tests. I’ve also used agents to review and explain code through static analysis to report on test coverage and score accessibility, security and maintainability.
Engineers that I’ve worked with have used these skills as the QE or testing mercenary in their teams to help them to understand quality and plug gaps in their skills. Testers that I’ve worked with have enjoyed using the Playwright and static analysis skills to help expand out into code based quality engineering and automation testing.
Hiring the wrong mercenary can ruin our adventure and make our challenges worse. We need to have an idea up front about what types of skills we’re going to need so that we can bring in the right party member for our adventure.
How do we know which gap to fill?
Brute force it (at a cost)
Why don’t adventuring parties just hire mercenaries that cover all the dungeons and dragons classes every time they adventure? Then they’d have the skills to solve any problem that comes along, right?
One word… treasure.
Players in dungeons and dragons love treasure and seldom like paying out their hard earned gold on frivolous purchases. I’ve yet to meet a party who wanted to spend all of their gold on mercenaries when they could spend it on other cool stuff like magic weapons.
Likewise in our engineering teams we could just build AI tools and skills that cover every possible quality and testing gap that exists, but that’d cost us. We’d be spending a lot of tokens and time to do this in order to solve every challenge and potentially improve our software. So long as we know what every challenge and gap that’s possible in testing is, that is.
Hire as we go
An adventuring party will encounter many new unexpected challenges as they go, who knows what trials await them in the dungeons that the dungeon master has planned for them. A kind dungeon master might let the party run back to town as they find a need for hiring a mercenary to fill in the gaps in their skills.
Unlike adventurers in dungeons and dragons, we get the benefit of not having to go to town to hire our mercenary; all we have to do is build an agent of skill at the time we encounter a gap in our team’s testing skills.
This is cheaper as it means we just build things as we need them, reducing waste, but it does mean we’re reactive rather than planned which comes with its own issues. The main one is that fire fighting and reactive testing can be more panicked and so may miss things. We may find that we just don’t have the time to pick up that testing, or explore our options for it, when we encounter it and the problem gets ignored and missed.
Reconnaissance (scouting ahead)
A very clever adventuring party might choose to go and look at what’s coming up; either through scouting ahead to gain intel on what’s happening or talking to people to gain information in advance. Carousing in the pub might yield a lot of scuttlebutt about what the party might have to deal with.
Engineering teams can do the same using techniques such as risk storming to look forwards to the types of things that might need testing and what gaps we have. One of the main points of writing a test plan can be to identify upcoming risks so we can look at what testing we might need to do… and what mercenaries we might need to hire to fill our skills gaps to achieve that.
This approach does mean we need a good understanding of testing and quality risks so that we can identify what will be needed. One approach for this is to have a risk storming or reconnaissance agent that can help us to identify the types of testing we will need in our team… a mercenary to fill in the gaps in our testing knowledge.
Picking your mercenary
Before we build or install any agent there are a few things that we need to ask ourselves:
- Do we actually need one? The bards couldn’t see their gaps until the door appeared, which is why we look for ours before the challenge arrives.
- Which one do we need? We hire for the skills and classes that we lack and try not to build an agent that just duplicates us.
- When do we need it? When the challenge right in front of us calls for it and not when the hype says we should need one.
This matters now because AI is like letting a party move through the dungeon much more quickly. A party that’s moving quickly and recklessly might stumble upon a trap (or a dragon) that they’re not prepared to fight, and engineering teams using AI are the same… they’re moving so fast that quality gaps arrive before we’ve noticed them. That’s why we need a plan for how to fill the gaps now, even if we don’t hire the mercenary (build the AI agents) yet.
Can we trust a mercenary?
In dungeons and dragons one of the problems with hiring a mercenary is that they likely won’t share the values of the party. Let’s assume the group of adventuring bards are all good, paragons of virtue… the types of people who’d save kittens from trees, how likely is it that a hired killer they find in a seedy bar is going to be the same kind of person?
What stops that mercenary doing the wrong thing, slitting the wrong throat and causing all sorts of problems for the party?
With using AI to fill in our testing gaps we can have the same problems. What stops them going rogue and doing something we don’t want (too much or too little testing) or even something dangerous?
Guardrails
Any good adventuring party would create a contract with the mercenaries they hire. We need to decide what we let our AI agents do, what access we give them and what sources of information we give them for context.
A good contract won’t guarantee that the mercenary will be good and behave itself, but at least it helps give direction and lets us see when they’re misbehaving.
Making sure we’re ready for them
I’ve spoken about this at length in my AI Readiness Radar and its associated post but we need to make sure we have our own foundations in place to really use our mercenaries well.
Bringing in a mercenary before we’re ready will mean they’re likely to run off and do what they want and we won’t be able to control them.
Healthy scepticism
The large figure in the dingy bar shoots a greasy grin at you, one gold tooth glinting in the candlelight, “trust me baby… what can go wrong?” After rolling an insight check you decide that you probably don’t want to fully trust this mercenary completely.
We need to decide how far we trust the AI agents we use, which means that we need a healthy scepticism; because we can’t assume the agent shares our values or understands our quality bar. This is made harder when AI agents are big people pleasers and trained to flatter us and make us want to trust them (research suggests we rate people pleasing AI as more trustworthy than people and can’t reliably tell when it’s wrong).
Like with any untrustworthy mercenary we need to be checking its work and making sure what it does lines up to our needs and values.
Retirement or replacement
Lastly, we need to know when our time with our contracted mercenary is over, because a short term contract doesn’t need to become permanent all the time. We should be looking at our needs and assessing whether using any AI agents to fill in our gaps are worth it and look to retire or replace them as needed.
Replacement is important to consider because AI agents can drift, context can get out of date or results can just degrade. We need to have meaningful feedback loops in place so we can assess if our AI testing agents are still of value to us.
An adventuring party who’s not fighting anyone anymore isn’t likely to keep paying that barbarian all their treasure to just hang around and do nothing, ask yourself… do you still need that usability testing AI agent?
The catch…
Whilst a mercenary can fill a gap in the adventuring party quickly, reliance on it means the party is never going to be able to overcome those challenges for themselves. Worse still, reliance on a mercenary to do all their fighting for them might lead our party of bards to never pick spells that would help them in future combat encounters, leaving them defenceless in the future.
Our teams still have to learn; any AI testing agents are a stopgap for teams who don’t have the time to multi class into quality right now and they aren’t a replacement for coaching in the long term. Because if AI does all the testing then what happens on the day that the mercenary leaves (or more likely the tokens run out) our team will be just as defenceless and not be able to operate alone.
I don’t actually have a battle tested mitigation for this and honestly, I don’t think any team does yet, because AI is so new. We haven’t seen what happens when we plug all of our gaps with mercenaries… which means it’s something we need to keep looking at.
Wrapping up
Like with any good adventuring party your engineering team needs a variety of classes and skills to overcome the challenges thrown at you.
Use AI mercenaries to fill the gaps in your party and bring culture add to your team, but do it thoughtfully by considering which gaps you have; think about the challenges coming ahead and what skills you need to plug now.
Thanks for reading! I love talking about quality to anyone who wants to listen!