Yes you can run exploratory testing with ai

AI can actually run your exploratory testing… it just needs some help deciding what’s worth exploring, what good enough looks like and when to stop.

There’s a lot of posts and discourse on social media that claims that AI can’t do testing, especially nothing as complex as exploratory testing. In fact this is something I said back in my post can generative AI write good exploratory tests… that the answer was only “sort of”. What’s changed since then isn’t the AI model but how I think about framing my AI prompts.

It turns out that AI really can explore through a browser (or the code), apply heuristics and surface real insights, bugs and quality narratives, so long as our prompt to do something is good! Like with any AI driven development, naive basic asks won’t give a good response / outcome, the real skill is knowing how to frame exploratory testing that finds real and useful outcomes.

In this post I’ll share how I run exploratory testing using AI with a Claude skill and why my specific testing expertise is still needed.

Why exploratory testing matters more than ever

When your team is shipping code and features faster than ever what’s going to stop them delivering the wrong thing (or AI slop)?

Exploratory testing!

As I’ve talked about to developers and engineers before uncertainty and the unknown in what we’ve built (or are going to build) means more risk for building the wrong thing. Moving faster and taking understanding and context away from humans by using AI means there’s more uncertainty about what we’re building which is riskier. We need to find ways to reduce that risk by testing and exploring early throughout the software development lifecycle to create certainty.

As I’ve seen in many product and engineering teams that I’ve worked with people generally don’t know what good looks like for testing and this extends to exploration. Testing gets limited to verifying what was asked for through the happy path, something that limits us to knowing that something was correct rather than good. Often I’ve seen siloed thinking where risks related to how code or products work in the real world get pushed later and later… into a testing phase or from customer feedback; this won’t work for AI development.

Teams are moving faster than ever with AI accelerating code and feature development, the existing bottlenecks of human led review and validation will just get worse. Without a way to look at what we build for how good it is at pace, the slop can ship faster than anyone can catch it.

Using AI for exploratory testing

I’m currently using AI to both generate exploratory test ideas and to drive exploratory testing sessions. I’ve found these to both be really successful in identifying useful risks and identifying issues that I care about.

Hot take: Most people aren’t very good at exploratory testing. As I’ve written about before most people tend to assume exploratory testing is unstructured, usually a bug hunt that comes from “clicking around and finding out”. We see a lot of discussions talking about intuition or having a testers mindset for this type of testing… that there’s some dark art to being able to test a system. Good exploratory testing follows a structure: it has a goal for what it’s looking for, a scope of what to look at and identifies tools and heuristics to follow to help guide testing. When I think about structuring my exploration I tend to follow the guidance set out in Explore It! by Elisabeth Hendrickson.

Structured tests are the thing that’s moved me from “AI kinda works…” to seeing that this actually can work. When creating skills for AI to do exploratory testing I’ve focused on providing structure for the exploration to guide and set expectations for the testing it does.

Creating exploratory tests

1. Identifying risks

For structured exploration I want to explore to find out information related to a specific risk, so the first part of creating tests is determining the risks related to what’s going to be built. To do this I create AI agents that take each feature request / AC and look for risks based on tried and tested quality heuristics; these are easily identifiable across the internet with some big and well trusted names (Elisabeth Hendrickson, Ministry of Testing) attached to them.

Since quality (and testing) can be infinite I create agents to categorise each risk for Likelihood x Impact and create a score (between 1 – 10) which can be used to pragmatically select what we test for:

• High risk appetite: select risks with a score of >7
• Medium risk appetite: select risks with a score of >5
• Low risk appetite: select all risks

This is something that can be tweaked in accordance to an organisations views about risk and what good enough looks like, or can be bypassed for types of risk (such as – surface all security risks regardless of risk score). This also means we don’t waste tokens by creating tests for risks that we wouldn’t care about / would never fix… in fact we can point this skill to previously discovered defects that were fixed by the team to give a view on risks we’d care about.

2. Writing tests

From the risks that have been identified I create agents that take them and frame them into something an AI can use for exploration. To do this I set the agent to write tests per risk that follow the “explore target, with tools and heuristics, to discover information about a specific risk” format. Each of my tests then has a clear focus and a goal that I am able to control in my up front test design.

When specifying the tools and heuristics for each test (the “with” part) this will differ by context and should draw upon what’s available. I try to provide examples of what I like when critiquing the AI’s outputs so that it can learn what’s important to me. It’s important, especially early on, not to just fire and forget an AI prompt but to look at it and start steering it to better outcomes; that’s human in the loop / botsitting in action and provides a better result.

You can find my Claude skill for creating exploratory testing charters from a spec in my QE skills Github repo.

Running exploratory tests

Once I’ve got the test charters I pass them into Claude to actually run the exploration. This works because the structure of the test charter (framing, scope and risk based goal) gives the AI a targeted success criteria that keeps testing tight and meaningful.

1. Setting up the test runner

To explore via the UI I run Claude in a Chrome browser using the Claude chrome plugin so that it can navigate pages, read the page DOM and interact like a user would. For deeper work I use Claude Code within a downloaded repo with my AI enabled Playwright framework included so that it can explore across the UI, API and code layer. The set up for both are really easy and you can get AI to help you to do this.

2. Running the exploratory testing session

I give the AI prompt the charter and let it work, usually saying the type of report that I like to see as an output. Claude then navigates to the target I’ve given it and applies the tools and heuristics to start finding information, then reports on it. This is where botsitting matters because Claude will be verbose in what its trying to do, so you can interrupt and say “I don’t want this” or steer it to something that matters.

Here’s a charter I actually ran, for accessibility and usability across a web page. You can see that I’ve provided structure for the exploration by using the explore / with / to discover to set a target, goal and scope and that I’ve also provided clear heuristics to use.

3. Reviewing the report

Reports will differ because of non-determism, so if you have a report type you like make sure to feed the prompt with an example. I ask for a clear reference of which heuristic found the issue and a narrative as to why it matters, along with a scoring of how bad this issue is. here is an abridged set of the results from my previous prompt to show an example of what the exploration comes back with.

4. Deciding when to stop

The charter’s risk appetite does most of this for me; it didn’t try to test everything and instead it gave me a pragmatic set of findings mapped to the risks I said I cared about then it stopped. That’s the up front design paying off for me.

By keeping an eye on what the AI is doing I can also tell it to stop directly, or I can even give it a timebox for exploration in terms of time or token cost. I then refine these based on the outcomes I see, using my experience and judgement.

Tips for using AI for exploratory testing

It’s worth noting that if you don’t immediately get good results that you might need to train the AI on what good looks like and build its context. Reply to the tests or test results it gives you to say what you like and don’t like and tell it to commit that to memory (or update the skill). This should be seen as a living process that will need to change to give you the results that you want and need.

Because we can both create and execute exploratory tests using AI agents it means this is automatable! We can set these skills and agents up in a pipeline to run quickly, helping to provide signals of quality whilst preserving speed of delivery. I’ve found it successful to run exploratory test ideation following a Triforce / Three Amigos session (which I also have created a Claude Skill for) and running executing exploratory testing following a deployment to a test environment.

Getting Claude to execute running tests against your browser to drive exploratory testing from the UI is simple, you can download a browser plugin that allows this to happen. I’ve also found that Claude code will work well across UI, API and the code layer if you ask it to explore from a downloaded repo that has an AI enabled Playwright framework included, this will provide more detailed exploration and allow for codifying tests you want to rerun as Playwright acceptance tests.

Wrapping up

There will always be people who don’t think you can automate testing. They’ll say that it cannot think, reason or understand a customer (it can if you tell it to), that it spits out slop (it will if your prompt is lazy), doesn’t have a tester’s mindset (it can if you structure your tests right) or that it just hallucinates (correct it does, that’s why we need more testing and human review). But are these objections well founded or are they coming from a fear of not being needed anymore and losing their jobs (a very valid concern).

The things I’d say are real barriers for automated AI based exploration are actually the cost of tokens (so make sure you scope risk appetite correctly), people might lose their testing skills (so keep them sharp by reviewing AI based testing) and that it’s hard to provide meaningful context for your product / organisation (a foundation that needs defining). Additionally, for many people a big barrier will be reframing what exploratory testing really is… away from ad-hoc random bug hunting and towards something structured that actually is automatable.

We can automate exploratory testing, but this only works as well as the structure: charter, risk appetite and heuristics you give the AI. That’s where a skilled quality professional comes in; you need to decide what’s worth exploring, what “good enough” looks like through context and heuristics make for good testing and when to stop. The needs of testing have moved away from using your skills and intuition whilst doing the testing and more towards using them when framing the exploration that needs to happen.

If you want to try this out then please use the Claude Skills in my QE skills Github repo and let me know how they work for you.

Thanks for reading. If this resonated (or if you think that I’ve got it wrong) there’s more to read on the blog… or get in touch via LinkedIn. I’ll talk quality to anyone who wants to listen!