Can an AI agent test your software?
Partly, yes. You can write a test as a sentence, such as "add the T-shirt to the cart and check that the cart shows it". An AI agent then runs it on the site, the way a person would. It works best when the agent remembers the steps that already worked and an independent check confirms the result.
In late October InfoQ, a publication for software developers, covered Bug0 QA Agent, a library from the company LinearBytes. Whoever writes the test lists the steps in plain English inside Playwright, a tool that checks websites automatically. The first time, the agent looks at the page and tries the actions. When a step works, it saves it and replays it next time without calling the AI.
How does it work, in practice?
- You write the steps as sentences: open the site, pick the product, add it to the cart.
- The first time, the agent reads the page and decides where to click.
- A step that works is saved and replayed in moments, with no AI calls.
- If the site changes and the saved step stops working, the agent works it out again and saves it again.
- Two different models, Claude and Gemini, judge the final result. If they disagree, a third model decides.
There are limits. Saving only works for steps with a single action: longer steps go through the agent on every run, and the final checks call the AI every time. Bug0 presents it as an open-source library. InfoQ points out that the licence lets you read and use the code, with limits on anyone who wants to compete with it for the first two years.
Who already uses it in a large company?
Epsilon, a data and marketing company in Publicis Groupe, explained on the AWS blog how it uses Amazon Nova Act, Amazon's agent that works inside the browser. Its tests are instructions in English, such as "confirm the rewards balance is visible". People who do not write code can read them, and they hold up better when the design changes.
Epsilon did not throw away its old coded tests; it added the new ones alongside. By its own internal measures, an 18-step flow takes 5 to 10 minutes to write, against 30 to 90 minutes for the same test in code. Once it finds the right path, it saves and replays it, and it says saved paths always pass. These are the company's figures, published by AWS and not checked by anyone else.
Why does it matter for your business?
If you run an online shop or a web-based management system, every update can break something: payment, sign-in, sending a quote. Classic automated tests catch it, but they break too when a button changes. Someone has to fix them by hand, and often people stop doing it.
A test written as a sentence is close to how your office describes the work. Your sales manager can read "send a three-line quote and check the customer gets the email" and say whether the check is right.
Cost belongs in the sum. Every AI call has a price and takes a few seconds per step. That is why it matters how many steps you can save and replay without AI.
There is also a risk: an agent can say a test passed when it did not. Google, in its PageBreak project on the security of its own sites, counts a problem only when a check written by people confirms it. Our read: the same principle applies to tests, where the AI runs the steps and a fixed check confirms the result, such as the email reaching the customer or the row saved in the management system.
What about us?
In our software delivery service, quality assurance is part of the work, from analysis to production. The Oncologica LIMS, which we run in production, has processed more than 6 million samples with a 0.5% scan error rate.
How we would use an agent for testing: we would start with the two or three flows that bring in orders or cases, alongside the tests you already have. A fixed check would confirm every important result, not just the AI's judgement.
So what do you do?
- If your site or management system often breaks after updates, start with the flow that breaks most often and write it as sentences.
- If you already have automated tests that work, keep them: the agent adds to them, it does not replace them.
- If a mistake is expensive, such as a payment or an order, have a fixed check confirm the result.
- If your steps are long and change often, first measure what the AI calls cost on each run.
“A test written in plain words can be reviewed by the people who know the work. The result, though, is confirmed by a check that does not fail the way the AI does.”
DBrothers — on software tests run by an agent
Frequently asked questions
Can I write the tests in my own language?
The examples published by Bug0 and Epsilon are in English, and neither source mentions tests written in other languages. Try it on your own case before you rely on it.
Do I need a developer?
To install and connect the tool, yes: Bug0 is a library for Playwright and Nova Act runs on AWS. To write and review the steps, no, and that is the main advantage.
What does it cost?
Bug0's library can be downloaded from its public repository, while calls to the AI models are paid separately. Bug0 also sells a managed service, with a dedicated engineer, from 2,500 dollars a month, according to its pricing page.
Name the flow on your site or management system that breaks most often after an update. We start there to write the first plain-language tests and decide how to confirm their result.
