How do you verify AI agent task completion?
You look at what it changed in your systems, not at what it said. That is the idea behind ThinkingBox, a test built by Microsoft's Copilot Studio team with Toloka and presented with Hugging Face on 3 October 2026. It holds 507 rebuilt office tasks, in sectors such as online retail, travel and banking. At the end of each task it does not read the agent's answer. It checks the data left in the database.
The example in the first article is a complaint. A customer's kitchen appliance is stuck with the courier, 15 days past its expected delivery date. The agent takes nine correct steps. It finds the order, checks the tracking, reads the refund policy and opens a ticket. Then it closes the ticket as resolved. But the courier problem is still open, and the ticket should have stayed on hold. The replies were fine, the database was not.
What did this check uncover?
That many failures are invisible. In one part of the test the authors counted almost 80,000 failed attempts. Two in three had ended without errors, after the agent had used a tool that writes data. Only the database check found wrong values, extra effects or missing steps.
The second point is consistency. Every task runs 20 times, always from a clean start. A model can succeed at least once on almost every task and complete few of them in all 20 runs. In the authors' words, an agent that processes a refund correctly once and gets the next four wrong is not a working refund agent.
Can the same check also train the agent?
Yes, according to the authors' second article, published on 22 October. They used the database check as a score to train Qwen3.8, an open-weight model from Alibaba, on tasks different from those in the test.
After 50 training rounds, the tasks completed in all 20 runs went from 38 to 107 out of 507. In a business travel task, the model used to change the booking and refund without the manager's approval in 15 runs out of 20. After training it left booking and payments untouched in all 20. There is a cost too, as tasks solved at least once fell from 453 to 446.
Two caveats. The tasks are rebuilt and the customers are not real. And both sources come from the same authors, so independent confirmation is still missing. The code is public, so others can rerun the tests.
Why does it matter for your business?
Picture an agent that records supplier invoices in your accounting system. When the day ends it writes "120 invoices recorded". The useful check looks at the system instead. It should hold 120 entries with the right amounts and due dates, no payment sent by mistake, and the invoice waiting for approval still on hold.
Our read: an agent in production is judged like a new colleague. Saying it has finished is not enough. You check the work, and you check it several times, until the results are stable.
- For each task, write down the expected end state: which data changes, which stays the same, what must not happen.
- After every run, check that data automatically, separately from the agent's reply.
- Before production, repeat the same test several times. Count how often it succeeds in a row, not whether it succeeds once.
- Look for extra effects too, such as an email sent, a refund or a change nobody asked for.
- Keep a log of actions with inputs and outcomes, so you can see where something broke.
What about us?
On our Security page we say every agent call goes through a single gate and is logged with its inputs and outcome, and that sensitive actions wait for a person. On our Pricing page we say that after launch we report results against what the process cost at the start.
How we would use the ThinkingBox idea: for each agent we would write checks on the end state of your systems, and run it several times on the same cases before giving it real data.
So what do you do?
- If your agent only answers questions, judge the answer.
- If your agent writes to an accounting system, a CRM or a database, judge it by what stays written there.
- If a vendor shows you one successful run, ask how many times out of ten it repeats it exactly.
“An agent is done when the data confirms it, not when it says so.”
DBrothers — our read on ThinkingBox
Frequently asked questions
Do I need a developer to set up these checks?
To write them, yes. But the rule is set by whoever knows the process, meaning what the system should show when a case is closed properly.
Do newer models make fewer mistakes?
Not necessarily more consistently. In the test Claude Opus 5.5 has a higher average score than Claude Opus 5, but it succeeds in all 20 runs on exactly the same number of tasks: 241.
Can I use ThinkingBox on my own processes?
The code is public under the MIT licence, but the tasks are rebuilt examples. Your processes need checks written on your own data.
If you have an agent that writes to your systems, we pick one task and write the end-state check together, before giving it more work.
