DSAgentBench: Put AI Agent Delivery Through Four Gates

What Does DSAgentBench Measure?

Writing executable code is not the same as delivering a dependable analysis. DSAgentBench is designed to test the wider job: whether an AI agent can complete end-to-end data-science work inside a real computer environment, where it must coordinate tools, respond to intermediate outputs and produce a result that can be checked.

The paper describes workflows spanning data wrangling, exploration, modelling, visualisation and validation. It also names the environments that real work crosses: notebooks, integrated development environments, terminals, browsers and databases. That is much closer to an analyst preparing a result for a leadership meeting than to a code-generation demo.

Each task includes a deterministic evaluator. The evaluator checks analytical correctness, visual outputs and model performance rather than stopping when a code cell runs successfully. That distinction matters for an SMB: a finished script is an intermediate artefact; a defensible business answer is the deliverable.

Why Can Code Still Fail to Deliver a Business Result?

Data-analysis requests often sound simple: clean a file, find a trend, create a chart and recommend what to do next. Each verb hides a decision. Which file is authoritative? What period should be included? Are missing values meaningful? Does the chart answer the question the manager asked, or just display an attractive pattern?

An agent can produce plausible code while still using the wrong source, applying an undocumented filter or answering a different question. The risk is not limited to technical errors. A workflow can be syntactically correct and operationally useless if nobody can explain the assumptions or approve the next action.

That is why the benchmark is a useful prompt for workflow design. It turns “the agent completed the task” into a harder question: completed according to which evidence, which evaluator and which owner?

What Do the Reported Results Actually Say?

In the paper's reported evaluation of 15 closed- and open-source models, the strongest agent reached a 56.70% task-success rate, while all open-source agents remained below 1% under the tested setup. The authors identify frequent failures in tool orchestration, operating-system grounding and multi-step reasoning.

These figures are useful as a capability signal, not as a forecast for your business. They describe the paper's task set, model versions and computer environment. They do not mean that an agent will deliver 56.70% of your reports, nor that every open-source system will perform below 1% on your data. The disciplined reading is narrower: long, multi-tool work still needs explicit acceptance criteria and human oversight.

The authors have also made the public DSAgentBench repository available under an Apache-2.0 licence. Public code makes inspection easier, but it is not the same as an independent reproduction or a production-readiness guarantee.

How Should a Small Team Set Delivery Gates?

Translate the research into a four-column acceptance card. The card should be short enough to use on every pilot and specific enough to stop a workflow before it touches live customer or financial data.

1. Data gate: confirm the approved files, time period, fields and access scope. Record what the agent could not see as well as what it used.
2. Calculation gate: check formulas, filters, joins and intermediate results. Require a path back to the source rather than accepting a number without provenance.
3. Outcome gate: ask whether the chart, model or summary answers the original business question. A technically polished output that does not support the decision fails this gate.
4. Responsibility gate: name the reviewer, the approval condition and the next action. If nobody owns the decision, the workflow is not ready for more autonomy.

This is an RxAI implementation recommendation, not a feature claimed by the paper. If you want to map one existing process into a reviewable workflow, start with the AI consulting and automation services page and bring one repeatable job rather than a catalogue of tools.

What Should an Agent Return With the Answer?

Do not accept only a final spreadsheet, dashboard or paragraph. Ask for a compact evidence bundle alongside the result so another person can understand what happened without rerunning the entire session.

• The source files, date range and filters used.
• The key steps, formulas or transformations that shaped the result.
• The output files and a short explanation of what each visual or table is meant to show.
• Uncertainty, missing data, failed steps and assumptions that still need confirmation.
• The recommended next action and the person who must approve it.

A deliverable without evidence is a draft, not a decision.

This format also improves the human review experience. The reviewer can focus on the assumptions and decision boundary instead of reverse-engineering a black box after the result has already been circulated.

How Can You Run a Safe Pilot?

Start with a de-identified CSV or a public dataset, not a live customer export. Choose one narrow question with a reversible output: for example, identify a change in weekly demand, reconcile two non-sensitive lists or produce a chart for an internal planning discussion.

1. Write the question and the acceptable output before opening the agent.
2. State which files and tools are allowed, and prohibit unapproved data sources.
3. Require the evidence bundle and four-gate acceptance card with the first run.
4. Have a named person review the data, calculation, business fit and next action.
5. Record what failed, adjust the task or data format, and rerun before expanding scope.

A small pilot gives the team something more valuable than a polished demo: a record of where the workflow is reliable, where it needs human intervention and what evidence is still missing.

Where Should Human Review Stay?

Human review should sit at the points where an error can change a customer commitment, financial position, compliance decision or operational priority. The reviewer's job is not necessarily to repeat every calculation. It is to validate the source, challenge the assumptions, confirm the answer fits the question and approve or stop the next action.

Make the checkpoint visible in the workflow. A button that says “reviewed” is not enough unless the team has defined what review means, what evidence must be present and what happens when the reviewer rejects the output. For higher-consequence work, add a second reviewer or a manual fallback before connecting the agent to live actions.

What Should SMBs Measure Next?

Track the reliability of the workflow, not just agent activity. Useful operating measures include:

• source or access errors caught before a result is shared;
• rework time after human review;
• reviewer turnaround and the number of blocked runs;
• how often the result answers the original business question; and
• whether the same task can be repeated with the same evidence standard.

These are RxAI measurement suggestions, not benchmark outcomes. They help a small team decide whether to improve the task definition, data preparation, evaluator or model before adding more tools and permissions. When you are ready to discuss the boundary, contact RxAI with the workflow, its current pain point and the decision it needs to support.

Sources

• Rahman et al., “DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?” — submitted 11 August 2026. Supports the benchmark design, task and model counts, evaluators, reported results and failure modes.
• vis-nlp/DSAgentBench repository — public project page displaying an Apache-2.0 licence. Used to verify the public implementation and licence context only.

Benchmark figures above are the authors' reported results in the specified test setup. The four delivery gates, pilot method and measurement suggestions are RxAI recommendations for Australian SMBs.