--- type: Blog Post title: "DSAgentBench: Put AI Agent Delivery Through Four Gates" description: "DSAgentBench tests whether AI agents can complete end-to-end data-science work in real computer environments. For Australian SMBs, the practical lesson is to put evidence, correctness, business fit and human accountability into four delivery gates before granting more autonomy." resource: "https://www.rxai.com.au/blog/2026-08-13-dsagentbench-ai-agent-delivery-gates.html" tags: [AI agents, DSAgentBench, data-science automation, workflow governance, Australian SMBs, human review] timestamp: "2026-08-13T09:36:05+10:00" category: "Automation" source_package: "2026-08-13_dsagentbench-agent-delivery-gates" source_checked: "2026-08-13 (arXiv DSAgentBench paper and vis-nlp/DSAgentBench GitHub repository)" --- # DSAgentBench: Put AI Agent Delivery Through Four Gates ## Excerpt DSAgentBench tests whether AI agents can complete end-to-end data-science work in real computer environments. For Australian SMBs, the practical lesson is to put evidence, correctness, business fit and human accountability into four delivery gates before granting more autonomy. ## What Does DSAgentBench Measure? Writing executable code is not the same as delivering a dependable analysis. DSAgentBench is designed to test the wider job: whether an AI agent can complete end-to-end data-science work inside a real computer environment, where it must coordinate tools, respond to intermediate outputs and produce a result that can be checked. The paper describes workflows spanning **data wrangling, exploration, modelling, visualisation and validation**. It also names the environments that real work crosses: notebooks, integrated development environments, terminals, browsers and databases. > **RxAI Insight:** The useful test for an AI agent is not “did it write code?” It is “can a reviewer trace the answer from approved data to an accountable decision?” ## Why Can Code Still Fail to Deliver a Business Result? Data-analysis requests often sound simple: clean a file, find a trend, create a chart and recommend what to do next. Each verb hides a decision. Which file is authoritative? What period should be included? Are missing values meaningful? Does the chart answer the question the manager asked? An agent can produce plausible code while still using the wrong source, applying an undocumented filter or answering a different question. A workflow can be syntactically correct and operationally useless if nobody can explain the assumptions or approve the next action. ## What Do the Reported Results Actually Say? In the paper's reported evaluation of 15 closed- and open-source models, the strongest agent reached a 56.70% task-success rate, while all open-source agents remained below 1% under the tested setup. The authors identify frequent failures in tool orchestration, operating-system grounding and multi-step reasoning. These figures are a capability signal, not a forecast for your business. They describe the paper's task set, model versions and computer environment. They do not predict performance on your data. The authors have also made the public [DSAgentBench repository](https://github.com/vis-nlp/DSAgentBench) available under an Apache-2.0 licence. Public code makes inspection easier, but it is not the same as an independent reproduction or a production-readiness guarantee. ## How Should a Small Team Set Delivery Gates? Translate the research into a four-column acceptance card: 1. **Data gate:** confirm the approved files, time period, fields and access scope. 2. **Calculation gate:** check formulas, filters, joins and intermediate results, with a path back to the source. 3. **Outcome gate:** confirm that the chart, model or summary answers the original business question. 4. **Responsibility gate:** name the reviewer, approval condition and next action. This is an RxAI implementation recommendation, not a feature claimed by the paper. If you want to map one process into a reviewable workflow, start with the [AI consulting and automation services](../services.html) page. ## What Should an Agent Return With the Answer? Ask for a compact evidence bundle alongside the result: - source files, date range and filters used; - key steps, formulas or transformations; - output files and the purpose of each visual or table; - uncertainty, missing data, failed steps and assumptions; and - the recommended next action and approving person. > A deliverable without evidence is a draft, not a decision. ## How Can You Run a Safe Pilot? Start with a de-identified CSV or public dataset, not a live customer export. Choose one narrow question with a reversible output. 1. Write the question and acceptable output before opening the agent. 2. State which files and tools are allowed. 3. Require the evidence bundle and four-gate acceptance card. 4. Have a named person review the data, calculation, business fit and next action. 5. Record failures, adjust the task or data format, and rerun before expanding scope. ## Where Should Human Review Stay? Human review should sit where an error can change a customer commitment, financial position, compliance decision or operational priority. The reviewer's job is to validate the source, challenge assumptions, confirm the answer fits the question and approve or stop the next action. A visible “reviewed” button is not enough unless the team defines what review means, what evidence must be present and what happens when the reviewer rejects the output. ## What Should SMBs Measure Next? Track workflow reliability, not just agent activity: - source or access errors caught before sharing; - rework time after human review; - reviewer turnaround and blocked runs; - whether the result answers the original business question; and - whether the task can be repeated with the same evidence standard. These are RxAI measurement suggestions, not benchmark outcomes. When you are ready to discuss the boundary, [contact RxAI](../contact.html) with the workflow, its pain point and the decision it needs to support. ## Sources 1. [Rahman et al., “DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?”](https://arxiv.org/abs/2608.10366) — submitted 11 August 2026. Supports the benchmark design, task and model counts, evaluators, reported results and failure modes. 2. [vis-nlp/DSAgentBench repository](https://github.com/vis-nlp/DSAgentBench) — public project page displaying an Apache-2.0 licence. Used to verify the public implementation and licence context only. Benchmark figures are the authors' reported results in the specified test setup. The four delivery gates, pilot method and measurement suggestions are RxAI recommendations for Australian SMBs. ## Frequently Asked Questions ### What is DSAgentBench? DSAgentBench is a benchmark for testing whether AI agents can complete end-to-end data-science workflows inside real computer environments, including tool coordination and deterministic evaluation of outputs. ### Does 56.70% mean an agent will be 56.70% accurate in my business? No. The figure is the strongest result reported by the paper under its task set, model versions and computer environment. It is not a forecast for an Australian business or a guarantee about any particular model. ### What four delivery gates should SMBs use? Use data, calculation, outcome and responsibility gates. Check approved sources, formulas and intermediate results, business fit, and the named reviewer and next action. ### How should a team start safely? Use a de-identified or public dataset for one narrow, reversible task. Require an evidence bundle and human review before expanding permissions or connecting the agent to live data and actions.