Building Reliable Data Analytics Agents: Lessons from the KDD Cup

來源原標題

Building Reliable Data Analytics Agents: Lessons from the KDD Cup

作者

Jiwei Liu

原始標籤

Agentic AI / Generative AI · Data Science · AI Agent · Data Analytics / Processing · data preprocessing · LLM Techniques

原始連結

https://developer.nvidia.com/blog/building-reliable-data-analytics-agents-lessons-from-the-kdd-cup/

摘要原文

The NVIDIA KGMON team placed second in the KDD Cup 2026 Data Agents competition with a system built around a simple idea of making an agent’s harness smaller, clearer, and easier to verify. The competition asked agents to answer natural-language questions over heterogeneous data sources, including databases, CSV and JSON files, prose documents, PDFs, and briefing videos. Every task required more than retrieval , with the agent inspecting available data, choosing the right tools, reasoning across sources, producing a final answer file, and handling traps that appear in analytical workflows. KDD also required teams to work with a small, fixed LLM to power the agent, making the harness the main optimization surface. The team aimed to make the task easier for the available model through preprocessing, constrained tools, persistent state, and evaluation. These techniques are especially useful when building reliable systems around smaller open models . This post shares the playbook behind their result. It isn’t a general recipe for every data science agent, but the broader lesson applies to many agent systems. Reliability often comes less from making the model more open-ended and more from building the right harness around it. Two principles behind the system Two principles shaped the system. First, constrain the action space. Too many ways to inspect data, call tools, write files, or recover from mistakes can lead to failures. KGMON unified data access, exposed a small tool set, and required final answers to follow a single output path. Second, make every attempt inspectable. A run can fail because of a bad tool call, an incorrect join, a missed document rule, or an answer-format issue. Execution traces, repeated attempts, and trajectory inspection helped the team identify failures and refine the harness. Figure 1 shows the overall workflow. Figure 1. The harness preprocesses data, limits the agent’s tools, and preserves state, with execution traces supporting evaluation and refinement The following techniques offer practical guidance for building reliable data analytics agents. 1. Convert structured sources into one query surface Each KDD task could combine SQL databases, CSV and JSON files, documents, and videos. KGMON converted CSV and JSON files into tables in the existing SQLite database, giving the agent one SQL interface for all structured data. KGMON’s custom, persistent Python environment exposed two built-in functions for inspecting and querying that unified data surface: schema() sql(query) These custom functions gave the agent a single way to discover and query structured data. They were built into KGMON’s environment rather than provided by Python or SQLite. The single SQL interface reduced routing failures and wasted turns, leaving the fixed model more room to reason over the data. Apply this approach: Normalize access to structured data before the agent starts. A temporary SQLite database, a virtualized query layer, or a governed warehouse interface can provide a stable, narrow, and well-documented query surface. 2. Seed the agent with schema context up front KGMON added a schema-scouting step before the main reasoning loop. The system inspected tables, columns, possible join keys, duplicate names, look-alike fields, units, null patterns, and row-grain issues. The agent received this schema context at the start of each task, saving an early discovery turn. Schema scouting reduced errors caused by wrong columns, missed joins, or confusion about what each answer row should represent. Figure 2 shows the unified SQL interface and the schema context provided to the agent. Figure 2. Converting CSV and JSON files into SQLite tables gives the agent one SQL interface, while schema scouting provides context before analysis begins Apply this approach: Add a read-only preflight step that briefs the agent on available tables, likely joins, key columns, units, suspicious fields, and known ambiguities. This can be more useful than another retrieval tool. 3. Build a small, opinionated tool harness KGMON limited the agent to helpers for schema inspection, SQL queries, document lookup, prose extraction, and answer writing. The environment exposed the following functions: schema() # inspect tables and columns sql(query) # query context.db write_answer(df) # write the final answer atomically prose_helper() # answer from prose or extract prose into SQL Middleware repaired malformed tool calls so one bad call did not end an attempt. A stateful Python environment retained variables between tool calls, letting the agent reuse intermediate results. Predefined functions such as schema() , sql(query) , and write_answer(df) reduced boilerplate, syntax errors, and file-handling mistakes. This saved turns and kept the small LLM focused on analysis. Figure 3 shows the environment’s functions, persistent state, and error handling. Figure 3. A persistent Python environment retains variables between tool calls, repairs malformed calls, and writes answers through a single helper function Short, valid attempts left more turns for additional runs, evaluation, and ensembling—combining results from multiple attempts. Apply this approach: Design tools around workflow requirements and remove duplicate paths. A single approved way to run SQL or write an answer reduces opportunities to corrupt state or produce invalid outputs. 4. Treat prose as a first-class input, but keep it separate from structured data Analytics workflows often include PDFs, Markdown files, documents, policy text, instructions, and reports. In some KDD tasks, these sources contained thresholds, rules, definitions, and table-like records needed to answer questions. Large documents can consume an LLM’s context window. KGMON blocked direct reads of entire files through Python’s open() function or .read() method, providing tools that limited previews and searches by character count or regular expression (regex) matches. After finding a relevant section, the agent could invoke prose_helper , a custom tool that passed document chunks to a separate LLM call with temperature set to 0 and reasoning disabled. It returned answers or extracted tables, keeping raw document contents out of the main agent’s context. Figure 4 shows the document-inspection workflow. KGMON used prose_helper in two modes: mode="answer" # extract a rule, threshold, or short answer mode="table" # extract repeating records into a SQL table Figure 4. Targeted document inspection identifies relevant text for prose_helper(), which extracts individual facts or converts repeated records into SQL tables Extracting rules from prose let the agent apply them in SQL analysis while keeping its working context focused. Important caveat: Table extraction suited competition tasks that embedded structured information in documents. Production systems may need only targeted prose lookup; table extraction can remain optional. Apply this approach: Provide a document-inspection tool that answers targeted questions and cites sources without loading entire documents into the main agent’s context. Add table extraction when documents contain repeated records that need joins or filters. 5. Preprocess video when it is part of the task Some KDD tasks included briefing videos. To avoid the compute cost of processing video inside the agent loop, KGMON extracted keyframes, transcribed audio, aligned transcript segments with frames, and supplied the resulting evidence to the agent. Each task included at most one video, often containing slide-based constraints or distractor values. Transcript-aligned keyframes connected spoken context to the correct visual evidence. Figure 5 shows the preprocessing steps. Figure 5. Video preprocessing pairs keyframes with transcript segments so the agent can interpret visual evidence in context Important caveat: Preprocessing suited the competition setup.