From Data to Dialogue: How S&P Global Energy Made Its Structured Data Estate Conversational with Databricks Genie Agents and MCP

來源原標題

From Data to Dialogue: How S&P Global Energy Made Its Structured Data Estate Conversational with Databricks Genie Agents and MCP

機構

Databricks

原始標籤

All · Company · Customers

原始連結

https://www.databricks.com/blog/data-dialogue-how-sp-global-energy-made-its-structured-data-estate-conversational-databricks

摘要原文

• Domain experts curate focused Genie Agents per dataset group without writing agent code, establishing a governed semantic layer. • Genie Agents act as managed MCP servers that are composed via a FastMCP proxy into composite endpoints for cross-domain queries. • This architecture significantly shortened time-to-market for conversational data products while preserving Unity Catalog governance. If you have ever tried to make a large, complex structured data estate available to AI agents and assistants, you have likely run into the same wall we did: agents are only as good as the context they can reach, and enterprise data rarely lives in one neat, well-documented place. At S&P Global Energy, our data spans Chemicals, Crude Oil, Refined Products, Gas & Power, Liquified Natural Gas (LNG) and more — and each commodity is itself a rich family of datasets. LNG alone includes facility specifications, cargos , outages, supply and demand fundamentals, netbacks, historical and forecast prices, and contracts. Chemicals span capacity, production, utilization, trade, demand by end use and by derivative, inventory change, and country- and region-level supply–demand balances. Our other commodities follow similar patterns. This data lives across Databricks and several non-Databricks sources. Our goal was ambitious but simple to state: make our entire structured data estate available for external consumption by AI agents through the Model Context Protocol (MCP) — so that our customers’ agents and assistants, as well as our own, could ask questions in natural language and get trusted, governed answers. We evaluated several approaches. What worked best for us, by a wide margin, was Databricks Genie Agents , exposed as managed MCP servers, composed into domain-specific bundles with an MCP proxy layer. Large language models are remarkably good at conversation and reasoning, but they cannot answer questions about your data unless you build a bridge to it. For structured enterprise data, that bridge has historically meant one of the following: Each of these approaches shares the same problem: the people who understand the data best — our SMEs and analysts — are not the people building the access layer. Every insight had to pass through an engineering backlog. Our time to market for a new conversational data experience was measured in months. We needed an approach where domain experts could curate and publish conversational access to data directly, engineering could standardize how agents connect, and governance stayed centralized. That is exactly what Genie Agents plus MCP gave us. Our architecture has three layers, and each layer is owned by the people best suited to it. The diagram above shows the end-to-end flow — including what sits inside the S&P Global Energy network and what sits in the external client environment. This is where the magic starts, and notably, it requires no code. Our SMEs begin by selecting the tables relevant to a business domain: They then group related tables and create one Genie Agent per dataset group — not one giant agent per commodity. Each sub-category of a commodity becomes its own focused Genie Agent. Within LNG, for example: Every other commodity follows the same pattern with its own sub-categories. Chemicals, for instance, has group-level Genie Agents for capacity, production, capacity utilization, trade, demand by end use and by derivative, inventory change, and country- and region-level supply–demand balances ; Crude Oil, Refined Products, and Gas & Power are organized similarly. The result is a fleet of small, sharply scoped Genie Agents rather than a handful of sprawling disconnected AI tools. Inside each agent, SMEs add the context that makes text-to-SQL actually work in the real world: descriptions of tables and columns, example queries, trusted assets for high-stakes metrics, and business definitions (for example “floating storage is defined as cargoes idling for 3 days or more in vessels travelling below a threshold speed”). This is the step that generic text-to-SQL solutions skip — and it is the step that determines whether users trust the answers. The key organizational insight: curation became a domain activity, not an engineering activity. The person who knows what “floating storage” means in an LNG context is the person teaching a Genie what it means. Here is where Databricks did the heavy lifting for us. Each Genie Agent is exposed as a Databricks managed MCP server out of the box, at an endpoint of the form: https://<workspace-hostname>/api/2.0/mcp/genie/{genie_space_id} There is nothing to deploy and nothing to host. Each server exposes a small, clean tool surface — essentially two tools per agent: This two-tool, ask-then-poll pattern turns out to be a great fit for agentic workloads: questions run asynchronously against a SQL warehouse, and the agent polls with the conversation and message ID returned by the query tool until the response is ready. Just as importantly, these managed servers are governed by Unity Catalog. A Genie Agent — or the user behind it — can only reach the agents and underlying tables they have permission to see. Authentication is handled by the platform. We did not have to build a security layer around our AI access; we inherited the one we already had. One Genie Agent per dataset group keeps each agent focused and accurate. But real business questions routinely cross groups: “How did the recent outages at Sabine Pass affect cargo premiums into Asia?” touches both the Outages and Cargo Genie Agents at once, and cross-commodity questions like “How are naphtha prices affecting chemical production margins?” reach across Refined Products and Chemicals Genie Agents. Rather than building one giant agent (which degrades answer quality) or forcing every client to configure a dozen separate servers, we used FastMCP’s proxy and composition capabilities to create composite MCP endpoints — typically one per commodity, mounting that commodity’s group-level Genie MCP servers behind a single server with name-spaced tools. Higher-level composites can bundle several commodities the same way: # Each group-level Genie Agent is a managed MCP server on Databricks cargo = FastMCP.as_proxy(genie_mcp_config(“lng_cargo_agent_id”), name=“cargo”) outages = FastMCP.as_proxy(genie_mcp_config(“lng_outages_agent_id”), name=“outages”) netbacks = FastMCP.as_proxy(genie_mcp_config(“lng_netbacks_agent_id”), name=“netbacks”) # Compose the group Genies into one commodity bundle lng = FastMCP(name=“lng-composite”) lng.mount(cargo, prefix=“cargo”) lng.mount(outages, prefix=“outages”) lng.mount(netbacks, prefix=“netbacks”) # The same pattern repeats for Chemicals, Crude Oil, Refined Products, Coal … (Illustrative snippet — adapt to your FastMCP version and auth setup.) The result: an agent connects to one composite endpoint per commodity and sees a curated set of group tools — cargo_genie_query_agent, outages_genie_query_agent, netbacks_genie_query_agent, and so on, each paired with its genie_poll_response counterpart. The agent’s LLM decides which group Genie to route a question to, or fans a cross-group question out across several, then synthesizes the results. This gave us the best of both worlds: narrow, high-accuracy group-level Genie Agents underneath, and broad, commodity- and estate-wide conversational access on top. For our business stakeholders, the technical details above translate into a few very tangible outcomes. Time to market collapsed. Previously, standing up a new conversational data experience meant a full development cycle: requirements, API design, text-to-SQL engineering, testing, deployment. With this architecture, launching a new dataset group — or an entire commodity — means an SME creates and curates the corresponding Genie Agents — the MCP endpoint exists the moment the agent does. SMEs became publishers, not requesters.