Chart integration

[rohitpathak21]

Summary

This PR adds a new Chart Toggle option to the agent configuration UI, enabling inline rendering of charts in chat conversations. When this toggle is enabled, the LLM responds with a strict Echarts options tructure for chart data, which the frontend parses and renders using ECharts.

The main motivation is to allow users to visualize analytical or structured data directly within conversations—similar to the artifact feature but specifically for charts. This unlocks real-time visualization use cases such as AI usage metrics, cost breakdowns, or monitoring reports.

An example application of this feature is an internal MCP tool we've developed, which generates AWS billing data formatted as ECharts JSON. When combined with this chart toggle, the data is rendered inline as a cost dashboard, effectively acting as a conversational mini-Cost Explorer.

There are opportunities for design improvement and extended parsing flexibility in future iterations.

Change Type

New feature (non-breaking change which adds functionality)

Testing

Manual testing was performed to validate the following:
Toggling the chart feature on/off in agent settings correctly alters rendering behavior.
Responses from the LLM in the :::chart{} format are parsed and rendered using ECharts in the chat window.
Existing functionality like artifacts, markdown, and code rendering remain unaffected.

Checklist

• [ ✅] My code adheres to this project's style guidelines
• [ ✅] I have performed a self-review of my own code
• [✅ ] I have commented in any complex areas of my code
• I have made pertinent documentation changes
• [ ] My changes do not introduce new warnings
• I have written tests demonstrating that my changes are effective or that my feature works
• Local unit tests pass with my changes
• Any changes dependent on mine have been merged and published in downstream modules.
• A pull request for updating the documentation has been submitted.

I have added two video files showing the working of this functionality

Screencast.from.2025-07-24.04-48-28.mp4

Screencast.from.2025-07-24.04-50-42.mp4

[owengo]

It would be great to have support for graphics. It seems that your solutions works like this:

1. The LLM calls an MCP tool ( or whatever ) to gather some external data
2. Then it instantiates the chart copying all the data returned by the tool

I works probably very well for a small amount of data but it is not scalable.

With another LLM frontend 2 years ago I have tried cubejs, and it was working like this:

1. The prompt explained to the llm the structure of the data in the backend ( the cubejs spec )
2. The LLM instantiates the chart with a backend query as the datasource ( the language was some kind of pseudo sql specialized in aggregations )
3. The charts rendering was directly connected to the backend

So the llm did not "see" the data, but it was an advantage because:

• it was very fast, the inference time was only related the time to generate the query
• it was "accurate": no copy-paste errors from the llm
• no problem llm context saturated by the volume of data, and paying twice for it as input and output

Would it be possible the same kind of solution with echarts? ie: have a query in the datasource ( which could be generated by an MCP by the way )

[fstadt]

Possible solution: Provide a way to specify a url or mcp resource instead of the echarts json. The url could then be fetched to get the echarts json itself. Then the LLM would just need to correctly copy-paste the url / resource identifier.

Could possible be used for other artifacts as well. If the llm wants to "see" the data behind the graph itself, it could still do so by fetching the resource or url.

I still like the ability to provide the echarts json directly though since it might be useful even without integrating a specialized mcp server (just by using server instructions for echarts).

[rohitpathak21]

It would be great to have support for graphics. It seems that your solutions works like this:

1. The LLM calls an MCP tool ( or whatever ) to gather some external data

2. Then it instantiates the chart copying all the data returned by the tool

I works probably very well for a small amount of data but it is not scalable.

With another LLM frontend 2 years ago I have tried cubejs, and it was working like this:

1. The prompt explained to the llm the structure of the data in the backend ( the cubejs spec )

2. The LLM instantiates the chart with a backend query as the datasource ( the language was some kind of pseudo sql specialized in aggregations )

3. The charts rendering was directly connected to the backend

So the llm did not "see" the data, but it was an advantage because:

* it was very fast, the inference time was only related the time to generate the query

* it was "accurate": no copy-paste errors from the llm

* no problem llm context saturated by the volume of data, and paying twice for it as input and output

Would it be possible the same kind of solution with echarts? ie: have a query in the datasource ( which could be generated by an MCP by the way )

@owengo Thank you for taking the time to thoroughly review my PR and providing such detailed feedback. I really appreciate you sharing your experience with CubeJS and the query-based approach - it's clear you've thought deeply about the scalability challenges of chart rendering systems.

You raise valid points about performance and scalability. However, I'd like to clarify a few aspects of our current implementation and the challenges we're trying to solve:

Our system needs to handle truly arbitrary data sources and query types that can't be predefined. While CubeJS works excellently for known schemas with predefined cubes, our use case involves dynamic query generation (Snowflake, MongoDB, custom APIs, GraphQL endpoints, etc.), unknown data structures that change per request and can't be defined upfront, and varied data formats (nested objects, flat arrays, mixed types) that need intelligent interpretation.

The approach we've implemented actually addresses many of your performance concerns. Our MCP tool already optimizes data flow by converting NLP → SQL → aggregated data, returning chart-ready datasets (not raw records). We return summarized data like ["Q1", "Q2", "Q3"] and , not thousands of individual records. The LLM handles the "last mile" of intelligent formatting of any data structure into proper ECharts options.

I'm definitely open to exploring a hybrid solution that combines the best of both approaches. We could implement a mode-based system where known, high-volume data sources use query mode with direct backend connections, while arbitrary/unknown data uses our current embedded approach. We could also add an optional backend integration layer that detects if incoming data matches known schemas and routes accordingly, with fallback to our current approach for arbitrary data.

I'll admit I'm not deeply familiar with CubeJS, but from what I've researched, it seems to require predefined data models that need to be configured upfront, known database schemas with established relationships, and standardized query patterns within the OLAP paradigm. While this works great for traditional BI scenarios, it can't handle the truly dynamic, arbitrary data visualization that our system aims to provide. Our current approach can create charts for literally any data structure that an MCP tool can return.

Would you be interested in collaborating on defining what a hybrid approach might look like? I think there's definitely value in keeping the current flexible approach as the default for arbitrary data, adding query-based optimization for specific high-volume use cases, and creating a configuration layer that lets users choose the appropriate mode.

I'd love to hear your thoughts on this direction and whether you have specific backend integration patterns in mind that would work well with ECharts. Thanks again for the thoughtful feedback - it's exactly this kind of architectural discussion that makes features better!

[rohitpathak21]

Possible solution: Provide a way to specify a url or mcp resource instead of the echarts json. The url could then be fetched to get the echarts json itself. Then the LLM would just need to correctly copy-paste the url / resource identifier.

Could possible be used for other artifacts as well. If the llm wants to "see" the data behind the graph itself, it could still do so by fetching the resource or url.

I still like the ability to provide the echarts json directly though since it might be useful even without integrating a specialized mcp server (just by using server instructions for echarts).

@fstadt @owengo
You both raise good points, but I think there's a key question: where should the intelligence live?

The URL/resource approach shifts "arbitrary data handling" from the LLM (where it excels) to my rendering logic (where it becomes much more complex). When fetching from a URL, the data format can still be anything - nested objects, flat arrays, unknown schemas, etc.

The core value of my approach is using the LLM as an intelligent data transformation layer. It handles pattern recognition and makes smart visualization decisions that would be exponentially harder to code programmatically.

The performance concerns assume large raw datasets, but our MCP tools return pre-aggregated, chart-ready data. The LLM does "last mile" intelligent formatting, not heavy data processing.

I'm open to a hybrid approach for high-volume known schemas, but for "create charts from any data source," the LLM-based approach provides flexibility that's hard to replicate otherwise.

Thoughts on preserving this intelligent data handling while addressing scalability concerns?

[ronak21691]

@danny-avila is there plan to get this into main? We were thinking to use it.