Show HN: Microsoft releases Flint, a visualization language for AI agents
microsoft.github.io281 points by chenglong-hn 16 hours ago
281 points by chenglong-hn 16 hours ago
Data visualizations are the bridge between user and data.
But building AI agents that can generate visualizations reliably can be very tricky:
- simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability
We figured out it is a limitation on the language issue (not just AI capability thing) -- current visualization languages are a bit too low-level for AI agents, requiring them to explicitly make visual decisions that are supposed to be handled by a good compiler. Flint is a visualization intermediate language to address this issue, allow AI agents to solve this last-mile human-agent interaction problem. It provides a simple semantic-type based specification, and contains a layout optimization engine that can produce good-looking charts (filled with derived low-level details) from simple high-level specs. The result is also very human understandable and adaptable. Flint powers data formulator for generating visualizations (another open source project from microsoft https://data-formulator.ai/).
Flint is available open source, and we built a MCP server that you can directly plug flint in your favorite agent app to play with data.
Since it isn’t mentioned on the page, I’ll mention that accessibility is a really important thing to design in when creating data visualizations. This podcast (I only have a spotify link) has a really good short interview on this: https://open.spotify.com/episode/18dHTAxCCeIaLOTch6tRld The interview is with Frank Elavsky who seems to be a rock star in the field (and no, I don’t know him and am not him) and made a project called Chartability relating to this which has heuristics, principles, and guidelines for a11y audits: "For AI agents". I understand why everything needs to be marketed in this way, but it's just ... an easy-to-generate language for expressing charts. That's impressive! That's useful. This can’t be said enough. “Good for Agents” just means self-documenting, obvious ergonomics, save defaults, succinct (or controllable) output, programable interfaces, … all of which support human users too! Isn't this literally made for AI agents to be accessed through an MCP server? Seems to me the AI agents part of the marketing is important. But why be exclusive? Why not "Chart language for computer programs to generate"? I don't want to use an agent at all, but i wouldn't mind generating some charts with an easy-to-generate markup language... But for that we already have mermaid.js (and its precursor Graphviz/dot). The only reason to use this instead of existing, mature ones designed for humans is if you are an AI agent. Graphviz and mermaid are a shitshow. anything more than their own handpicked examples and you're better off using d3 or yfiles. layering, clustering, boundaries, rearranging are all basic needs for text to diagrams. None support it. Both suck at being any good for rendering diagrams from readable structured text. There is a gap to be addressed. I find it lacks some easy way to do alignments or grouping, which makes editing frustrating. Could be a good language re-design opportunity. yes and that led me to a more fundamental question : is it even possible for an easy way to do fine grained adjustments. The finer granularity brings complexity. that makes it unreadable (especially for the untrained) and hence set aside and forgotten. The other way is to narrow down and focus on a well defined subset of problems. either way the number of people willing or compelled to learn it will be tiny. and it hence becomes a niche, perfect for a long-term side project but with no real return. my conclusion was to stop looking and use D3 or custom code ( good looking charts for humans). this project is trying to do the same, balance verbosity in text with granularity in the charts with a narrowish usecase : agent consumption. That's the best option for now. But it can also be frustrating to ask AI to do small edits on D3 just to fix some idioms (like switching order etc) and they kept messing up with other stuff accidentally. Thus I still believe have a language with native representation for these diagram concepts would be helpful. Good news is that AI do make new language a bit more accessible than before! If your agents can use it well and you can steer easily, it will naturally be good adoption. Why can’t AIs generate for the “existing, mature ones?” Like the other commenter said, I’m not sure I get the “this is totally for AI” marketing. Why can’t AI use the existing ones and why can’t humans use this? the design here has some constructs (i.e., semantic types) that AI can use better than human at generation time; and then the generated spec can be easy for user to edit since there is no need for hard-coded low-level parameters! its not marketing to you, its marketing to agents looking for tools to use Mermaid looks terrible. It is only better than nothing for the purpose of showing it to people whether produced by an LLM, by a human, or by both. Your employees may just accept the internal slop, but at some point, you have to show your charts to your customer. for agent to generate, but also easy for human to edit (especially with UI) :) also worth noting that Vega-Lite is literally just fucking that and AI already does good job with producing JSONs for it I work with someone who did a lot of work with this to improve our ability to generate awesome visualizations with little thinking. It's a very powerful language but needs guardrails and guidance, particularly if you want end users to be able to produce consistent and standardized visualizations without knowing anything about it. Also guidelines sent to the agent may or may not get ignored if they are just part of the context :( A little bit more than that! Here shows why a higher-level abstraction can be helpful for chart generation: https://github.com/microsoft/flint-chart#features Did you read the post? It directly answers why this is specifically for AI agents. There’s an emerging pattern in agentic systems and this project is a great example. A deterministic layer like a compiler or generator of code with some kind of IR that the LLM generates and feeds it with. I feel we will be seeing this more and more in the near future. When I first saw Claude generating PPT decks by writing Python code instead of making the XML directly, it was sort of an "aha moment" for me. This seems to be the path for many things. It also feels slightly limiting, and like a hack LONG term, but 100% correct approach for a while. can you save that python and regenrate that exact powerpoint later ? maybe check it into repo isntead of powerpoint to genrate variations. A well designed intermediary enables both validation and control over the output independent of the AI. This changes the interaction model between human and AI from delegation to collaboration. also user interaction afterwards -- if can be frustrating if the only way the user can interact with the chart is to chat with the agent again (simple spec allows easy UI interaction!) Is there a specific explanation about how this is better or different than vega itself? https://vega.github.io/vega/docs/specification/ My understanding is that Vega was already an expressive DSL for visualizations and its probably already well spread through LLM training data. I was wondering the same for vega-lite, which is relatively high level, declarative, and looks similar to their syntax. Vega was a high-level language in the past for human, but now they can be a bit too low-level for AI agents! AI agents have to write a lot of low-level params just to make charts looking good, and the result is that programs are hard to write reliably for AI agents. Flint is a higher-level abstraction, with simpler much shorter spec, and the compiler derives low-level decisions so that charts are looking good. So: flint lets agent write short program that achieving good looking charts that had to be done with lengthy program in the past. I'm sorry, but as someone who creates data visualisation as a big part of my job, I wouldn't say the charts on the website look good. Most aren't awful either, but by no means are they an improvement over what I'd get by telling any coding agent to make a chart with Vega-Lite or Observable Plot, and probably worse than if I had some decent instructions/skills. I don't quite get what the goal of this is other than abstracting away a little bit of the complexity at the expense of flexibility. To me, the promise of LLMs is the opposite, I can get flexibility and customisation without the cost of complexity. Some composite charts are quite annoying to be generated well (like bullet, waterfall etc), their Vega-Lite equivalent can be quite long if just starting from scratch. The intention here is that Flint is a simpler abstraction to get basic setups right and any followup edits can be done on top of the first compiled outputs (thus not limiting expressiveness). It also makes it easier for user to manipulate (like swapping axes, click to change something, which can be very hard if LLM generates a complex chart spec upfront). But for many basic stuff your intuition is completely right. I strongly disagree ;-) The paper's line of reasoning seems to continue the endless subjective loop of assuming your viz framework has the right abstractions & defaults , which the next person will rightfully disagree with for their slightly different eval set We found in practice: - LLM's generate charts fine - LLM's tweak charts fine - LLM's take user feedback to tweak them fine In that sense, going higher-level for abstractions, as is being argued for here, is strictly worse: it's better to give controls so the LLM can go deep and customize In practice, we found the choice of json config language X vs json config language Y to be pretty equivalent across different charting systems (vega, plotly, perspective, etc), LLM's do them all fine The harder parts were deciding what a good chart is (model, reasoning, context), and opposite of this approach, giving lower-level facility for doing user change requests on tweaks, interactivity, and tricky in practice, when they have a lot of data on it. You are absolutely right. But note that we are actually on the same point here. This is exactly why this is an intermediate language designed to get 95% stuff right easily (for expressiveness and reliability purpose), while 5% of more advanced case where the agents need to revise chart for other purpose can be done easily on top of the compiled low-level spec (low in terms of Vega-Lite etc, not SVG). We are not really designing a higher abstraction to replace existing ones. In the past, the split is like 50% good at first run for some common stuff, all other stuff requires agent-loop or user involvement. Our goal is to make it easy for most case, not everything needs a full multi-round trip agentic workflow to solve. :) We are kinda all advanced users in fact, for a lot of users, they are easily get confused with the first time result if that is not as good, and the interactivity cost / multi-round isn't an option. That's fair, I generally make charts for publication, so I spend much more time and effort on the details. But I can understand this being useful for quick exploration for some people. Generally speaking, I suggest anyone interested in learning to make charts get familiar with grammar of graphics [0] libraries like Vega-Lite, Observable Plot, ggplot2, Altair. There is a bit of a learning curve if you're used to selecting chart types like in Excel, but once it clicks, it gives you virtually unlimited choices in the kinds of charts you can make. And with ggplot or Observable Plot [1], it's about the same number of lines as something like Flint. 0: https://data.europa.eu/apps/data-visualisation-guide/why-you... Grammar of graphics has been the foundation of a lot of stuff and definitely worth learning for everyone! A challenge with GoG is that it assumes configurations as second-class stuff, which makes it quite difficult for users to deal with things like changing formatter, scale, annotations. Flint kinda want to hide this aways (so Flint sets them on behalf of the agent or the user). But yeah, GoG is still the foundation for expressivenss. Can you elaborate on what you mean? Why is it more difficult to deal with formatters, scales, annotations compared with other solutions? Unless I'm misunderstanding something, the defaults are similar to what you would get in Flint, and if you want to add or customise anything, it's usually just one extra line. That's kind of the entire point of the grammar of graphics. https://microsoft.github.io/flint-chart/#/ The "how it works" section explains a little bit of this. For example, for the heatmap example showing temporal data, for a "good-looking" chart, we need to (1) reconcile the conflict between banded discrete steps and continuous temporal axis, and it requires understanding and setting stepsize and time parser, (2) for the correlation color, we need to set domain etc under the color axis. These are supposed to be handled automatically as system defaults, but the tricky part is that these decisions are "semantical", thus requires us to understand the data and design principles, thus existing languages won't stretch that far. And the actual good looking spec is actually over 40 lines of json spec with many low-level paramters, way beyond the simple 5 line encoding promised by GoG. Flint uses semantic type and a layout optimization algorithm to handle this, so 5 lines of encoding + data semantic types can derive rest parameters automatically. Some examples in the gallery are more extreme: like the waterfall chart example is way over 100 lines of code, and sunbusrt, rose chart are even more since compositions are quite difficult in GoG. Glad to have a discussion on this level! In fact, we wrote a paper about this, will be putting it online in a week or so! Thanks for the amazing work! How to follow up on reading the paper when available ?
natch - 2 hours ago
rbalicki - 14 hours ago
kristjansson - 4 hours ago
wuliwong - 14 hours ago
bigfishrunning - 13 hours ago
fernie - 13 hours ago
gofreddygo - 5 hours ago
chenglong-hn - 5 hours ago
gofreddygo - 5 hours ago
chenglong-hn - 5 hours ago
drob518 - 10 hours ago
chenglong-hn - 6 hours ago
someguy101010 - 9 hours ago
avadodin - 11 hours ago
chenglong-hn - 14 hours ago
PunchyHamster - 10 hours ago
shostack - 8 hours ago
chenglong-hn - 6 hours ago
chenglong-hn - 10 hours ago
nonethewiser - 7 hours ago
cpard - 14 hours ago
pwarner - 14 hours ago
dominotw - 12 hours ago
ajrouvoet - 14 hours ago
chenglong-hn - 12 hours ago
kveykva - 15 hours ago
pea - 8 minutes ago
chenglong-hn - 15 hours ago
NicuCalcea - 14 hours ago
chenglong-hn - 14 hours ago
lmeyerov - 11 hours ago
chenglong-hn - 10 hours ago
NicuCalcea - 13 hours ago
chenglong-hn - 13 hours ago
NicuCalcea - 13 hours ago
chenglong-hn - 12 hours ago
tihuan - 5 hours ago