Is it Right? Characterizing Errors in Large Language Model-Produced Data Visualizations

Large language models (LLMs) have shown potential as natural language interfaces for producing data visualizations. However, LLM-produced visualizations are often riddled with errors. To better understand the types of errors LLMs make on these tasks and how they tie back to prompt wording, using OpenAI’s GPT-4o we conducted two studies in which we qualitatively coded errors associated with (1) a set of 237 manually-developed, highly structured queries and (2) a set of 150 naturally derived, colloquial queries from an established benchmark. Based on this, we present a characterization of the types of errors made by LLMs on visualization tasks. These include errors related to data handling, charts set-up, and code construction. Below you can explore our results across the two sets (Test A and Test B). For each prompt, we include the prompt characteristics, the text, code, and chart produced by the model, and the types of errors in those responses.