Visual Graph Scaffolds for Structural Reasoning in Large Language Models

Abstract:

Graphs have been used to enhance large language models (LLMs) for structured reasoning, mostly as external knowledge sources provided at test time. In this paper, we take a different view: graphs can also organize reasoning itself. Inspired by graph-structured mind maps, we study whether visual graphs can serve as internal reasoning assistance for multi-hop question answering. Teacher-provided reasoning traces are rewritten as graph mind maps and used to guide a student model. Experiments reveal a clear modality gap: flattened textual graph structures provide limited benefits once direct answer hints are removed, while visual graph guidance remains effective without direct answer clues and continues to help after supervised fine-tuning and KL-based distillation.