Researchers at MegaOne AI, in collaboration with the University of California, Berkeley, have published a new study on the asymmetric effects of multi-agent feedback in logic proof tutoring, available on arXiv as arXiv:2603.27076. The paper, titled “When Verification Hurts: Asymmetric Effects of Multi-Agent Feedback in Logic Proof Tutoring,” investigates the reliability of large language models (LLMs) in providing step-level feedback for propositional logic proofs, a domain requiring precise symbolic reasoning.
The study, led by MegaOne AI Senior Research Scientist Dr. Anya Sharma, explored how different configurations of LLM agents providing feedback impact student learning outcomes. The researchers focused on propositional logic proofs, a structured symbolic domain where correctness is objectively verifiable. They designed an experimental setup where students received automated feedback on their proof steps from various LLM configurations.
A key finding was the asymmetric impact of feedback types. While positive feedback from LLMs, confirming correct steps, consistently improved student performance, negative feedback, indicating errors, often led to detrimental effects. Specifically, students receiving negative feedback from LLMs showed a 15% decrease in their ability to complete subsequent proofs correctly compared to a control group receiving no negative feedback. This suggests that poorly calibrated or unhelpful negative feedback can hinder learning in symbolic reasoning tasks.
The research also quantified the performance of different LLM architectures in generating feedback. A fine-tuned GPT-4 variant achieved an 88% accuracy rate in identifying correct proof steps, while a general-purpose LLM without specific fine-tuning for logic proofs achieved only 62% accuracy. This highlights the importance of domain-specific training for LLMs deployed in educational contexts, especially for tasks requiring high precision.
Furthermore, the study investigated the “over-correction” phenomenon, where students, after receiving negative feedback, would often revert to previously incorrect steps or abandon valid reasoning paths. This behavior was observed in 25% of instances where students received negative feedback, indicating a potential for LLM feedback to disrupt a student’s problem-solving process rather than guide it effectively.
The findings suggest that while LLMs hold promise for automated tutoring in structured domains, careful consideration must be given to the design and delivery of feedback, particularly negative feedback. Future work will focus on developing adaptive feedback mechanisms that can dynamically adjust the type and intensity of feedback based on student progress and error patterns to mitigate these asymmetric effects.
