TRRAIL and Error: Evaluating Multi-Agent NeSy LLMs for Logic, Law, and Medical Question Answering

Abstract

Logical reasoning remains challenging for LLMs, particularly in domains where intermediate reasoning must be inspected and checked. Neuro-Symbolic (NeSy) methods such as SymbCoT, LogicLM, and ProSLM expose structured IRs, but many use static pipelines and are evaluated mainly on logic benchmarks. We investigate multi-agent self-correction in NeSy question answering across law, medicine, and logical reasoning, using Chain-of-Thought (CoT) as a baseline and gpt-5.4-nano and gpt-5.3-chat-latest as backbones. We also introduce the TRRAIL score, an LLM-as-judge rubric for observable reasoning artifacts. TRRAIL abbreviates Transparent, Reasoning-relevant, Reasoning-grounded, Answer-supported, Interpretable, and Logical. Across the tested configurations, refinement benefits the Prolog pipelines, whereas Lean refinement decreases several task and TRRAIL metrics. The results therefore support refinement as a configuration-dependent mechanism rather than a universal improvement. Backbone capability and judge choice also materially affect measured performance and TRRAIL scores, underscoring that artifact verifiability, predictive accuracy, and human trust are distinct questions.

Publication
Advances in Cognative Systems
Date
Links