Manuscript
Selective Risk Control for LLM Decision Agents under Uncertainty
Large language model agents are increasingly asked to gather evidence, make recommendations and call tools. But a model can appear better because it declines difficult cases, not because its active decisions are better. This work makes that distinction the subject of the evaluation.
Research question
When evidence is sparse or contradictory, does binding adversarial critique improve decision behavior—or does it mainly change when an agent abstains?
Experiment overview
A controlled benchmark contains 1,200 evidence bundles and 8,100 evidence snippets. Five decision architectures were evaluated across three seeds, resulting in 18,000 saved local LLM decisions. The protocol compares natural coverage, risk capture, matched-coverage behavior, a thresholded monolithic baseline and abstention-cost sweeps.
| Decision space | Architectures | Controls |
|---|---|---|
| Pass / Kill / Abort | Monolithic, sequential, non-binding critique, binding critique, multi-role | Matched coverage, threshold baseline, abstention penalties, stratified rerun |
Main finding
Binding critique behaves primarily as a conservative risk gate. It catches substantially more difficult cases by abstaining, and improves utility when abstention is inexpensive. At matched coverage, however, it does not consistently improve active-decision performance. A thresholded monolithic policy can outperform it on committed regret at the binding gate’s natural coverage.
Limits
The benchmark is a controlled screening environment and its regret evaluator is a proxy, not real economic value. The paper does not claim formal, distribution-free guarantees. External checks are preliminary, and the central empirical result should be read as a design lesson for evaluation—not as a blanket statement that critique architectures are safer.