Self-Critique Evaluator Loop
Trains a judge model on its own synthetic comparisons of candidate outputs and uses it as a reward model or quality gate for the main agent
Add to Pack
Saved locally in this browser for now.
Cite This Pattern
Nikola Balic (@nibzard) (2026). Self-Critique Evaluator Loop. In *Awesome Agentic Patterns*. Retrieved October 14, 2026, from https://agentic-patterns.com/patterns/self-critique-evaluator-loop
@misc{agentic_patterns_self-critique-evaluator-loop,
title = {Self-Critique Evaluator Loop},
author = {Nikola Balic (@nibzard)},
year = {2026},
howpublished = {\url{https://agentic-patterns.com/patterns/self-critique-evaluator-loop}},
note = {Awesome Agentic Patterns}
} Use when
- Human-labeled preference data is too expensive or goes stale
- Evaluation must keep pace with model and domain changes
- A small human-labeled anchor set is available for checks
Avoid when
- No human anchor set or adversarial tests to detect evaluator collusion
- Judge criteria for the domain cannot be defined objectively
Problem
Human-labeled preference datasets are expensive to produce, slow to refresh, and quickly stale as base models and domains change. Teams need scalable evaluation signals that can keep pace with model evolution without waiting on large annotation cycles. Risk of evaluator collapse and bias amplification must be mitigated.
Solution
Train a self-taught evaluator that bootstraps from synthetic data:
- Generate multiple candidate outputs for an instruction.
- Ask the model to judge and explain which is better (reasoning trace).
- Fine-tune that judge on its own traces; iterate.
- Use the judge as a reward model or quality gate for the main agent.
- Periodically refresh with new synthetic debates to stay ahead of model drift.
Dual-model variant: using a separate critic model to evaluate the generator (reducing bias at higher cost) is the RLAIF pattern; see that entry rather than restating it here.
To prevent evaluator collapse, keep evaluation prompts and generation prompts partially decoupled, inject adversarial counterexamples, and benchmark against a small human-labeled anchor set.
How to use it
- Start with one narrow domain and define objective judge criteria before training.
- Maintain a fixed holdout set with periodic human audits to detect evaluator drift.
- Use the evaluator as a gate first, then expand to reward-shaping once reliability is proven.
- Track disagreement rates between evaluator and human reviewers.
- Consider dual-model setup (separate critic) for reduced bias in high-stakes domains.
Trade-offs
- Pros: Scales evaluation coverage quickly and reduces dependence on expensive human labeling; trains the judge from synthetic comparisons only.
- Cons: Can overfit to synthetic preferences and needs careful anti-collusion safeguards; risk of evaluator-model collusion.
References
-
Self-Taught Evaluators - Wang et al., 2024 (primary source)
-
Reflexion: Language Agents with Verbal Reinforcement Learning - Shinn et al., 2023
-
Constitutional AI: Harmlessness from AI Feedback - Bai et al., 2022
Distinct from RLAIF: RLAIF uses AI feedback as the training signal for the generator; this pattern trains the judge itself on its own synthetic comparisons.