Cross-family LLM judges
Research
Trigger I wanted to know whether an LLM can rank short-video openings before they are published, and how far to trust it.
Start from the evidence
Before writing a prompt I had four research agents collect what is known about LLMs predicting clicks and engagement. No model is naturally good at it. On the Upworthy A/B benchmark, GPT-4 reached 38% against a 33% baseline, and persona panels did worse than a single plain prompt.
Design
Binary floor checks first, then blind pairwise picks among the survivors in both orders, keeping only swap-consistent picks. Three model families, because judges from one family agree with each other far more than with reality. Every number in a candidate has to be literally true.
Result on real videos
Five Shorts from one channel: a hit at 6.3x the channel median, two medians and two flops. Claude got 90% of pairs right, Gemini 78%, Qwen 52%. Pooled, 73% with Kendall τ +0.80.
Both flops were Kubernetes videos and the hit was about Discord, so topic reach probably explains both the views and the accuracy. The write-up says so next to the numbers.