Click OracleHow we measured Click Oracle
The headline accuracy numbers come from one run of a frozen engine on held-out data. The baseline comparisons, the memorization checks and the large-set check are separate runs, described below with what was planned in advance and what was added later. Generated Nov 10, 2026 11:04 PM MST.
The data
The Upworthy Research Archive is an open collection of real, randomized headline A/B tests run by Upworthy from 2013 to 2015, published by J. Nathan Matias, Kevin Munger, Marianne Aubin Le Quere and Charles Ebersole (Scientific Data, 2021) under CC BY 4.0. Each test showed different headlines for the same story, with the same image and teaser, to random visitors and counted clicks.
- Tuning data: the archive's exploratory and holdout packages, 4,172 usable tests. Used to choose the question wording and to fit the calibration.
- Test data: the archive's confirmatory package, 9,841 usable tests (43,056 headline versions). Run once, after everything was frozen. No test overlaps the tuning data.
- Cleaning rules (fixed before any model call): only versions that share the same image, so only the headline differs; at least 1,000 impressions per version; exact duplicate headlines merged; 10,068 test groups from June 25, 2013 to January 10, 2014 dropped because the archive's maintainers found randomization problems in that window; 54 groups dropped for a traffic split imbalance (p < 0.001).
What the engine does
For every pair of options it asks the ranking model which one a typical person is more likely to click, once in each order, and averages the two answers so position bias cancels. Up to 24 options, every pair is judged. Above that, each option first meets 8 random opponents, then the top 16 play a full round robin. A Bradley-Terry model turns the head-to-heads into strengths, with an uncertainty for each option, and one calibration scale (0.458, fitted on tuning data only) turns strength gaps into win odds. A call is Clear when #1 is expected to beat #2 at least 62% of the time, Lean from 55%, and Too close to call below that.
The ranking model is TypeSafe's Jev (jev-1.14.0), a decision model that answers typed questions with probabilities. Claude, by Anthropic, writes options when you ask it to. Claude does not rank.
Results on held-out tests
The engine files and calibration were frozen by hash, and the test protocol was written down (in our repository, not with an outside registry) before the single test run. Every result in this section comes from that run.
One randomly chosen pair from each of the 9,841 test groups, so every pair is a separate experiment. Accuracy counts the 2,762 pairs whose real test had a statistically clear winner (two-sided p < 0.05). Ranges are 95% Wilson intervals.
| Band | Share of all pairs | Right on clear-result tests | 95% range |
|---|---|---|---|
| Clear | 19% | 555 / 675 = 82% | 79 to 85% |
| Lean | 39% | 684 / 1,027 = 67% | 64 to 69% |
| Too close | 42% | 624 / 1,060 = 59% | 56 to 62% |
| All | 100% | 67% | 66 to 69% |
Calibration: across all 9,841 pairs, predicted odds and the real outcome (scored as the probability that one version truly had the higher click rate) differed by 0.9 percentage points on average.
Picking the best of a set
On 8,624 test groups with 3 or more versions (average 4.71), our #1 had the highest real click rate 33% of the time (95% range 32 to 34%), against 22% by chance. Its click rate averaged 11% above the average version in the same test. When the verdict was Clear (315 groups), #1 was the real best 51% of the time, with a 27% lift.
Baselines
What we preregistered, and what we changed. The protocol named one comparison model, Claude Haiku, on a seeded subset of 400 independent test pairs. Only about 100 of those pairs had a statistically clear real winner, which made the comparison too small to separate methods. Here are those preregistered results in full, with Sonnet added afterwards:
| Preregistered subset (400 pairs) | Right on clear-result pairs | 95% range | Pairs counted |
|---|---|---|---|
| Click Oracle (all versions of each test ranked) | 57% | 47 to 66% | 100 |
| Jev on the pair alone | 60% | 50 to 69% | 100 |
| Claude Haiku (claude-haiku-5-5) | 54% | 44 to 65% | 81 |
| Claude Sonnet (claude-sonnet-5-5), not preregistered | 67% | 56 to 76% | 78 |
| Text-feature model | 56% | 46 to 65% | 100 |
Pairs counted are clear-result pairs where the method made a pick (Claude often answered exactly 50%). On this small set Click Oracle scored below Sonnet and below Jev on the pair alone, with wide, overlapping ranges.
The larger comparison (added after seeing the result above). We then drew a second seeded sample of 600 held-out pairs that all had a statistically clear real winner, and ran every method on it. This set was chosen after the preregistered run, so treat it as a follow-up, not a preregistered result. Claude saw each pair in both orders, asked for a probability, 25 pairs per request. Click Oracle ranked all versions in each test, as it does in the product, and the pair was read from that ranking; Jev on the pair alone is shown for a like-for-like comparison. Ties count as half right.
| Method (600 pairs) | Right | 95% range | Confident calls |
|---|---|---|---|
| Click Oracle | 65% | 61 to 69% | 144 pairs, 82% right |
| Jev on the pair alone | 66% | 62 to 70% | same model |
| Claude Sonnet (claude-sonnet-5-5) | 67% | 63 to 71% | 30 pairs, 90% right |
| Claude Haiku (claude-haiku-5-5) | 55% | 51 to 59% | few |
Sonnet scored higher, but the difference is within noise (paired McNemar test, p = 0.40); that does not show they are equal. Click Oracle beat Haiku (p < 0.001). Measured model cost per pair in these runs: about $0.0000213 for Click Oracle's model, $0.0000827 for Haiku and $0.00119 for Sonnet (list prices), so Sonnet cost about 56 times more. On all decisive held-out pairs, a logistic model on simple text features (trained on tuning data) was right 58% (2,755 pairs where it made a pick) and "the longer headline wins" 53% (2,657 pairs with different lengths).
Could the model have memorized this archive?
The archive has been public since 2019, so the ranking model may have seen these headlines in training. We could not find a public set of headline A/B outcomes recent enough to rule that out. Two checks:
- Naming the source: telling the model the headlines came from Upworthy in 2013 to 2015 did not help: 65% against 66% without the name on the 600-pair set (p = 0.75), and 48% against 60% on the preregistered 400-pair subset. Recall of the archive would be expected to improve with the cue.
- Rewritten headlines: Claude rewrote both headlines of 200 decisive pairs with the same meaning and devices but different words. Accuracy went from 68% to 64%, and 79% of picks stayed the same. A memorizing model would be expected to fall toward 50%.
Neither check proves the absence of memorization. That is why per-account calibration exists: upload your own recent tests and you get accuracy measured on data no model could have seen.
Limits
- One publisher, one audience, one decade-old style of headline. Other formats (YouTube titles, subject lines, ads, hooks) use the same engine but were not measured here.
- Close variants are hard for everyone. About 42% of comparisons are Too close to call, and the label means it.
- For YouTube, the thumbnail drives much of the click. Click Oracle ranks text only.
- Large sets (over 24 options) use a tournament, not every pair. On 3 Claude-written sets of 120 options, compared with judging every pair, the tournament's #1 was the full round robin's #1 in 2 of 3 sets and in its top 3 in 3 of 3, the two top 10s shared 9 of 10 options on average, and rank correlation was at least 0.987, using about 1/11 of the comparisons. In the one set where #1 differed, the top 3 were near rewordings of each other and the verdict was Too close to call. This checks the design, not real outcomes, because the archive has no tests that large.