The Thirteenth Stroke

Ninety-eight paid volunteers from a community research pool, average age about twenty-four, were split into advisors and guessers. The advisors looked at photographs of people and estimated their weight, spreading their confidence over several weight ranges. The guessers, paid by a scoring rule for getting the weight right, chose which advisor to listen to in each of eight rounds, and each advisor earned two dollars multiplied by the share of guessers who picked him. By round eight the advisors' peak confidence had risen from 52 per cent to 65 per cent and their ranges had narrowed. The range an advisor backed most contained the true weight 15 per cent of the time. The guessers chose on stated confidence, not on whether the advisor had just been right, and 88 per cent of them said so afterwards. Joseph Radzevick and Don Moore, who ran it, concluded that customers in such a market are "getting what they paid for."

My verdict on the literature has two halves. The room is right to prefer the confident advisor wherever the speaker has nothing to gain from inflating, because there confidence tracks knowledge, and a paid advisor fails that condition by construction. The premium he collects lasts exactly as long as nobody checks: when a confident claim tied to a checkable fact is shown wrong, the confident advisor loses more credibility than the hedged one and the room switches sides. Radzevick's guessers saw each round how their advisor had done and kept buying confidence, and the reason, I think, is that they had nobody to switch to: the advisors there were overprecise as a group, and a miss was a range the weight fell outside rather than a commitment that failed. The reversal has two conditions: confidence stated in words and attached to a specific claim, and a check that reaches the listener. Confidence carried in a voice or a posture survives being wrong unless it was pinned to the claim that failed, and an advisor whose claims are never scored is never caught.

The Room Is Not Wrong

Paul Price and Eric Stone showed undergraduates two fictional financial advisors each judging 24 stocks. One was reasonably calibrated. The other had every judgment pushed 15 points toward the extreme, with the number of correct calls held identical, so both were right 75 per cent of the time. Across three samples, 71, 64 and 63 per cent chose to hire the extreme one. In the third sample they could see he was the more overconfident of the two and still estimated his percentage correct as higher. The authors' reading was that people "assume that a more confident advisor makes more categorically correct judgments and is more knowledgeable."

Before calling that a bias, look where it comes from. Briony Pulford and colleagues paired British undergraduates on perceptual tasks, gave one partner strong evidence and the other weak, paid both for the pair's joint answer, and let them talk. The better-informed partner was more confident and carried the decision, on 86 per cent of trials against 8 in the second experiment. Moving pairs from a table to a Slack text channel made no significant difference, and they wrote that "confidence is communicated primarily through verbal signals and not through facial expressions, gestures, and tone of voice." In a cooperative setting the heuristic is how knowledge moves from the person who has it to the person who does not.

So the room has reasons. Every one of them was earned in a setting where the speaker and the listener wanted the same thing, as Pulford's paid pairs did, and where most people happened to be right. The second condition is quieter. Asher Koriat built virtual pairs from individual answers and took the more confident member's answer each time: the pairs beat the better individual, but because confidence "monitors the consensuality rather than the accuracy of a decision," where most people were wrong the confident head gave a worse answer than the better head alone. Pulford's group notes that the heuristic is efficient only under a norm linking expressed confidence to actual confidence. Price and Stone's advisors broke that norm by design, and so does anyone whose income depends on being picked.

The Confident Witness Loses More

In 2007 Elizabeth Tenney, Robert MacCoun, Barbara Spellman and Reid Hastie gave 48 undergraduates a written trial summary. The eyewitness said either "Yes sir, absolutely. I'm certain of it" or "No sir, I'm not certain of it," and for half the readers the witness was then shown to have been wrong about a collateral detail, the time of the sighting. With no error on the record the credibility means ran 5.7 for the confident witness against 5.0 for the hedger. With the error they crossed, 4.6 against 5.2. The paper tests the crossover, not each gap on its own. A second experiment asked 103 students which of two opposing witnesses they believed, before and after both were caught on one peripheral detail. First time, 75.5 per cent sided with the confident witness; second time, 58.4 per cent sided with the hedger. Both lost credibility, the confident one more. The hedger's final lead, 3.0 against 2.8, was too small to count as a finding, so the robust result is not that the hedger ends ahead but that the confident witness falls further and the room changes its mind. The paper ends: "Otherwise, the thirteenth stroke of the clock will cast the other twelve in doubt." This essay is named for that sentence: one confident claim holds every other claim in the document hostage.

The result has been reproduced by people with no stake in it. Melinda Pozzi and Diana Mazzarella preregistered a replication of Tenney's 2008 design and ran it on Prolific with adults, 108 and 109 of them, reading written depositions for about seven minutes. Before any check the confident witness was rated 4.38 against 3.79. Sixty-three per cent believed him. When both witnesses were shown wrong on the checked claim, the confident one fell to 2.67 and the hedger rose to 4.10. Twenty-two per cent still believed the confident one. Shown that his claim had rested on strong evidence, the confident witness recovered to parity and was again believed by 63 per cent, though never back to his starting rating. In both experiments the hedger was never penalised at any stage; "underconfidence is not damaging," the authors wrote.

Colin Vullioud, Fabrice Clément, Thom Scott-Phillips and Hugo Mercier filmed actors giving identical wrong advice. One said "You can believe me, I'm sure it's him." The other said "I don't know but I think... I'm really not sure." Of the Swiss undergraduates asked, 85 per cent followed the confident sender. Once the advice was shown wrong, 83 per cent would punish him and 69 per cent would reward the hedger. When both senders had been right, the pattern inverted and only 22 per cent wanted to punish the confident one, so what they punished was a commitment that failed. Against a sender introduced with a stated record of twelve right out of thirteen, the confident sender was followed about equally. After one identical miss, trust in the sender with the record held at 70 per cent in the incentivised version, and most of the confident sender's followers switched in the unincentivised one, a gap that was significant in both experiments and their doubled replications.

Stated numbers behave like stated words. Sunita Sah, Don Moore and Robert MacCoun had 184 participants estimate weights from photographs, Radzevick and Moore's task. The advisor claimed either 85 to 100 per cent or 25 to 35 per cent confidence, and after each of five photographs he was revealed to have been right or wrong. This room had what the market lacked, a hedged advisor and a miss shown as a miss. The confident-but-wrong advisor got the lowest credibility rating, 2.23 against 2.84 for the hedger who was equally wrong; in the paper's summary, "while accurate advisors benefit from displaying confidence, confident but inaccurate advisors receive low credibility ratings." The second half of that sentence held; the first did not reappear when the cell was rerun in their next study. One detail cuts the other way: how far people moved their estimates toward the advice tracked accuracy and not confidence, so the penalty showed in what they said about the advisor, not in what they did with his numbers.

Confidence Without a Claim

Everything above depends on the confidence being pinned to a claim that can fail. Tenney, with Nathan Meikle, David Hunsaker, Moore and Cameron Anderson, tested what happens when it is not. Four hundred and forty-four MTurk workers, in a preregistered design, evaluated a candidate for a collaboration whose overconfidence was conveyed either in written statements or in a video of his demeanour. In both channels the overconfident candidate won at first: 68 per cent chose him. After his actual performance was revealed, 60 per cent of those who had read his words now preferred the cautious candidate, while 67 per cent of those who had watched his video stayed with him. "Nonverbal overconfidence reaps reputational benefits because of its plausible deniability," the authors wrote, and video confidence lost its advantage only when it was "clearly tied to a falsifiable claim." The channel made no significant difference for Pulford because her partners had evidence to exchange; Tenney's observers had only the candidate to look at.

Jessica Kennedy, Anderson and Moore found the same escape in live groups. When members learned one another's actual scores, those who had overrated their own rank were not demoted. Their lead shrank, no penalty followed, and over the whole session they still came out ahead. Their studies, the authors wrote, "feature people displaying nonverbal signals of confidence without the explicit and falsifiable claims of being right on a particular item." A check must also be used. Giving adults a second task while they judged informants "impaired their ability to make use of calibration information," Tenney's group found in 2011, where Pozzi and Mazzarella's unhurried adults used it reliably, so load is a boundary rather than the default.

One further result trims my case for hedging. Karmarkar and Tormala's 2010 claim that an expert gains influence by admitting uncertainty did not survive a Registered Report. Erik Løhre and colleagues put 1,018 American adults in front of a restaurant review by either a renowned food critic or a community-college network administrator. The review gave "a confident 4 out of 5" or "a tentative 4 out of 5," and they found no interaction. Certainty helped the expert, with an effect size of 0.28, though small enough that "there is not much to lose from being open about uncertainty," which is a different claim from gaining by it.

The courtroom adds a ceiling. When Robert Cramer, Stanley Brodsky and Jamie DeCoster showed 299 mock jurors filmed expert testimony in a capital sentencing, the expert who said "I am reasonably certain" in a steady voice was rated more credible than the one who said "I am certain" of every statement, loudly and leaning forward; the two were rated equally confident, so the penalty ran through trustworthiness and likability.

Where Nobody Checks

In a second experiment with 80 new participants, Radzevick and Moore paired each advisor with a single guesser so that nobody competed for anyone. There the advisors' peak confidence stayed flat, 54 to 56 per cent, and the guesser's decision to keep listening tracked how the advisor had done the round before; the advisors were still overprecise, so pairing stopped the escalation without making anyone accurate. Sah, Moore and MacCoun then took the check away. In their second study, 377 participants rated the same photo-weight advisors, and with free feedback the confident-and-wrong advisor was again punished. With no feedback, the confident advisor was rated more credible, 3.54 against 3.20, and moved estimates further, whether or not he had been right. When a look at the answer cost one lottery ticket per photograph, only 44 per cent ever bought one, and they bought fewer on the confident advisors, so the advisor most in need of checking was the one least checked. Sah and her co-authors wrote that "when feedback on advisor accuracy is unavailable or costly, confident advisors hold sway regardless of accuracy."

Advisors know this. In Alex Van Zant's 2022 studies, people advising others on ten rounds of stock predictions stated more confidence than people making the same calls for themselves, and only the private decision-makers adjusted after being wrong. Among 94 working professionals in eight fields from accounting to sales, 37 per cent under a self-promotion incentive said they had deliberately conveyed more confidence than they knew was justifiable. In the comparison condition, 16 per cent said so.

Philip Tetlock's twenty-year study collected 82,361 probability estimates from 284 experts. Those who reported more demand from policy makers, consultants and the media were more overconfident, a correlation of .33. "Experts in demand were more overconfident than their colleagues who eked out existences far from the limelight," Tetlock wrote. The correlation fell to non-significance once his measure of cognitive style was controlled, and he says of the link that "causality surely flows in both directions."

Competition does not always push one way. When fifteen brokerage mergers removed an analyst from a stock's coverage, Harrison Hong and Marcin Kacperczyk found that the remaining analysts' optimism bias rose modestly, "consistent with competition reducing bias." There competition means rival monitors of one public number, not rivalry for a buyer who cannot check. The same scored market pays for something stranger. Across 38,096 analyst-firm observations, Gilles Hilary and Charles Hsu found that the consistency of an analyst's forecast errors predicted his power to move prices, with an effect two to four times the size of accuracy's, and that consistency, not accuracy, made demotion less likely. A legible, repeatable bias was worth more than being right.

I write signed technical verdicts, due-diligence memos and audits of AI systems, each with a stated scope limit and a line saying what would change my mind, and I have lost pitches with that document to vendor slides that carried neither. The buyer in that room is Sah's advisee in the costly-feedback condition: the check exists, it costs something, and the slide gives no reason to buy it.

Four Forecasters at Zierikzee

At Zierikzee in the Netherlands, four experienced forecasters who had never issued probability forecasts and did not know the local base rates spent a year overforecasting every element, threshold and period, with the weather arriving daily and no formal performance feedback. Outcome feedback alone is not the same as a score. In October 1981 Allan Murphy and Harald Daan sat down with each of them and went through his own reliability diagrams and skill scores. In the second year reliability improved for every event and for three of the four men, though all four still overforecast. Skill on wind speed rose from 13.9 to 25.4 per cent. On visibility it rose from 12.4 to 22.4 per cent. The authors could not separate the meeting from the year's experience and said so.

The professions with the best calibration on record are the ones where the score is in the game itself. Gideon Keren wrote that weather forecasters' calibration "can probably be accounted for by, among other things, the prompt and continuous feedback as well as by long enduring practice," and his own data extend the point to bridge. Sixteen national-tournament players, across 448 assessments of whether a contract would be made, were almost perfectly calibrated. Twenty-eight club players with years behind them were overconfident. They said 100 per cent on 29.2 per cent of their assessments. The experts said it on 2.5 per cent. The amateurs had practice and feedback too; experts differ, Keren concluded, "not necessarily in the amount of practice per se, but rather by constructing an appropriate structural representation of the problem."

In a medical outpatient clinic, James Bushyhead and Jay Christensen-Szalanski found physicians diagnosing pneumonia from chest X-rays they did not order for every patient who had it. The doctors came to lean on the clinical signs that looked predictive in that partial record, and those signs predicted pneumonia less well than the signs associated with all the cases; incomplete feedback, the authors wrote, "may perpetuate inefficient chest radiograph ordering patterns." A loop can also be too slow to teach. Over ten years of the Duke CFO survey, more than 13,300 one-year forecasts of the S&P 500 came with 80 per cent confidence intervals. The realised return fell inside them 36.6 per cent of the time, one outcome a year to learn from, bundled with everything else that happened.

Calibration also turns up where no score was ever fed back, which is why I put the weight on the buyer rather than the forecaster. Fifteen analysts in Canada's Intelligence Assessment Secretariat attached numeric probabilities to 3,881 judgments between 2005 and 2011, in a unit David Mandel and Alan Barnes describe as working without systematic feedback on forecasting quality. On the 1,514 that could be scored against an unambiguous outcome they were very well calibrated and if anything underconfident. The Good Judgment Project collected 494,552 forecasts over three years. Confidence averaged 65.4 per cent against a hit rate of 63.3 per cent, and the forecasters were calibrated in their first month and did not improve with tenure. Calibration can exist without a scoreboard. What cannot exist without one is a buyer who knows it is there.

Twenty to Eighty

In March 1951 a National Intelligence Estimate judged that a Soviet-bloc attack on Yugoslavia that year "should be considered a serious possibility." Sherman Kent, who ran the Office of National Estimates, had meant odds of around 65 to 35 in favour; the chairman of the State Department's Policy Planning Staff had read the phrase as odds very considerably lower. Kent polled his Board: "the low man was thinking of about 20 to 80, the high of 80 to 20." His remedy kept the words and tied a short list of them to odds, "probable" at 75 per cent give or take about 12. "Almost certain" sat at 93.

The spread Kent found on his Board is the normal condition of the language. Among 924 readers of eight real sentences from US intelligence reports, in a preregistered experiment, the overlap between what they understood and what Intelligence Community Directive 203 stipulates averaged 32 per cent unaided. Printing the directive's range in brackets after the word lifted it to 66 per cent.

Vagueness has a cost on the forecaster's side too. Jeffrey Friedman, Joshua Baker, Barbara Mellers, Tetlock and Richard Zeckhauser measured it by taking 888,328 Good Judgment forecasts and rounding them to the granularity of words. Collapsing them into two bins made Brier scores 31 per cent worse. Rounding to the seven bins of ICD 203, where "likely" means 55 to 80 per cent, cost under 1 per cent on average. The best forecasters lost about 6 per cent. A Brier score is the mean squared gap between a stated probability and the outcome scored 1 or 0, so that on the 0-to-1 scale a 60 per cent forecast of something that occurs scores 0.16.

The hedge that costs is the qualifier, bolted onto the claim from outside rather than carried inside it. Across eleven preregistered studies with 4,806 participants, Celia Gaertig and Joseph Simmons found that an advisor who prefaced accurate advice with "I am not sure, but..." was rated worse and followed less than a confident one. Advice that carried its uncertainty in the content, a 20-point range or "a 57% chance," was not penalised. In a forced choice 82 per cent preferred the advisor who gave an approximate probability to one who gave certain advice. In Gaertig and Simmons's words, "Advisors benefit from expressing themselves with confidence, but not from communicating false certainty." The advice in those studies was always right, so they measure taste for a form of words rather than reaction to a miss.

In five experiments with 5,780 people, Anne Marthe van der Bles and colleagues found that giving a statistic as a numeric range left trust in the source unchanged. Adding the sentence that "there is some uncertainty around this estimate, it could be somewhat higher or lower" cut trust in the number by about half a standard deviation across the experiments, and trust in the source as well. The study measured trust, not whom the readers would have hired. Gaertig and Simmons's "I am not sure" is a statement about the speaker; van der Bles's "it could be somewhat higher or lower" is a verbal shrug about the estimate; "55 to 80 per cent" is a statement about the world, and listeners treat the three differently.

A Line That Can Be Checked

The written expert document, left to the market, does not state its limits. In 300 biomedical papers from thirty journals, Gerben ter Riet's team found that 27 per cent acknowledged no limitation. Among the 219 that did, 84 per cent did not temper their conclusions, and the team concluded that many authors "appear not to be aware of limitations or are reluctant to admit them even after peer review." The buyer of such a document is buying a credence good, a repair or a taxi ride in a strange city, whose quality the buyer cannot judge even after paying. In the first large experiment on them, 936 students played sixteen rounds of a game in which a seller diagnoses and treats a problem the buyer cannot judge, and Uwe Dulleck, Rudolf Kerschbamer and Matthias Sutter found that competition "drives down prices and yields maximal trade, but does not lead to higher efficiency as long as liability is violated." What raised efficiency was liability, the seller having to deliver sufficient quality. The nearest thing a written verdict has to liability is a claim that someone who reads it can score against the outcome.

From this I take two things the evidence will bear. The first is a number or a range attached to the claim rather than a qualifier attached to the speaker, the form the room has been shown to accept. The second is a sentence naming what would change the verdict, and its warrant is for the writer, not the reader. Asher Koriat, Sarah Lichtenstein and Baruch Fischhoff had 200 paid volunteers answer general-knowledge questions and then write one reason their chosen answer might be wrong, or one reason it might be right, or both. Only the contradicting reason helped; in the authors' words, "Only the listing of contradicting reasons improved the appropriateness of confidence." No study tests whether a reader rewards the line; what it does is make the number beside it worth accepting. The scope limit I also write has no experiment behind it. The witness experiments test a claim that can fail, and a scope limit written as a claim is one, so it can be found wrong like any other.

Training works, cheaply, and not by making people hedge more. Across four years of the IARPA tournaments, a module built to take under an hour, teaching forecasters to start from comparison classes, hunt for information, adjust and update, and write post-mortems, "consistently improved accuracy (Brier scores) by 6 to 11% over the control condition," Welton Chang and his co-authors report, and in two of the four years they could trace the gain to where trained forecasters put their effort. In Lichtenstein and Fischhoff's 1980 training study, twelve participants sat through eleven sessions of 200 questions with a printed calibration graph after each, and nearly all of the improvement came, in their words, "after receipt of the first feedback."

The public record is what makes the score one anyone can read. Of 55 large cardiovascular trials funded by the NHLBI, 57 per cent of those published before 2000 reported a benefit on their primary outcome. Of those published from 2000, all of them registered in advance, 8 per cent did. In psychology, 96 per cent of standard papers testing a hypothesis reported support for it, against 44 per cent of Registered Reports. Both author teams decline a causal reading, and the formats differ in who chooses them. A claim fixed in advance comes out as predicted far less often, so the miss is where anyone can see it, and that is the one job a record does for a scope limit: it turns the limit into a claim that can be scored. A check reaches only confidence pinned to a claim it can falsify, stated rather than performed. A scope limit never read by the person who could compare it with the outcome is a hedge in the costly-feedback condition, where nobody buys the ticket.

The fifteen Canadian analysts wrote their probabilities as numbers in the internal versions of their reports. The finished reports went to readers without them. The forecasts stayed where they were until two outside experts scored them for a study and found that the unit had been better than it had let itself sound.