Underwriting Confidence: A Distinction From the Intelligence World - Lesson 2

One of the first things you are taught in the military intelligence schoolhouse is that an intelligence professional should never express “likelihood” and a “confidence level” in the same sentence. This is one of the intelligence community’s highest written and unwritten analytical standards, and it is a rule that sounds pedantic, but it is not. To an untrained eye, the sentence “we assess with high confidence that an attack is likely” can read like strong, careful, and judicious language. However, if we were to go by the book, it is bad tradecraft. The sentence invites the reader to fuse two separate claims into a single impression of certainty, which can be detrimental. This also allows your reader or audience to walk away knowing less than they think they already know about the situation at hand, which means you have failed the basic underpinning of your job as an analyst. I wrote in my last essay that an intelligence analyst is paid to be calibrated rather than to be certain. This rule is where calibration begins, and I will show you the difference between how to use them correctly.

The two claims deserve to be pulled apart. Likelihood is a statement about the world: will the regime fall, will the convoy move, will the cease-fire hold? Confidence is a statement about the assessment itself: how good is the sourcing, how complete is the picture, how sound is the reasoning, how much would a single new report change our minds or paint the picture more clearly? The two questions are answered by different evidence, and they move independently. You can hold high confidence that something is unlikely; deep, corroborated reporting can tell you a capability simply does not exist, the way thorough diligence can tell you an edge does not. You can hold low confidence that something is likely; thin, fragmentary reporting can all lean the same direction. Both are honest judgments. They demand very different things of the decision-maker.

The most famous now-public example is the Abbottabad compound. In the spring of 2011, CIA and other agency analysts were asked to estimate the probability that Osama bin Laden was living inside; the answers that reached the President reportedly ran from roughly forty percent to ninety-five percent. Same compound, same intelligence, wildly different probabilities. That spread was not a failure of the intelligence collected; it was information. It told the room that the evidence was circumstantial, that no one had direct confirmation, and that the decision would have to be made under genuine uncertainty rather than under the comfortable illusion of a precise number. The likelihood estimates varied. The confidence picture was the real briefing.

As we all know, the finance and investment industry has a rich vocabulary for likelihood. We produce price targets, base cases, bull and bear scenarios, probabilities of recession, and expected returns down to the nth decimal place. For confidence, we mostly have one word: conviction. I truly believe that conviction is doing too many jobs at once, and we need to break it down to be clearer. Sometimes it means the expected outcome is unusually favorable. Sometimes it means the analyst is unusually sure of the estimate. Sometimes it simply means the position is large. A word that can mean any of three things reliably communicates none of them, which I find to be an odd trait in a word that is meant to signal certainty.

From my perspective in the allocator's seat, the problem becomes concrete. Consider that you have two managers. The first has a three-year record of exceptional returns, earned in a single market regime, with attribution you can only partially verify. The second has an eighteen-year record of modest, persistent outperformance, two drawdowns survived and explained, and full transparency into the book. You can call the first manager's prospects favorable if you like; the honest confidence level is low, because the sample is small, the regime is one, and part of the evidence is the manager's own marketing. The second manager earns high confidence in a more modest likelihood. "High conviction" cannot tell these two situations apart. The distinction can, and they demand entirely different decisions.

If we were to pull the axes apart, it would produce four combinations, and each one is an instruction. 1. Favorable likelihood held with high confidence is the rarest: act and size the commitment to match. 2. The combination I find most interesting is a favorable likelihood held with low confidence, because that judgment arrives with its own instructions. In my old world, a gap like that generated a collection requirement, a specific tasking that names exactly what to go find. On the investment side, the allocator's version is the same kind of discipline. Size small or wait. Structure the commitment so that you keep learning, and write down, in advance, what specific “collection requirement” you need in order to raise your confidence. If the evidence comes in, then you can proceed with a clear conscience. If it never does, staying small was the right answer all along. 3. Unfavorable likelihood held with high confidence calls for one of the most underrated skills in allocation, the fast no, declining quickly and without ceremony, so the pipeline stays clear for better work. Lastly, 4. unfavorable likelihood held with low confidence belongs on a watchlist, not in front of a committee.

Confidence also belongs inside position sizing, not just alongside it. My brothers and sisters in finance know this formally. The Black-Litterman model carries a separate input for how strongly you hold each view; the estimation-error literature says the same thing in mathematics. Uncertainty about your estimate should shrink the bet, even when the estimate itself is attractive. Practice rarely honors it. Two commitments with identical expected returns deserve different sizes if one estimate rests on eighteen years of audited evidence and the other on three years and a story. Most sizing frameworks have a column for the return assumption. Very few have a column for how much weight that assumption can bear.

The distinction is what earns its keep most visibly in front of a committee. I used to write for one IC; now I write for another. The acronym survived the career change, and so did the job: tell the room what you think will happen, and tell them, separately, how much weight the judgment can carry. A recommendation that separates the two changes the meeting. Instead of arguing about the conclusion, the committee can interrogate the evidence underneath it, which is a far more productive argument, and it transforms the post-mortem. When a commitment disappoints, the first question that comes up is which judgment failed. If the unlikely simply happened, that is the cost of operating under uncertainty, and the process can stand. If we claimed high confidence on thin evidence, that is a process failure, and it needs to be fixed. Without the distinction, every miss can look identical and teach nothing.

I will offer one honest caveat, because this series should bend where the analogy bends. In intelligence, justified confidence that everyone shares is the ultimate goal; the entire apparatus exists in order to give decision-makers qualified assessments that they can collectively trust. In markets, confidence that everyone shares is already priced in. The return for holding a view the whole market holds with high confidence is, give or take, the market return. What investors get paid for is justified confidence that the consensus does not yet share. An allocator's edge is rarely a different likelihood estimate. It is more often a confidence differential, earned the slow way, through work the crowd has not done. Just like in intelligence, the enemy always gets a vote, and in this industry, so does the consensus.

In closing, the one thing I want you to take away here is the practice that I try to hold myself to, and it is simple to state and demanding to keep. Every judgment gets two sentences instead of one. The first says what I assess will happen and how likely I believe it to be. The second says how much confidence that assessment deserves, and exactly what evidence would change it. Likelihood is a claim about the world. Confidence is a claim about the quality of your own work. Keeping them in separate sentences keeps you honest about both.