Probability & risk

How to reason when outcomes are uncertain: probabilities as beliefs, base rates and Bayes, expected value and its limits, regression and selection effects, fat tails, ruin, position sizing, optionality, compounding and forecasting. The general toolkit is in general thinking, how to turn these numbers into choices is in decision-making, and the mental errors that distort probability judgments are in cognitive biases. Algebra and logarithms are on math fundamentals.

Probabilistic thinking basics

A probability is a number from 0 to 1 that says how strongly you expect something. For one-off events ("will this launch hit 1,000 sign-ups in a month?") it can only be a degree of belief, not a long-run frequency. That is fine: beliefs can still be scored, compared and improved.

Conjunctions shrink fast. A plan with 8 independent steps, each 90% likely, succeeds with probability $0.9^8 \approx 0.43$. Disjunctions grow fast. Ten independent risks at 5% each give a $1- 0.95^{10} \approx 40\%$ chance that at least one bites.

Ranges and confidence intervals

• A 90% interval should contain the true value 9 times in 10. Most people's "90%" intervals are far too narrow: in Alpert and Raiffa's classic studies the true answer fell outside people's supposedly wide intervals far more often than the stated confidence allowed. Overconfidence is one of the most robust findings in judgment research.
• Widen until you would be genuinely surprised by a value outside the range, on either side.
• Anchor the ends separately: "what low value would surprise me?" then "what high value would?".
• For quantities that multiply (market sizes, timelines, returns), think in ratios: "×2 either side" rather than "± 10".

Words vs numbers

Vague words hide disagreement. Sherman Kent (CIA, 1964, "Words of Estimative Probability") proposed a mapping:

Better still: say the number. Words like "likely" or "a real possibility" mean very different things to different listeners, so two people can agree on the wording and disagree widely on the odds.

Fermi estimates (decompose an unknown into factors you can bound, multiply, and check) are the fastest way to get a prior when you have no data. Method and examples are on first principles.

Base rates and Bayes' theorem

The base rate (prior) is how common something is before you look at the specific case. The base-rate fallacy is judging by how well the evidence fits the story while ignoring how rare the story is.

Worked example: a positive screening test

A disease affects 1% of people screened. The test catches 90% of real cases (sensitivity 90%) and wrongly flags 9% of healthy people (specificity 91%). You test positive. What is the chance you have it?

Most people, including many doctors, answer "about 90%". The error is confusing $P(+ \mid D)$ with $P(D \mid +)$.

The same problem in natural frequencies

Gigerenzer and Hoffrage (1995) showed that counts of people are far easier to reason with than conditional probabilities. Imagine 1,000 people:

1,000 people screened
├── 10 have the disease (1%)
│   ├──  9 test positive   (90% sensitivity)
│   └──  1 tests negative
└── 990 are healthy
    ├── 89 test positive   (9% false positives)
    └── 901 test negative
 
Positives: 9 + 89 = 98. Sick among them: 9.
P(disease | positive) = 9 / 98 ≈ 9%

In Gigerenzer et al. (2007), only 21% of 160 gynaecologists picked the right answer to a mammography version of this problem when it was stated as probabilities; after training in natural frequencies, 87% did.

The same structure governs fraud alerts, intrusion detection, "this candidate interviewed brilliantly", and flaky test failures: when the thing you are hunting is rare, most alarms are false.

Bayesian updating as a habit

Bayes is easiest in odds form. Odds are $p/(1-p)$: 1% is 1:99, 50% is 1:1, 75% is 3:1.

The likelihood ratio (LR) asks one question: how much more likely is this evidence if the hypothesis is true than if it is false? Evidence that is equally likely either way (LR = 1) is worthless, however vivid.

Test example again. Prior odds 1:99. LR of a positive result $= 0.90 / 0.09 = 10$. Posterior odds 10:99, which is $10/109 \approx 9.2\%$. A second independent positive multiplies by 10 again: 100:99, about 50%.

The habit

1. Start from a base rate

   Pick a reference class ("seed-stage B2B startups", "migrations of this size in our codebase") and write down its frequency. This is the outside view.

2. Ask what the evidence is diagnostic of

   For each new fact, estimate how likely you would see it if you were right and if you were wrong. Ignore evidence with a likelihood ratio near 1.

3. Update in proportion

   Multiply the odds. Move a little on weak evidence and a lot on strong evidence. Many small updates beat rare lurches.

4. Write it down and revisit

   Record the prior, the evidence and the posterior. Without a record, hindsight rewrites what you believed.

Worked example: will a pilot convert? Of your last 20 enterprise pilots, 6 converted to paid (prior 30%, odds 3:7). The new pilot's champion has budget authority. Looking back, 5 of the 6 converters (83%) had such a champion, as did 4 of the 14 non-converters (29%). LR $= 0.83/0.29 \approx 2.9$. Posterior odds $3/7 \times 2.9 \approx 1.24$, so about 55%. Better than the base rate, still close to a coin flip: do not book the revenue.

Expected value, utility and variance

Expected value is the probability-weighted average outcome:

It is the right yardstick when the bet is small relative to your wealth and repeated many times (pricing insurance across a portfolio, choosing between A/B-test variants, spending engineering hours on bugs). It is the wrong yardstick when one outcome can wipe you out. Decision trees built on expected value are on decision-making.

Expected utility replaces money with how much the money matters to you. Daniel Bernoulli (1738) proposed logarithmic utility, $u(w) = \ln w$, to resolve the St Petersburg paradox: a coin game with infinite expected value that nobody would pay much to play.

Worked example: a positive-EV bet you should refuse

You have $100,000. A bet pays +$100,000 or −$80,000 on a fair coin.

A log-utility investor treats this +EV bet as equivalent to losing $37,000 for sure. The downside cuts deeper than the upside helps because each dollar matters more when you have fewer.

Why the average isn't enough

Always ask for the distribution, not just the mean: the median, the 10th and 90th percentiles, and the worst case. A 1% annual chance of ruin sounds small; over 30 years it is $1- 0.99^{30} \approx 26\%$.

Small samples and regression to the mean

The law of large numbers: as the sample grows, the sample average converges on the true average. Its error shrinks like $1/\sqrt{n}$, so 4× the data only halves the noise.

The "law of small numbers" (Tversky and Kahneman, 1971, Psychological Bulletin) is the mistaken belief that small samples are as representative as large ones. They found that even trained research psychologists expected small studies to replicate far more reliably than the math allows.

The hospital problem (Kahneman and Tversky, 1972): a large hospital has about 45 births a day, a small one about 15. Which records more days when over 60% of babies are boys? The small one, by a wide margin, because small samples swing further. Most people say "about the same".

Regression to the mean

When a measurement is part skill and part luck, an extreme result is likely to be followed by a less extreme one, because the luck does not repeat. Francis Galton described it in 1886 ("Regression toward mediocrity in hereditary stature"): the children of very tall parents were tall, but on average less tall than their parents.

In standardized units, if two measurements correlate with coefficient $r$:

With $r = 0.5$, someone two standard deviations above average this time is expected to be one above next time. With $r = 0$ (pure luck), expect average. With $r = 1$ (pure skill), no regression.

Rule: whenever you select on an extreme and then measure again, you need a control group that was selected the same way and not treated.

Selection effects

A sample is only evidence about the population it was drawn from by the process that drew it. If the process filters on the outcome, the sample lies.

Survivorship bias

In the Second World War, Abraham Wald of Columbia's Statistical Research Group wrote memoranda (around 1943) on estimating how vulnerable each part of an aircraft was from the damage on planes that returned. The insight: hits on returning planes mark places a plane can be hit and survive; the planes hit elsewhere never came back to be counted. The memoranda were published by the Center for Naval Analyses in 1980 and explained by Mangel and Samaniego (JASA, 1984).

Other selection effects

Test: "What would the data look like if my hypothesis were false, and would I have seen it?"

Correlation, causation and Simpson's paradox

Correlation between A and B has at least five explanations. Causation is only one of them.

Ways to get closer to causation: a randomized experiment (A/B test) where possible; a natural experiment or discontinuity; a dose–response relationship; a plausible mechanism; consistency across settings and methods; the cause preceding the effect. These echo Bradford Hill's 1965 considerations for inferring causation in epidemiology.

Simpson's paradox

A trend in every subgroup can reverse when the groups are combined, because the groups are different sizes and the grouping variable is related to both the "treatment" and the outcome.

Kidney stones (Charig et al., BMJ, 1986):

A is better for small stones and for large stones, yet worse overall, because A was given mostly to the harder, large-stone cases.

UC Berkeley, 1973. Graduate admissions looked biased against women in aggregate: about 44% of 8,442 men were admitted versus about 35% of 4,321 women. Bickel, Hammel and O'Connell (Science, 1975) looked department by department and found no pattern of discrimination against women by admissions committees (if anything, a small bias in their favor). Women had applied disproportionately to the more competitive departments with low admission rates for everyone.

Rule: before trusting an aggregate comparison, ask what the groups differ on, and look within strata. Then ask which is the causally right comparison: stratifying on a confounder helps; stratifying on a consequence of the treatment can mislead.

Thin tails, fat tails and power laws

A normal distribution puts about 68% of values within 1 standard deviation, 95% within 2 and 99.7% within 3. Its tails fall off so fast that a 5σ event is essentially impossible. Real markets do not behave like that: on 19 October 1987 the Dow Jones fell 22.6% in one day, a move of more than 20 standard deviations by normal daily-volatility standards.

A power law (Pareto) tail has

Black swans

Taleb's black swan is an event that (1) lies outside regular expectations, (2) has an extreme impact, and (3) is explained after the fact as if it had been predictable. You cannot forecast specific black swans. You can decide in advance how exposed you are to them.

Ergodicity and the ruin problem

A process is ergodic when the average over many people at one moment (the ensemble average) equals the average for one person over a long time (the time average). Most risky choices you make are not ergodic: you live one path, and you cannot average across parallel versions of yourself.

Russian roulette (a favorite example of Taleb's): six people each playing once for $1m have a five in six chance each of winning. One person playing six times has a $(5/6)^6 \approx 33\%$ chance of surviving, and someone who keeps playing is eventually certain to die. Same expected value per round; entirely different fate.

Peters's coin (Ole Peters, "The ergodicity problem in economics", Nature Physics 15, 2019): each round, wealth rises 50% on heads or falls 40% on tails.

After 50 rounds the expected wealth is $1.05^{50} \approx 11.5\times$ the start, yet the typical player has $0.9^{25} \approx 0.07\times$. A tiny minority of lucky paths holds nearly all the ensemble's wealth. For multiplicative bets, maximize the growth rate (the expected log), not the expected value.

Rules for ruin

Position sizing: the Kelly criterion

John Kelly (Bell Labs, 1956, "A New Interpretation of Information Rate") found the fraction of your bankroll to stake on a favorable repeated bet that maximizes the long-run growth rate. For a bet that pays $b$ to 1, with win probability $p$ and loss probability $q = 1 - p$:

It comes from maximizing the expected log of wealth after one bet:

If $f^* \le 0$ there is no edge: don't bet.

Worked example

A repeated even-money bet ($b = 1$) that you win 60% of the time: $f^* = 0.6 - 0.4/1 = 0.2$. Stake 20% of current wealth each round.

Another example: a 2-to-1 payout won 40% of the time: $f^* = 0.4 - 0.6/2 = 0.1$.

Why practitioners use fractional Kelly

• Half Kelly keeps about 75% of the growth with half the volatility. The growth curve is flat near the top and falls off a cliff past it, so undershooting is cheap and overshooting is expensive.
• Your edge is estimated, not known. If you overestimate $p$, full Kelly is really over-Kelly. At 2× the true Kelly fraction, long-run growth is roughly zero or negative.
• Full Kelly is a rough ride. Deep drawdowns are routine: in the standard continuous approximation, a full-Kelly bettor has a 50% chance of seeing their bankroll halve at some point.
• Real bets are not independent coin flips. Correlated positions, fat tails and illiquidity all argue for smaller sizes.

For a founder, Kelly is a way of thinking rather than a formula: size each bet (a hire, a market, a personal guarantee) so that being wrong is survivable and being right still matters.

Risk, uncertainty and margin of safety

Frank Knight (Risk, Uncertainty and Profit, 1921) distinguished risk (outcomes unknown, but the probabilities are measurable, as in dice or actuarial tables) from uncertainty (the probabilities themselves are unknown). He argued that profit is the reward for bearing uncertainty that cannot be insured.

Margin of safety

Benjamin Graham titled the last chapter of The Intelligent Investor (1949) "Margin of Safety as the Central Concept of Investment": buy only well below your estimate of value, so that errors in the estimate and bad luck still leave you whole. Engineers call it a safety factor; founders call it runway. It is the practical answer to Knightian uncertainty: when you cannot trust the probabilities, make sure the plan survives being wrong. Size the margin to the uncertainty of the estimate and the cost of being wrong. Domain-by-domain examples and where it misleads are on general thinking.

Asymmetry, optionality and convexity

An asymmetric bet has limited downside and large upside. Optionality is the right, but not the obligation, to act later when you know more. A payoff is convex when it gains more from good surprises than it loses from bad ones.

Jensen's inequality is the math behind it. For a convex payoff $f$, $E[f(X)] \ge f(E[X])$, so more variability in $X$ raises the average payoff. For a concave payoff the inequality flips and variability hurts.

How to use it:

• Seek convex exposures: cheap experiments, small early bets on many ideas, options on future choices (an MVP before a platform, a contractor before a hire).
• Avoid concave ones you don't get paid enough for: picking up pennies in front of a steamroller, guarantees, uncapped liabilities, single points of failure.
• Value reversibility: a reversible decision is a cheap option; decide fast. An irreversible one deserves the slow process in decision-making.
• The barbell (Taleb): most resources very safe, a small portion in high-upside bets, little in the middle.
• Options have a price. Keeping every door open costs focus. Exercise options (commit) when the information has arrived.

Compounding and the rule of 72

Growth at rate $r$ per period for $n$ periods:

Doubling time solves $(1 + r)^t = 2$:

using $\ln(1 + r) \approx r - r^2/2$ for small $r$. With $r$ in percent, $69.3/r$ is exact for continuous compounding; for annual compounding the $+0.35$ correction means $72/r$ is almost exact near 8% and within a few per cent of the truth from about 4% to 12%. 72 also divides evenly by 2, 3, 4, 6, 8, 9 and 12.

Uses: 7% a year doubles in about 10 years; a product growing 10% a week doubles about every 7 weeks; 3% annual inflation halves purchasing power in about 24 years.

Losses compound too, and asymmetrically:

This is why volatility drags on growth: the geometric mean return is roughly the arithmetic mean minus half the variance ($g \approx \mu - \sigma^2/2$). Money, skills, reputation, code quality and technical debt all compound; so do small daily frictions. More on investing on the investing sheet.

Common statistical traps

Forecasting and scoring

Philip Tetlock's Expert Political Judgment (2005) tracked tens of thousands of predictions by experts over nearly two decades and found most did little better than simple baselines; generalists who drew on many ideas (his "foxes") beat those with one big theory ("hedgehogs"). In the IARPA ACE forecasting tournament (2011–2015), Tetlock and Barbara Mellers's Good Judgment Project recruited volunteers, scored every forecast, and identified the top performers as superforecasters: roughly the top 2%. GJP won the tournament; Good Judgment reports that its forecasts were over 30% more accurate than intelligence analysts with access to classified information. The story is told in Superforecasting (Tetlock and Gardner, 2015).

The Brier score

For binary events, with forecast probability $f_t$ and outcome $o_t \in \{0, 1\}$:

0 is perfect, 0.25 is what always saying 50% earns, 1 is confidently wrong every time. Lower is better. Brier's original 1950 version sums over both outcomes, doubling the score to a 0–2 range; that is the scale Tetlock's books use.

The squared error punishes confident misses hard, and the score is proper: you minimize your expected score only by reporting what you actually believe.

Habits of good forecasters

Paraphrased from Tetlock and Gardner's findings:

Useful ways to practice: keep a forecast log with dates and probabilities, and play on public forecasting platforms such as Metaculus or Good Judgment Open.

Risk checklist

RISK CHECKLIST (run before any bet you can't cheaply undo)
 
Base rate
[ ] What is the reference class, and how often does this work?
[ ] How is my case different, and would I bet on that?
 
Downside
[ ] What is the worst realistic outcome? Is it survivable?
[ ] Could this be ruin: money, reputation, health, trust?
[ ] Does the downside compound, cascade or trigger others?
[ ] Is anything here irreversible?
 
Distribution
[ ] Thin- or fat-tailed domain? Am I using averages where
    tails dominate?
[ ] What are the median, 10th and 90th percentile outcomes?
[ ] Are my risks correlated (one customer, one cloud, one
    founder, one bank)?
 
Evidence
[ ] Is my data survivorship- or selection-biased?
[ ] Could this be regression to the mean?
[ ] Confounders? Is the aggregate hiding a Simpson reversal?
[ ] How many things did I test before this looked good?
 
Sizing
[ ] Is the position small enough that being wrong is fine?
[ ] Am I estimating the edge honestly (use half of it)?
[ ] Does leverage turn a bad outcome into ruin?
 
Shape
[ ] Is the payoff convex (capped loss, open upside)?
[ ] Can I buy an option (pilot, prototype, trial) first?
[ ] Where is my margin of safety, and how big is it?
 
Record
[ ] Written probability, range and reasoning, dated.
[ ] Named signals that would make me change my mind.
[ ] Review date set.

References

• Amos Tversky and Daniel Kahneman, "Belief in the law of small numbers" (opens in a new tab), Psychological Bulletin 76(2), 1971: small-sample intuitions of researchers
• Daniel Kahneman and Amos Tversky, "Subjective probability: a judgment of representativeness" (opens in a new tab), Cognitive Psychology 3(3), 1972: the hospital problem
• Gerd Gigerenzer and Ulrich Hoffrage, "How to improve Bayesian reasoning without instruction: frequency formats" (opens in a new tab), Psychological Review 102(4), 1995: natural frequencies
• Gerd Gigerenzer et al., "Helping doctors and patients make sense of health statistics" (opens in a new tab), Psychological Science in the Public Interest 8(2), 2007: the gynaecologist study and risk communication
• Francis Galton, "Regression toward mediocrity in hereditary stature" (opens in a new tab), Journal of the Anthropological Institute 15, 1886: origin of regression to the mean
• Daniel Kahneman, Thinking, Fast and Slow (Farrar, Straus and Giroux, 2011): chapter 17 on regression and the flight instructors
• Marc Mangel and Francisco Samaniego, "Abraham Wald's work on aircraft survivability" (opens in a new tab), Journal of the American Statistical Association 79(386), 1984: what Wald actually did
• P. J. Bickel, E. A. Hammel and J. W. O'Connell, "Sex bias in graduate admissions: data from Berkeley" (opens in a new tab), Science 187, 1975: the Simpson's paradox case
• C. R. Charig et al., "Comparison of treatment of renal calculi by open surgery, PCNL and ESWL" (opens in a new tab), BMJ 292, 1986: the kidney-stone data
• Joseph Berkson, "Limitations of the application of fourfold table analysis to hospital data" (opens in a new tab), Biometrics Bulletin 2(3), 1946: Berkson's paradox
• Nassim Nicholas Taleb, The Black Swan: The Impact of the Highly Improbable (Random House, 2007): Mediocristan, Extremistan, black swans
• Ole Peters, "The ergodicity problem in economics" (opens in a new tab), Nature Physics 15, 2019: time vs ensemble averages
• Jason N. Doctor, Peter P. Wakker and Tong V. Wang, "Economists' views on the ergodicity problem" (opens in a new tab), Nature Physics 16, 2020: the expected-utility reply to Peters
• J. L. Kelly Jr, "A new interpretation of information rate" (opens in a new tab), Bell System Technical Journal 35(4), 1956: the Kelly criterion
• Frank H. Knight, Risk, Uncertainty and Profit (Houghton Mifflin, 1921): risk vs uncertainty
• Benjamin Graham, The Intelligent Investor (Harper, 1949): margin of safety
• Chris Dixon, "Performance data and the 'Babe Ruth effect' in venture capital" (opens in a new tab) (a16z, 2015): power-law venture returns
• Marc Alpert and Howard Raiffa, "A progress report on the training of probability assessors", in Kahneman, Slovic and Tversky (eds), Judgment under Uncertainty: Heuristics and Biases (Cambridge University Press, 1982): overconfident intervals
• Ward Edwards, "Conservatism in human information processing", in B. Kleinmuntz (ed.), Formal Representation of Human Judgment (Wiley, 1968): under-updating on evidence
• Joseph P. Simmons, Leif D. Nelson and Uri Simonsohn, "False-positive psychology" (opens in a new tab), Psychological Science 22(11), 2011: p-hacking
• Andrew Gelman and Eric Loken, "The garden of forking paths" (opens in a new tab) (2013): data-dependent analysis
• Joshua B. Miller and Adam Sanjurjo, "Surprised by the hot hand fallacy? A truth in the law of small numbers" (opens in a new tab), Econometrica 86(6), 2018: the hot-hand reassessment
• Thomas Gilovich, Robert Vallone and Amos Tversky, "The hot hand in basketball" (opens in a new tab), Cognitive Psychology 17(3), 1985: the original hot-hand study
• Glenn W. Brier, "Verification of forecasts expressed in terms of probability" (opens in a new tab), Monthly Weather Review 78(1), 1950: the Brier score
• Philip E. Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction (Crown, 2015): the Good Judgment Project
• Philip E. Tetlock, Expert Political Judgment (Princeton University Press, 2005): foxes and hedgehogs
• Sherman Kent, "Words of Estimative Probability" (opens in a new tab) (CIA, 1964): mapping words to numbers