Forecasting

Superforecasting in plain words

We state probabilities about our own results, then score ourselves. The method comes from the book Superforecasting: The Art and Science of Prediction by Philip Tetlock and Dan Gardner (2015). The idea is simple. Profit alone is a slow, noisy scorekeeper. A forecast that was written down first, with a number on it, can be checked one question at a time.

Start from base rates

A base rate is how often something happened in comparable past cases. If a strategy beat SPY in 4 of its last 10 comparable months, 40% is where the forecast starts. Only then do we ask whether anything today is different. This habit (Tetlock calls it the outside view) keeps a good story from talking us into 90%.

Say a precise number

"Probably" cannot be scored. "68%" can. Our forecasts are numbers between 0.01 and 0.99, written down before the answer is known and kept in git, so they cannot be edited afterward. Every question names the strategy, the event, the window and how it resolves.

Brier score, with an example

The Brier score is the squared miss: (probability minus what happened) squared, where what happened is 1 for yes and 0 for no. 0 is perfect. Lower is better.

A worked example

We say 70% that a strategy beats SPY over the next 20 trading days.

• It does beat SPY: (0.7 - 1) squared = 0.09. A small miss.
• It does not: (0.7 - 0) squared = 0.49. A big miss, because we were confident.
• Always saying 50% scores 0.25 whatever happens.

If our 70% calls really come true 7 times in 10, the average is 0.7 x 0.09 + 0.3 x 0.49 = 0.210, better than 0.25. If they come true only half the time, we score worse than the coin flip, and the scoreboard will say so.

Why compare against a coin flip

A forecaster who always says 50% is never overconfident and never useful. Any real skill has to beat that, so every question we ask is also issued as a coin-flip twin at 0.50. If our Brier score is not lower than the twin's, the forecasts add nothing.

Why we wait for 30 forecasts

Until 30 forecasts have resolved, we do not call the result. One forecast at 90% that fails scores 0.81. With 5 resolved forecasts that one miss moves the average by 0.16; with 30 it moves it by 0.027. Small samples are mostly noise. Repeats of the same question (for example a rerun of the same window) count once, because they are not new evidence.

The two kinds of forecast

Forward questions (every Monday)

For each strategy in the forward test we ask three questions over the next 20 trading days: does it beat SPY, does it beat textbook momentum, and does it fall more than 10% from its starting value. The probability is the base rate over that strategy's own point-in-time backtest of about the previous 9.5 years (rolling 20-day windows). Because the champion was fitted on that history, its base rates are in-sample, so read them with care.

Run questions (every evolution run)

Before a run looks at its unseen days, it forecasts three things about its own champion: beats the benchmark, beats textbook momentum, and ranks at or above the 90th percentile of fair random portfolios. The probability is (hits + 1) / (n + 2) over that arena's earlier scoreboard rows, so a first-ever run starts at 50%. The two families are scored apart.

No AI is involved in these forecasts. They are the strategies' own statistics, written down first.

Scorecard

Mean Brier score of our base-rate forecasts against coin-flip forecasts of the same questions (lower is better; a coin flip scores 0.25). Fewer than 30 resolved forecasts is too few to say anything.

Run forecasts by arena

Calibration

How to read the calibration chart

Calibration asks: of the times we said 30%, did it happen about 30% of the time? The chart below groups our forecasts into bins of stated probability (0.0 to 0.1, 0.1 to 0.2, and so on). Each dot is one bin. Across is what we said, up is how often it came true.

• A dot on the dashed diagonal means the stated odds matched reality.
• A dot below the diagonal means we were overconfident: it happened less often than we said.
• A dot above it means we were too timid.
• A bin with 2 or 3 forecasts can land anywhere by chance. The table under the chart gives each count.

Recent forecasts

20 most recent forecasts.