or, Scope-sensitive snafus in summing speculations

(1)

Should I expect monkeypox to be a big deal for the world? Well, fortunately, Metaculus has a pair of questions that ask users to predict how many infections and deaths there will be in 2022:

203 users(!) made 817 predictions of infections, and Metaculus helpfully aggregates those into a "community prediction" of ~248k infections. 77 users made 180 predictions of deaths, with a community prediction of 541.

The y-axis is on a log scale (as are the predictors' distributions). This is a good choice! Whatever you expect the most-likely case to be, there's definitely a chance with things like this that one a misestimation or shift in one factor can make it bigger or smaller by a multiple, not just an additive amount.

What's not a good choice is to report the median outcome of the aggregate position as the "community prediction". This causes a headline reported value that is way too low. Like, four to seven times too low (at least for my intended purposes).

Because the predictors gave (and are scored on) probability distributions, Metaculus will happily give you an aggregate distribution, of which the 248k "community prediction" is the median scenario (the middle of the three dashed lines):

However, on the same plot, the aggregate distribution predicts a 10% chance of at least 4,950k infections. If it's 10% likely to be 5 million infections, then that's already lot more concerning than the 250k in the community prediction! And when I say I'm interested in how much monkeypox to expect, I

(1)

The following proposition is paraphrased from Savage 1971 (pdf), the original paper that introduced proper scoring rules way back when. Savage characterized the space of all possible scoring rules two ways -- in the now-common form using a general convex function and its subgradient, and in the equivalent "schedule of demands" form that has a natural interpretation as being given the opportunity to trade into a limit order book.

Proposition 1. Any (strictly) proper scoring rule \(S(p,q)\) can be written as \(S(p,q):=\alpha+S^*(p,q)\), where \(\alpha\in\mathbb R\) is a constant and \(S^*(p,q)\) is the profits-or-losses from trading into a limit order book (with nonzero size available at each price) when your initial position is some \(\beta\), your fair value is \(p\), and the contract value resolves to \(q\).

Proof. Let \(\phi:[0,1]\to\mathbb R\) be a (strictly) monotone-increasing function with \(\phi(0)<0\) and \(\phi(1)>0\). We will interpret \(\phi\) as a limit order book, with prices \(a\in[0,1]\):

• \(\phi(a)\) is the cumulative size offered (available to buy) at or below \(a\).
• \(-\phi(a)\) is the cumulative size bid (available to sell) at or above \(a\).

Given \(\phi\), we want to know the profits or losses (PnL) of trading towards a fair value of \(p\).

Let \(a_0\) be either the point where \(\phi\) crosses \(0\), the first point after \(\phi\) jumps across \(0\), or the last point before \(\phi\) jumps across \(0\). This means that \(a_0\) is

In Zvi's 2/11 Covid update, he turned to Metaculus for help. He looked at the numbers. Becase the man is an inveterate trader, he saw odds that were Wrong On The Internet and just couldn't stop himself from creating an account to bet against it. And then he saw the payout structure and decided he was done after making a single prediction.

I spent some time with the Metaculus site and figured out how they borked this one up enough to drive away Zvi Mowshowitz. I'll try to explain it here.

(1)

Here's a presumably current description of the scoring function that I found on the FAQ, slightly abridged:

Your score \(S(T;f)\) at any given time \(T\) is the sum of an "absolute" component and a "relative" component: \[S(T;f)=a(N)\times L(p;f)+b(N)\times B(p;f),\] where \(N\) is the number of predictors on the question.

If we define \(f=1\) for a positive resolution of the question and \(f=0\) for a negative resolution, then \(L(p;f)=\log_2(p/0.5)\) for \(f=1\) and \(L(p;f)=\log_2((1−p)/0.5)\) for \(f=0\). The normalizations \(a(N)=30+10\log_2(1+N/30)\) and \(b(N)=20\log_2(1+N/30)\) depend on \(N\) only.

The "betting score" \(-2\lt B(p;f)\lt 2\) represents a bet placed against every other predictor. This is described under "constant pool" scoring on the Metaculus scoring demo (...)

I'll try to describe that in friendlier terms.

When you make a prediction with \(30n-30=N\) other people, you put your internet points behind two different bets (which get multiplied by 10 to get Metaculus points, but let's skip that for now):

• You bet \(3+log_2(n)\) times against the house, with a

(1)

Political betting site PredictIt offers everyone the ability to (legally) bet (real money) on the outcome of political events. For example:

You can pay 39¢ for a Yes share in BUSH.RNOM16, which will be worth $1 if Jeb Bush wins the Republican nomination, and $0 if he does not. Similarly, you can pay 63¢ for a No share in BUSH.RNOM16, which will be worth $0 if he wins and $1 otherwise. (Another way to think about this is that you can sell a Yes share for 37¢ or buy one for 39¢. These numbers are different for pretty much the same reason that you can't sell your used textbooks back to Amazon for the full price you paid.)

If you have a strong information that Bush is, say, 50% likely to win, then you might buy a lot of Yes shares, and therefore drive the price up. In this way, the market records a sort of weighted-vote among the people willing to bet on the site -- the fact that it's stabilized at 37¢–39¢ should mean, in an efficient market, that all participants agree that the probability that he wins is above 37% and below 39%. If they disagreed, they could make money in expectation by betting their beliefs, which would move (or at least widen) the market.