or, Scope-sensitive snafus in summing speculations

(1)

Should I expect monkeypox to be a big deal for the world? Well, fortunately, Metaculus has a pair of questions that ask users to predict how many infections and deaths there will be in 2022:

203 users(!) made 817 predictions of infections, and Metaculus helpfully aggregates those into a "community prediction" of ~248k infections. 77 users made 180 predictions of deaths, with a community prediction of 541.

The y-axis is on a log scale (as are the predictors' distributions). This is a good choice! Whatever you expect the most-likely case to be, there's definitely a chance with things like this that one a misestimation or shift in one factor can make it bigger or smaller by a multiple, not just an additive amount.

What's not a good choice is to report the median outcome of the aggregate position as the "community prediction". This causes a headline reported value that is way too low. Like, four to seven times too low (at least for my intended purposes).

Because the predictors gave (and are scored on) probability distributions, Metaculus will happily give you an aggregate distribution, of which the 248k "community prediction" is the median scenario (the middle of the three dashed lines):

However, on the same plot, the aggregate distribution predicts a 10% chance of at least 4,950k infections. If it's 10% likely to be 5 million infections, then that's already lot more concerning than the 250k in the community prediction! And when I say I'm interested in how much monkeypox to expect, I

In Zvi's 2/11 Covid update, he turned to Metaculus for help. He looked at the numbers. Becase the man is an inveterate trader, he saw odds that were Wrong On The Internet and just couldn't stop himself from creating an account to bet against it. And then he saw the payout structure and decided he was done after making a single prediction.

I spent some time with the Metaculus site and figured out how they borked this one up enough to drive away Zvi Mowshowitz. I'll try to explain it here.

(1)

Here's a presumably current description of the scoring function that I found on the FAQ, slightly abridged:

Your score \(S(T;f)\) at any given time \(T\) is the sum of an "absolute" component and a "relative" component: \[S(T;f)=a(N)\times L(p;f)+b(N)\times B(p;f),\] where \(N\) is the number of predictors on the question.

If we define \(f=1\) for a positive resolution of the question and \(f=0\) for a negative resolution, then \(L(p;f)=\log_2(p/0.5)\) for \(f=1\) and \(L(p;f)=\log_2((1−p)/0.5)\) for \(f=0\). The normalizations \(a(N)=30+10\log_2(1+N/30)\) and \(b(N)=20\log_2(1+N/30)\) depend on \(N\) only.

The "betting score" \(-2\lt B(p;f)\lt 2\) represents a bet placed against every other predictor. This is described under "constant pool" scoring on the Metaculus scoring demo (...)

I'll try to describe that in friendlier terms.

When you make a prediction with \(30n-30=N\) other people, you put your internet points behind two different bets (which get multiplied by 10 to get Metaculus points, but let's skip that for now):

• You bet \(3+log_2(n)\) times against the house, with a