or, Scope-sensitive snafus in summing speculations

(1)

Should I expect monkeypox to be a big deal for the world? Well, fortunately, Metaculus has a pair of questions that ask users to predict how many infections and deaths there will be in 2022:

203 users(!) made 817 predictions of infections, and Metaculus helpfully aggregates those into a "community prediction" of ~248k infections. 77 users made 180 predictions of deaths, with a community prediction of 541.

The y-axis is on a log scale (as are the predictors' distributions). This is a good choice! Whatever you expect the most-likely case to be, there's definitely a chance with things like this that one a misestimation or shift in one factor can make it bigger or smaller by a multiple, not just an additive amount.

What's not a good choice is to report the median outcome of the aggregate position as the "community prediction". This causes a headline

In Zvi's 2/11 Covid update, he turned to Metaculus for help. He looked at the numbers. Becase the man is an inveterate trader, he saw odds that were Wrong On The Internet and just couldn't stop himself from creating an account to bet against it. And then he saw the payout structure and decided he was done after making a single prediction.

I spent some time with the Metaculus site and figured out how they borked this one up enough to drive away Zvi Mowshowitz. I'll try to explain it here.

(1)

Here's a presumably current description of the scoring function that I found on the FAQ, slightly abridged:

Your score \(S(T;f)\) at any given time \(T\) is the sum of an "absolute" component and a "relative" component: \[S(T;f)=a(N)\times L(p;f)+b(N)\times B(p;f),\] where \(N\) is the number of predictors on the question.

If we define \(f=1\) for a positive resolution of the question and \(f=0\) for a negative resolution, then \(L(p;f)=\log_2(p/0.5)\) for \(f=1\) and \(L(p;f)=\log_2((1−p)/0.5)\) for