(Source)
Neil Ferguson's Imperial College epidemiological model has finally been made public. But check out this qualifier.
(https://github.com/mrc-ide/covid-sim)
"The parameter files are provided as a sample only and do not necessarily reflect runs used in published papers."
And check out those parameters. Just as in their published paper, they're all very round. Do they have any relationship to empirical reality, or are they just chosen for the output they produce?
That is, are best estimates of parameters driving the output, or is the desired output + simplicity driving the choice of parameter estimates?
Given that Britain shut down based on this model - and not on the nearly nonexistent empirical evidence for the positive effect of lockdowns - this really matters. If lockdowns are ineffective in combating disease spread, but have substantial human costs, Ferguson bears a large share of responsibility for any unnecessary suffering and death.
I have concerns whether the model is fundamentally right or wrong - in the technical sense of whether the model suffers from misspecification. The question for me is whether the model can be tested by comparing predictions to actual outcomes.
E.g., if a model assumes 75% compliance and predicts 1 million deaths, and in reality, we get 82% compliance and 100 thousand deaths, to me, the problem isn't 75% versus 82% - it's 1 million versus 100 thousand.
I'll let round numbers stand unopposed. Those numbers were obviously hypothetical and illustrative.
The question for me is whether the predictions were at least reasonably accurate or not. We need to test the models by comparing the predictions to actual outcomes.
For example: suppose a model is correctly specified as linear, but you wrongly plug in X = 5 instead of the true X = 5.01. It's obvious that the prediction error won't be very large. But let's suppose that a model is correctly specified as quadratic, but you mis-model it as linear. Plugging in X = 5 instead of X = 5.01 still won't make a great difference - but using 5^1 versus 5.01^2 will make a huge difference.
The model mis-specification - linear versus quadratic - is much more important than X = 5 versus 5.01.
And one sure way to tell if a model has been mis-specified is to compare out-of-sample predictions to out-of-sample outcomes.