Another take on the Hyvärinen score for model comparison
This article is originally published at https://statisfaction.wordpress.com
- Exact log-Bayes factors (log-BF) and H-factors (HF) of M1 against M2, computed for 100 independent samples (thin solid lines) of 1000 observations generated as i.i.d. N(1,1), under three increasingly vague priors for θ1.
In a former post, Pierre wrote about Bayesian model comparison and the limitations of Bayes factors in the presence of vague priors. Here we are, one year later, and I am happy to announce that our joint work with Jie Ding and Vahid Tarokh has been recently accepted for publication. As way of celebrating, allow me to give you another take on the matter. Given some observations 
generated as independent draws from 
, consider the seemingly simple task of choosing between the following two Normal models
: 
, 
,
: 
, 
,
where 
is a known hyperparameter that controls how vague the prior on 
is. In such a setting, only model is well-specified (in the sense of containing the true data generating process), and we can reasonably demand from any sensible model selection criterion to be able to correctly select over . The log-Bayes factor, defined as 
, fits the bill asymptotically, as it is almost surely positive (thus choosing ) for all 
large enough. However, for any fixed sample size (no matter how large), making the prior on more vague by increasing 
drives 
to 0 and 
to 
, which has the undesirable effect of tricking the log-Bayes factor into preferring the misspecified model . This is illustrated with 
in the above picture, where 
is increased from 0 to 150, then 350. The alarming point is not so much that the value of 
varies when the prior changes (as it should, mathematically speaking), but rather that these variations can be unbounded and arbitrary, even for changes of prior that have no impact on the inference stage. This phenomenon is concerning when using Bayes factors to perform model selection, since the choice of one prior is able to dictate the conclusion of the procedure even though the “fit” of the model is virtually unchanged beyond a certain vagueness: indeed, as increases, the posterior 
stabilizes to 
, leading the posterior predictive distributions 
given by 
to essentially coincide with the data generating distribution, for any large fixed and all priors with large enough; yet, as soon as becomes too large, the Bayes factor mechanically (and wrongly) rules out against other misspecified models.
One possible solution to address this sensitivity to priors’ vagueness is to take a decision theoretic approach, by regarding the Bayes factor as a decision rule consisting in choosing the model 
that minimizes the prequential score 
, for the particular choice of score function 
, known as the log-score. A score function has an associated divergence function ![D_S(p,q) = \mathbb{E}_{p}[S(Y, q)-S(Y, p)]](https://s0.wp.com/latex.php?latex=D_S%28p%2Cq%29+%3D+%5Cmathbb%7BE%7D_%7Bp%7D%5BS%28Y%2C+q%29-S%28Y%2C+p%29%5D&bg=fff&%23038;fg=222&%23038;s=0 "D_S(p,q) = \mathbb{E}_{p}[S(Y, q)-S(Y, p)]")
, which quantifies the expected loss incurred when trying to use a distribution 
to predict a random outcome 
, whose true distribution is 
. In particular, the divergence associated to the log-score is the well-known Kullback-Leibler divergence ![\text{KL}(p,q) = \mathbb{E}_p[\log p(Y) - \log q(Y)]](https://s0.wp.com/latex.php?latex=%5Ctext%7BKL%7D%28p%2Cq%29+%3D+%5Cmathbb%7BE%7D_p%5B%5Clog+p%28Y%29+-+%5Clog+q%28Y%29%5D&bg=fff&%23038;fg=222&%23038;s=0 "\text{KL}(p,q) = \mathbb{E}_p[\log p(Y) - \log q(Y)]")
. Finding an alternative to the Bayes factor thus becomes a matter of using a different divergence, and Dawid & Musio (2015) suggest using
![D_\mathcal{H}(p,q) = \mathbb{E}_p[\,\|\nabla\log p(Y) - \nabla\log q(Y)\|^2\,]](https://s0.wp.com/latex.php?latex=D_%5Cmathcal%7BH%7D%28p%2Cq%29+%3D+%5Cmathbb%7BE%7D_p%5B%5C%2C%5C%7C%5Cnabla%5Clog+p%28Y%29+-+%5Cnabla%5Clog+q%28Y%29%5C%7C%5E2%5C%2C%5D&bg=fff&%23038;fg=222&%23038;s=0 "D_\mathcal{H}(p,q) = \mathbb{E}_p[\,\|\nabla\log p(Y) - \nabla\log q(Y)\|^2\,]")
,
sometimes referred to as the relative Fisher information divergence. Its key feature is that, contrary to the Kullback-Leibler divergence, its value remains unchanged when multiplying 
by an arbitrary constant (which is effectively what happens when making a prior more vague, as its normalizing constant becomes arbitrary). The associated score is the Hyvärinen score, defined as
.
One would then select the model minimizing the H-score, defined as 
, which is now robust to any arbitrary vagueness in the specification of priors, as shown in the picture (where H-factors simply refer to differences of H-scores). In practice, H-scores need to be estimated. This can be achieved consistently using sequential Monte Carlo methods (e.g. IBIS when the likelihood is available, or SMC2 for state-space models with intractable likelihoods). We can also prove (under strong assumptions) that the H-score leads to consistent model selection. If interested, more can be learned here; the code for the figure is provided here.
Thanks for reading !
Thanks for visiting r-craft.org
This article is originally published at https://statisfaction.wordpress.com
Please visit source website for post related comments.
