In my previous post I discussed my recent poster presentation. Here I’ll talk about some of the ideas that came out of my discussion with attendees (postgraduate psychology students). Is it easy for researchers to adopt Bayesian methods? What do statisticians need to do to facilitate this?
At the presentation I was pleased to see that a lot of people were curious about Bayesian methods and could see the conceptual appeal of updating your hypothesis with the data directly via probability theory. I did note that a big barrier to widespread adoption seems to be the ease of implementation. Having to put serious thought into a prior is an extra burden on researchers who already spend a lot of time designing experiments, considering statictical tests, etc. I think if we want widespread adoption we need to automate where we can and educate where we can’t.
If we aren’t careful, we could end up with poor inference by over-relying on things like uninformative priors when they aren’t appropriate. When confidence intervals were invented Neyman was under no illusion as to what they could and couldn’t do but somehow over the years fallacies of interpretation have dominated . It would be very dangerous and counterproductive to allow this to happen again with Bayesian methodology. For one thing, if the main reason we’re changing to Bayesian methods is because the inference is more intuitive and less prone to error, then we’re going to end up looking pretty foolish if we still have inferential errors despite changing to a more researcher-friendly set of methods. At the moment I hear about a “prior paralysis” where rhetoric about objectivity and using any and all prior knowledge conflict. Researchers want to be objective and not bias result, they expend a lot of energy ensuring this in experimental design. Yet, at the same time they wish to build on on our previous research which is pretty subjective: “we’re looking over here because research suggests there’s something interesting to find”.
A fear I heard a few times concerned “picking the wrong prior” or, as I’ve dubbed it above, “prior paralysis”. To combat this I think training in prior formulation would be helpful at an undergraduate level. Researchers can learn to pick from a large number of statistical tests despite, presumably, not knowing the underlying theory so there’s no reason they can’t learn to construct relatively informative priors with a good enough “recipe”. It might going something like this:
- Consider the likelihood for your data
This should be straight forward as assumptions about data parent distributions are fundamental to frequentist statistics and commonplace when constructing quantitative models.
- Select a conjugate prior if possible, otherwise some reasonable choice
For example, a variance cannot be negative so a distribution allowing it to be negative would be unhelpful. Conjugate priors are those which, when combined with a specfic likelihood, lead to standard distributions for the posterior. Statisticians could develop lists of reasonable starting priors for common parameters in a variety of fields to help with this.
- Consider the most likely value for your parameter and its plausible range
- Translate this into mean and variance
- Translate the mean and/or variance into parameters for your prior
This is of course easier for a standard distibution. It also sometimes easier to just focus on the mean analytically and then fiddle with parameters until it looks reasonable.
Elicitation, the process of extracting priors from researchers, is an interesting topic that I have little knowledge of. I wonder if they use a set process like I describe above? Hopefully I will be able to explore this topic further here in the future. However, there is little doubt in my mind that an informative prior is more useful in most cases than an uniformative one but how to get over the subjectivity issue?
If we’re comparing “informative” and “uninformative” priors we should note that we do not need precise knowledge about the variable to construct an informative prior (if we had precise knowledge why would we be doing the experiement?). It is often closer to excluding irrational values which would otherwise leave us with a prior that puts too much weight on ridiculous values. For example, a Bayesian model for the mean height of adult giraffes may want to put a vague uniform prior on the mean (i.e. uniformative). However it’s bounds will be very much informed by our prior knowledge about adult giraffes. For one thing it should clearly be bounded at 0m because a giraffe cannot be 0m high, equally a 50m high giraffe would be silly. It would appear that giraffes are somewhere around 5m high so we could give a relatively broad uniform prior from 0m to 20m, which gives the posterior enough wiggle room but isn’t putting too much weight on impossible values. We refer to this as a “weakly informative prior”. We could easily create a more informative prior by considering the reasonable range of giraffe heights, probably decided by talking with a giraffe expert, and constructing a normal distribution with the bulk of the data within that range.
If we are worried about the affect of our prior we could include some generic uninformative priors as alternatives and test for prior invariance. This checks that the data wasn’t trying to drag the prior someone you refused to let it go (in our example if the mean height of giraffes in our experiment was 35m our prior may not allow the posterior to move as far as it would if the prior spanned 0m-40m for example). You do need to be careful about the parameter values you set as impossible and assigning a very small probabilty to these values may be more appropriate. This advice is aimed at researchers but I believe statisticians need to play a stronger role in all fields.
On the whole I think the idea of the consultant statistician is becoming ever more relevant. We should be specifically brought on board to take charge of the statisticial analyses and become part of research groups (I do not belive this happens currently, but I could be wrong). PhD statistics students, like myself, should be commonplace outside statistics departments and supporting other PhD students. We can help with their analyses, perhaps developing and applying novel methods to the data as part of our own research, and generate a higher quality statistical awareness in our peers.
Research in 2014  suggested that statistical training did not make research staff in psychology any better equipped to interpret statistics than first-year students with no statistical training. There are many reasons for this (largely the discrepency between what frequentist statistics can tell us and what we want it to tell us), however I don’t believe that switching to Bayesian methods will immediately solve this problem. Researchers largely want a recipe where data goes in and inference comes out. Frankly, who can blame them? I know very little about psychology, but I know a lot about data analysis because I spent 5 years (and counting) studying it. People like me should be doing the leg work and if a recipe is not possible we need to provide training and consultation to scientific researchers as the bulk of our work.
We have a duty to solve the current crisis in statistical inference by supporting researchers as much as possible. We should be more willing to move departments and become involved in specific projects. We are specifically trained to move between vastly different datasets and, more importantly, ask questions and defer to researcher expertise when it comes to the researchers field. In an increasingly interdisciplinary lanscape statisticians are the social butterflies who can immerse themselves in any data-based problem with the right support. Pass me my wings I’ve got data to analyse.
 Morey, R.D. et al., 2015. The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23(1), pp.103–23.
 Hoekstra, R. et al., 2014. Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review, 21(5), pp.1157–1164.