Hypothesis testing and conditional inference

Ben Hansen sent me this paper by Donald Pierce and Dawn Peters, about which he writes:

I [Ben] stumbled on the attached paper recently, which puts forth some interesting ideas relevant to whether very finely articulated ancillary information should be conditioned upon or coarsened. These authors’ views are clearly that it should be coarsened, and I have the impression the higher-order-asymptotics/conditional inference people favor that conclusion.

The background on this is as follows:

1. I remain confused about conditioning and testing. I hate the so-called exact test (except for experiments that really have the unusual design of conditioning on both margins; see Section 3.3 of this paper from the International Statistical Review).

2. So I’d like to just abandon conditioning and ancillarity entirely. The principles I’d like to hold (following chapters 6 and 7 of BDA) are to do fully Bayesian inference (conditional on a model) and then use predictive checking (based on the design of data collection) to check the fit.

3. But when talking with Ben on the matter, I realized I still was confused. Consider the example of a survey where we gather a simple random sample of size n, fit a normal distribution, and then test for skewness (using the standard test statistic: the sample third moment, divided by the sample varicance to the 3/2 power). The trick is that, in this example, the sample size is determined by a coin flip: if heads, n=20, if tails, n=2000. Based on my general principles (see immediately above), the reference distribution for the skewness test will be a mixture of the n-20 and the n=2000 distribution. But this seems a little strange, for example, what if we see n=2000–shouldn’t we make use of that information in our test?

4. In this particular example, I think I can salvage my principles by considering a two-dimesional test statistic, where the first dimension is that skewness measure and the second dimesion is n. Then the decision to “condition on n” becomes a cleaner (to me) decision to use a particular one-dimensional summary of the two-dimensional test statistic when comparing to the reference distribution.

Anyway, I’m still not thrilled with my thinking here, so perhaps the paper by Pierce and Peters will help. Of course, I don’t really care about getting “exact” pvalues or anything like that, but I do want a general method of comparing data to replications from the assumed model.

2 thoughts on “Hypothesis testing and conditional inference

  1. I've been thinking about this, and the phrases "ignorable" and "missing at random" keep floating through my head. If it's legitimate to ignore the data collection process when making inferences about the parameters of the distributions of the complete data, then I'd have a hard time seeing why we should care about it when generating replicate data sets.

  2. Andrew, thanks for taking the time to spell out your thoughts on this at some length! I have three remarks.

    First, your two-dimensions-of-data view is a good one, I think. Were one able to pit the problem as a horse race between two hypotheses, one asserting skewness and the other not, then I'd imagine that the likelihood ratio statistic would do much what you propose, look away from the n-dimension and focus on the skewness-measure dimension.

    Second, if this talk of likelihood ratio statistics is too Neyman-Pearsony for you, think of the likelihood purists, Royall for instance, or maybe close your eyes and think about Bayes factors. Whatever your preferred paradigm, the precise specification of those two hypotheses is somewhat indeterminate, so that reasonable people would likely differ in the details of just what they should be, and by extension what the test statistic should be; if we're all going to agree on one test statistic, then everyone will have to compromise a bit. What I like about ancillarity and the conditionality principle is they offer a (reasonably) direct route to test statistics that are pretty good compromises, in this sense.

    Three, I still can't see how you can get by entirely without conditioning. Isn't there also some process by which it was decided whether to do the survey at all, and shouldn't you also be incorporating that decision point into your model, if you mean to avoid all conditioning away of complexities? Isn't it a possiblity a rogue interviewer could have absconded with half the data, so that there should be a dimension of your test statistic to represent that? Etc. Seems to me lots of stuff has to be ignored, and conditional inference only formalizes this.

Comments are closed.