3. We discussed in class the best currently available method for estimating the proportion of military servicemembers who are gay. What is that method? (Recall the problems with the direct approach: there is no simple way to survey servicemembers at random, nor is it likely that they would answer such a question honestly.)
Solution to question 2
2. Which of the following are useful goals in a pilot study? (Indicate all that apply.)
(a) You can search for statistical significance, then from that decide what to look for in a confirmatory analysis of your full dataset.
(b) You can see if you find statistical significance in a pre-chosen comparison of interest.
(c) You can examine the direction (positive or negative, even if not statistically significant) of comparisons of interest.
(d) With a small sample size, you cannot hope to learn anything conclusive, but you can get a crude estimate of effect size and standard deviation which will be useful in a power analysis to help you decide how large your full study needs to be.
(e) You can talk with survey respondents and get a sense of how they perceived your questions.
(f) You get a chance to learn about practical difficulties with sampling, nonresponse, and question wording.
(g) You can check if your sample is approximately representative of your population.
Solution: e and f. The purpose of a pilot study is to test out the data collection. The sample size will be too small for a, b, c, d, and g. In some of their earliest work, Kahneman and Tversky documented the common misconception of researchers that data from a small pilot study should closely match the population.
The question would have clearer if I’d inserted the word “small” before “pilot” in the preamble.