Lets say you are repeatedly going to recieve unselected sets of well done RCTs on various say medical treatments.

One reasonable assumption with all of these treatments is that they are monotonic – either helpful or harmful for all. The treatment effect will (as always) vary for subgroups in the population – these will not be explicitly identified in the studies – but each study very likely will enroll different percentages of the variuos patient subgroups. Being all randomized studies these subgroups will be balanced in the treatment versus control arms – but each study will (as always) be estimating a different – but exchangeable – treatment effect (Exhangeable due to the ignorance about the subgroup memberships of the enrolled patients.)

That reasonable assumption – monotonicity – will be to some extent (as always) wrong, but given that it is a risk believed well worth taking – if the average effect in any population is positive (versus negative) the average effect in any other population will be positive (versus negative).

If we define a counter-factual population based on a mixture of the study’s unknown mixtures of subgroups – by inverse variance weighting of the study’s effect estimates by their standard errors – we would get an estimate of the average effect for that counter-factual population that is minimum variance (and the assumptions rule out much – if any bias in this).

Should we encourage (or discourage) such Mr P based estimates – just because they are for counter-factual rather than real populations.

K?