4.6 Article

Split Samples and Design Sensitivity in Observational Studies

Journal

JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
Volume 104, Issue 487, Pages 1090-1101

Publisher

AMER STATISTICAL ASSOC
DOI: 10.1198/jasa.2009.tm08338

Keywords

Coherence; Multiple comparisons; Permutation test; Sensitivity analysis

Funding

  1. BSF [2008049]
  2. Measurement. Methodology and Statistics Program of the U.S. National Science Foundation [SES-0849370]

Ask authors/readers for more resources

An observational or nonrandomized study of treatment effects may be biased by failure to control for some relevant covariate that was not measured. The design of an observational study is known to strongly affect its sensitivity to biases from covariates that were not observed. For instance. the choice of an outcome to study, or the decision to combine several outcomes in a test for coherence, can materially affect the sensitivity to unobserved biases. Decisions, that shape the design are, therefore, critically important, but they are also difficult decisions to make in the absence of data. We consider the possibility of randomly splitting the data from an observational study into a smaller planning sample and a larger analysis sample, where the planning sample is used to guide decisions about design. After reviewing the concept of design sensitivity. we evaluate sample splitting in theory, by numerical computation, and by simulation, comparing it to several methods that use all of the data. Sample splitting is remarkably effective, much more so in observational studies than in randomized experiments: splitting 1,000 matched pairs into 100 planning pairs and 900 analysis pairs often materially improves the design sensitivity. An example from genetic toxicology is used to illustrate the method.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.6
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available