There Is Individualized Treatment. Why Not Individualized Inference?

Doctors use statistics to advance medical knowledge; we use a medical analogy to build statistical inference from scratch and to highlight an improvement. A doctor, perhaps implicitly, predicts a treatment's effectiveness for an individual patient based on its performance in a clinical trial; the trial patients serve as controls for that particular patient. The same logic underpins statistical inference: To identify the best statistical procedure to use for a problem, we simulate a set of control problems and evaluate candidate procedures on the controls. Recent interest in personalized/individualized medicine stems from the recognition that some clinical trial patients are better controls for a particular patient than others. Therefore, an individual patient's treatment decisions should depend only on a subset of relevant patients. Individualized statistical inference implements this idea for control problems (rather than for patients). Its potential for improving data analysis matches that of personalized medicine for improving health care. The central issue-for both individualized medicine and individualized inference-is how to make the right relevance-robustness trade-off: If we exercise too much judgment in determining which controls are relevant, our inferences will not be robust. How much is too much? We argue that the unknown answer is the Holy Grail of statistical inference.