SPECIES TREE ESTIMATION UNDER JOINT MODELING OF COALESCENCE AND DUPLICATION: SAMPLE COMPLEXITY OF QUARTET METHODS

We consider species tree estimation under a standard stochastic model of gene tree evolution that incorporates incomplete lineage sorting (as mod-eled by a coalescent process) and gene duplication and loss (as modeled by a branching process). Through a probabilistic analysis of the model, we derive sample complexity bounds for widely used quartet-based inference methods that highlight the effect of the duplication and loss rates in both subcritical and supercritical regimes.

Automated summary

This study investigates species tree estimation using a stochastic model that incorporates incomplete lineage sorting and gene duplication and loss. Through a probabilistic analysis, sample complexity bounds for quartet-based inference methods are derived, highlighting the impact of duplication and loss rates.