Your PDF has successfully downloaded.

You may be interested in finding more content on these topics:


You are not currently logged in.

Access JSTOR through your library or other institution:


Log in through your institution.

Journal Article

A Statistical Approach for Distinguishing Hybridization and Incomplete Lineage Sorting

Simon Joly, Patricia A. McLenachan and Peter J. Lockhart
The American Naturalist
Vol. 174, No. 2 (August 2009), pp. E54-E70
DOI: 10.1086/600082
Stable URL:
Page Count: 17
Were these topics helpful?
See somethings inaccurate? Let us know!

Select the topics that are inaccurate.

  • Download PDF
  • Add to My Lists
  • Cite this Item
We're having trouble loading this content. Download PDF instead.


Abstract: The extent and evolutionary significance of hybridization is difficult to evaluate because of the difficulty in distinguishing hybridization from incomplete lineage sorting. Here we present a novel parametric approach for statistically distinguishing hybridization from incomplete lineage sorting based on minimum genetic distances of a nonrecombining locus. It is based on the idea that the expected minimum genetic distance between sequences from two species is smaller for some hybridization events than for incomplete lineage sorting scenarios. When applied to empirical data sets, distributions can be generated for the minimum interspecies distances expected under incomplete lineage sorting using coalescent simulations. If the observed distance between sequences from two species is smaller than its predicted distribution, incomplete lineage sorting can be rejected and hybridization inferred. We demonstrate the power of the method using simulations and illustrate its application on New Zealand alpine buttercups (Ranunculus). The method is robust and complements existing approaches. Thus it should allow biologists to assess with greater accuracy the importance of hybridization in evolution.

Notes and References

This item contains 62 references.

Literature Cited
  • ['Álvarez, I., and J. F. Wendel. 2003. Ribosomal ITS sequence and plant phylogenetic inference. Molecular Phylogenetics and Evolution 29:417–434.']
  • ['Anderson, E. 1949. Introgressive hybridization. Wiley, New York.']
  • ['Arnold, M. L. 1997. Natural hybridization and evolution. Oxford University Press, New York.']
  • ['———. 2004. Transfer and origin of adaptations through natural hybridization: were Anderson and Stebbins right? Plant Cell 16:562–570.']
  • ['Barton, N. H. 2001. The role of hybridization in evolution. Molecular Ecology 10:551–568.']
  • ['Batt, G. E., J. Braun, B. P. Kohn, and I. McDougall. 2000. Thermochronological analysis of the dynamics of the Southern Alps, New Zealand. Geological Society of America Bulletin 112:250–266.']
  • ['Beerli, P. 2006. Comparison of Bayesian and maximum likelihood inference of population genetic parameters. Bioinformatics 22:341–345.']
  • ['Benjamini, Y., and Y. Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B 57:289–300.']
  • ['Bordewich, M., S. Linz, K. St. John, and C. Semple. 2007. A reduction algorithm for computing the hybridization number of two trees. Evolutionary Bioinformatics Online 3:86–98.']
  • ['Bruen, T. C. 2005. PhiPack: PHI test and other tests of recombination. McGill University, Montreal, Quebec.']
  • ['Bruen, T. C., H. Philippe, and D. Bryant. 2006. A simple robust statistical test for detecting the presence of recombination. Genetics 172:2665–2681.']
  • ['Buckley, T. R., M. Cordeiro, D. C. Marshall, and C. Simon. 2006. Differentiating between hypotheses of lineage sorting and introgression in New Zealand alpine cicadas (Maoricicada Dugdale). Systematic Biology 55:411–425.']
  • ['Burgess, R., and Z. Yang. 2008. Estimation of hominoid ancestral population sizes under Bayesian coalescent models incorporating mutation rate variation and sequencing errors. Molecular Biology and Evolution 25:1979–1994.']
  • ['Degnan, J. H., and N. A. Rosenberg. 2006. Discordance of species trees with their most likely gene trees. PLoS Genetics 2:e68.']
  • ['Edwards, S. V., and P. Beerli. 2000. Gene divergence, population divergence, and the variance in coalescence time in phylogeographic studies. Evolution 54:1839–1854.']
  • ['Ellstrand, N. C., and K. A. Schierenbeck. 2000. Hybridization as a stimulus for the evolution of invasiveness in plants? Proceedings of the National Academy of Sciences of the USA 97:7043–7050.']
  • ['Ferguson, D., and T. Sang. 2001. Speciation through homoploid hybridization between allotetraploids in peonies (Paeonia). Proceedings of the National Academy of Sciences of the USA 98:3915–3919.']
  • ['Fisher, F. J. F. 1965. The alpine Ranunculi of New Zealand. DSIRO, Wellington, New Zealand.']
  • ['Funk, D. J., and K. E. Omland. 2003. Species‐level paraphyly and polyphyly: frequency, causes, and consequences, with insights from animal mitochondrial DNA. Annual Review of Ecology, Evolution, and Systematics 34:397–423.']
  • ['Grant, B. R., and P. R. Grant. 1996. High survival of Darwin’s finch hybrids: effects of beak morphology and diets. Ecology 77:500–509.']
  • ['Grant, V. 1981. Plant speciation. 2nd ed. Columbia University Press, New York.']
  • ['Hey, J., and R. Nielsen. 2004. Multilocus methods for estimating population sizes, migration rates and divergence time, with applications to the divergence of Drosophila pseudoobscura and D. persimilis. Genetics 167:747–760.']
  • ['Holder, M. T., J. A. Anderson, and A. K. Holloway. 2001. Difficulties in detecting hybridization. Systematic Biology 50:978–982.']
  • ['Holland, B. R., S. Benthin, P. J. Lockhart, V. Moulton, and K. T. Huber. 2008. Using supernetworks to distinguish hybridization from incomplete lineage sorting. BMC Evolutionary Biology 8:202.']
  • ['Hudson, R. R. 1983. Properties of a neutral allele model with intragenic recombination. Theoretical Population Biology 23:182–201.']
  • ['Huson, D. H., T. Klöpper, P. J. Lockhart, and M. A. Steel. 2005. Reconstruction of reticulate networks from gene trees. Proceedings of the Ninth International Conference on Research in Computational Molecular Biology, pp. 233–249. Springer, Heidelberg.']
  • ['Jin, G., L. Nakhleh, S. Snir, and T. Tuller. 2006. Inferring phylogenetic networks by the maximum parsimony criterion: a case study. Molecular Biology and Evolution 24:324–337.']
  • ['Joly, S., and A. Bruneau. 2006. Incorporating allelic variation for reconstructing the evolutionary history of organisms from multiple genes: an example from Rosa in North America. Systematic Biology 55:623–636.']
  • ['Joly, S., J. R. Starr, W. H. Lewis, and A. Bruneau. 2006. Polyploid and hybrid evolution in roses east of the Rocky Mountains. American Journal of Botany 93:412–425.']
  • ['Jukes, T. H., and C. R. Cantor. 1969. Evolution of protein molecules. Pages 21–123 in H. N. Munro, ed. Mammalian protein metabolism. Academic Press, New York.']
  • ['Kingman, J. F. C. 1982a. The coalescent. Stochastic Processes and Their Applications 13:235–248.']
  • ['———. 1982b. On the genealogy of large populations. Journal of Applied Probability 19:27–43.']
  • ['Lee, C., and J. Wen. 2004. Phylogeny of Panax using chloroplast trnC‐trnD intergenic region and the utility of trnC‐trnD in interspecific studies of plants. Molecular Phylogenetics and Evolution 31:894–903.']
  • ['Legendre, P., and V. Makarenkov. 2002. Reconstruction of biogeographic and evolutionary networks using reticulograms. Systematic Biology 51:199–216.']
  • ['Linder, C. R., and L. H. Rieseberg. 2004. Reconstructing patterns of reticulate evolution in plants. American Journal of Botany 91:1700–1708.']
  • ['Lockhart, P. J., P. A. McLenachan, D. Havell, D. Gleny, D. Huson, and U. Jensen. 2001. Phylogeny, radiation, and transoceanic dispersal of New Zealand alpine buttercups: molecular evidence under split decomposition. Annals of the Missouri Botanical Garden 88:458–477.']
  • ['Machado, C. A., R. M. Kliman, J. A. Markert, and J. Hey. 2002. Inferring the history of speciation from multilocus DNA sequence data: the case of Drosophila pseudoobscura and close relatives. Molecular Biology and Evolution 19:472–488.']
  • ['Maddison, W. P. 1997. Gene trees in species trees. Systematic Biology 46:523–536.']
  • ['Maddison, W. P., and L. L. Knowles. 2006. Inferring phylogeny despite incomplete lineage sorting. Systematic Biology 55:21–30.']
  • ['Maddison, W. P., and D. R. Maddison. 2008. Mesquite: a modular system for evolutionary analysis. Version 2.5.']
  • ['Noor, M. A. F., K. L. Grams, L. A. Bertucci, Y. Almendarez, J. Reiland, and K. R. Smith. 2001. The genetics of reproductive isolation and the potential for gene exchange between Drosophila pseudoobscura and D. persimilis via backcross hybrid males. Evolution 55:512–521.']
  • ['Pamilo, P., and M. Nei. 1988. Relationships between gene trees and species trees. Molecular Biology and Evolution 5:568–583.']
  • ['Paun, O., C. Lehnebach, J. T. Johansson, P. J. Lockhart, and E. Hörandl. 2005. Phylogenetic relationships and biogeography of Ranunculus and allied genera (Ranunculaceae) in the Mediterranean region and in the European Alpine system. Taxon 54:911–930.']
  • ['Posada, D., and K. A. Crandall. 1998. Modeltest: testing the model of DNA substitution. Bioinformatics 14:817–818.']
  • ['Rambaut, A., and A. J. Drummond. 2005. Tracer. Version 1.3.']
  • ['Rambaut, A., and N. C. Grassly. 1997. Seq‐Gen: an application for the Monte Carlo simulation of DNA sequence evolution along phylogenetic trees. Computer Applications in the Biosciences 13:235–238.']
  • ['Rannala, B., and Z. Yang. 2003. Bayes estimation of species divergence times and ancestral population sizes using DNA sequences from multiple loci. Genetics 164:1645–1656.']
  • ['Rieseberg, L. H. 1997. Hybrid origins of plant species. Annual Review of Ecology and Systematics 28:359–389.']
  • ['Rieseberg, L. H., O. Raymond, D. M. Rosenthal, Z. Lai, K. Livingstone, T. Nakazato, J. L. Durphy, A. E. Schwarzbach, L. A. Donovan, and C. Lexer. 2003. Major ecological transitions in wild sunflowers facilitated by hybridization. Science 301:1211–1216.']
  • ['Rosenberg, N. A., and M. Nordborg. 2002. Genealogical trees, coalescent theory and the analysis of genetic polymorphisms. Nature Reviews 3:380–390.']
  • ['Sang, T., and Y. Zhong. 2000. Testing hybridization hypotheses based on incongruent gene trees. Systematic Biology 49:422–434.']
  • ['Sang, T., D. J. Crawford, and T. F. Stuessy. 1997. Chloroplast DNA phylogeny, reticulate evolution, and biogeography of Paeonia (Paeoniaceae). American Journal of Botany 84:1120–1136.']
  • ['Seehausen, O. 2004. Hybridization and adaptive radiation. Trends in Ecology & Evolution 19:198–207.']
  • ['Small, R. L., J. A. Ryburn, R. C. Cronn, T. Seelanan, and J. F. Wendel. 1998. The tortoise and the hare: choosing between noncoding plastome and nuclear Adh sequences for phylogeny reconstruction in a recently diverged plant group. American Journal of Botany 85:1301–1315.']
  • ['Small, R. L., R. Cronn, and J. F. Wendel. 2004. Use of nuclear genes for phylogeny reconstruction in plants. Australian Systematic Botany 17:145–170.']
  • ['Stebbins, G. L. 1959. The role of hybridization in evolution. Proceedings of the American Philosophical Society 103:231–251.']
  • ['Swofford, D. L. 2002. PAUP*: phylogenetic analysis using parsimony (*and other methods). Version 4.0b10. Sinauer, Sunderland, MA.']
  • ['Taberlet, P., L. Gielly, G. Pautou, and J. Bouvet. 1991. Universal primers for amplification of three non‐coding regions of chloroplast DNA. Plant Molecular Biology 17:1105–1109.']
  • ['Tate, J. A., and B. B. Simpson. 2003. Paraphyly of Tarasa (Malvaceae) and diverse origins of the polyploid species. Systematic Botany 28:723–737.']
  • ['Wang, R. L., J. Wakeley, and J. Hey. 1997. Gene flow and natural selection in the origin of Drosophila pseudoobscura and close relatives. Genetics 147:1091–1106.']
  • ['Wolfe, K. H., W.‐H. Li, and P. M. Sharp. 1987. Rates of nucleotide substitution vary greatly among plant mitochondrial, chloroplast, and nuclear DNAs. Proceedings of the National Academy of Sciences of the USA 84:9054–9058.']
  • ['Zomlefer, W. B. 1994. Guide to flowering plant families. University of North Carolina Press, Chapel Hill.']