Access

You are not currently logged in.

Access your personal account or get JSTOR access through your library or other institution:

login

Log in to your personal account or through your institution.

Finding the Number of Clusters in a Dataset: An Information-Theoretic Approach

Catherine A. Sugar and Gareth M. James
Journal of the American Statistical Association
Vol. 98, No. 463 (Sep., 2003), pp. 750-763
Stable URL: http://www.jstor.org/stable/30045303
Page Count: 14
  • Download ($14.00)
  • Cite this Item
Finding the Number of Clusters in a Dataset: An Information-Theoretic Approach
Preview not available

Abstract

One of the most difficult problems in cluster analysis is identifying the number of groups in a dataset. Most previously suggested approaches to this problem are either somewhat ad hoc or require parametric assumptions and complicated calculations. In this article we develop a simple, yet powerful nonparametric method for choosing the number of clusters based on distortion, a quantity that measures the average distance, per dimension, between each observation and its closest cluster center. Our technique is computationally efficient and straightforward to implement. We demonstrate empirically its effectiveness, not only for choosing the number of clusters, but also for identifying underlying structure, on a wide range of simulated and real world datasets. In addition, we give a rigorous theoretical justification for the method based on information-theoretic ideas. Specifically, results from the subfield of electrical engineering known as rate distortion theory allow us to describe the behavior of the distortion in both the presence and absence of clustering. Finally, we note that these ideas potentially can be extended to a wide range of other statistical model selection problems.

Page Thumbnails

  • Thumbnail: Page 
750
    750
  • Thumbnail: Page 
751
    751
  • Thumbnail: Page 
752
    752
  • Thumbnail: Page 
753
    753
  • Thumbnail: Page 
754
    754
  • Thumbnail: Page 
755
    755
  • Thumbnail: Page 
756
    756
  • Thumbnail: Page 
757
    757
  • Thumbnail: Page 
758
    758
  • Thumbnail: Page 
759
    759
  • Thumbnail: Page 
760
    760
  • Thumbnail: Page 
761
    761
  • Thumbnail: Page 
762
    762
  • Thumbnail: Page 
763
    763