Access

You are not currently logged in.

Access your personal account or get JSTOR access through your library or other institution:

login

Log in to your personal account or through your institution.

On Measuring and Correcting the Effects of Data Mining and Model Selection

Jianming Ye
Journal of the American Statistical Association
Vol. 93, No. 441 (Mar., 1998), pp. 120-131
DOI: 10.2307/2669609
Stable URL: http://www.jstor.org/stable/2669609
Page Count: 12
  • Download ($14.00)
  • Cite this Item
Preview not available

Abstract

In the theory of linear models, the concept of degrees of freedom plays an important role. This concept is often used for measurement of model complexity, for obtaining an unbiased estimate of the error variance, and for comparison of different models. I have developed a concept of generalized degrees of freedom (GDF) that is applicable to complex modeling procedures. The definition is based on the sum of the sensitivity of each fitted value to perturbation in the corresponding observed value. The concept is nonasymptotic in nature and does not require analytic knowledge of the modeling procedures. The concept of GDF offers a unified framework under which complex and highly irregular modeling procedures can be analyzed in the same way as classical linear models. By using this framework, many difficult problems can be solved easily. For example, one can now measure the number of observations used in a variable selection process. Different modeling procedures, such as a tree-based regression and a projection pursuit regression, can be compared on the basis of their residual sums of squares and the GDF that they cost. I apply the proposed framework to measure the effect of variable selection in linear models, leading to corrections of selection bias in various goodness-of-fit statistics. The theory also has interesting implications for the effect of general model searching by a human modeler.

Page Thumbnails

  • Thumbnail: Page 
120
    120
  • Thumbnail: Page 
121
    121
  • Thumbnail: Page 
122
    122
  • Thumbnail: Page 
123
    123
  • Thumbnail: Page 
124
    124
  • Thumbnail: Page 
125
    125
  • Thumbnail: Page 
126
    126
  • Thumbnail: Page 
127
    127
  • Thumbnail: Page 
128
    128
  • Thumbnail: Page 
129
    129
  • Thumbnail: Page 
130
    130
  • Thumbnail: Page 
131
    131