If you need an accessible version of this item please contact JSTOR User Support
Information and Sufficient Subfields
S. G. Ghurye
The Annals of Mathematical Statistics
Vol. 39, No. 6 (Dec., 1968), pp. 20562066
Published by: Institute of Mathematical Statistics
Stable URL: http://www.jstor.org/stable/2239302
Page Count: 11
You are not currently logged in.
Access your personal account or get JSTOR access through your library or other institution:
If you need an accessible version of this item please contact JSTOR User Support
Abstract
This paper is the result of an attempt to clarify and improve some results in the theory of statistical information. The term information is used to denote different things in different contexts. First of all, there is Shannon's information, ∑ pi log pi, defined for probability distributions on a finite sample space; this measures, in an esthetically satisfactory way, the entropy or amount of uncertainty in a distribution. Then there is Wiener's information, ∫ f(x) log f(x) dx, defined for an absolutely continuous distribution on the line (or in nspace); it was introduced by Wiener, with an acknowledgment to von Neumann, as a "reasonable measure" of the amount of information, having the property of being "the negative of the quantity usually defined as entropy in similar situations" ([10], p. 76). Finally, there is "information of one probability distribution P with respect to another Q," commonly known as KullbackLeibler information. On a finite sample space, this has the form ∑ pi log (pi/qi) =  ∑ pi log qi  ( ∑ pi log pi), and thus has some relationship to entropy; note that the second term, which is the entropy of {pi}, is the minimum of the first expression over all distributions {qi}. An interesting idea due to Gelfand, Kolmogorov and Yaglom [3] establishes a connection between the KullbackLeibler information for a finite probability space and that for any space: If P, Q are probability measures on a measurable space (Ω, F), P ≪ Q, and {Ai, i = 1, ⋯, n} is any finite measurable partition of Ω, then the supremum of ∑i log [ P(Ai)/Q(Ai)] P(Ai) over all finite measurable partitions is ∫Ω log (dP/dQ)dP. The only published proof of this result seems to be that due to Kallianpur [5], which uses martingale theory. In Section 1, we shall obtain a rather simple direct proof of this result (Theorem 1.1) and extend it to the case where Q is any σfinite measure (Theorem 1.2). Wiener's information is then seen to be the supremum of ∑ log [ P(Ai)/Q(Ai)] P(Ai) over countable partitions, with Q = Lebesgue measure. Section 2 will be concerned with KullbackLeibler information. We shall define conditional information relative to a subfield, establish a relation between this conditional information and sufficiency of the subfield (Theorem 2.2), and also show that this conditional information equals the difference between information contained in the field and that in the subfield (Theorem 2.3). These are extensions of results obtained by Kullback and Leibler in a somewhat limited context.
Page Thumbnails

2056

2057

2058

2059

2060

2061

2062

2063

2064

2065

2066
The Annals of Mathematical Statistics © 1968 Institute of Mathematical Statistics