JSTOR

I.  Introduction

The Standard and Poor's (S&P) Composite Index remains the index of choice for many investors, particularly institutional investors. As such, it serves as a major benchmark of stock market performance. It also has a long history, providing investors with a lengthy time series of returns from which to gauge, among other parameters, “average” returns on an index of commonly held stocks. Furthermore, the S&P monthly index series as originally constructed was extended back to January 1871 by Cowles, making possible a consistent time series of returns covering approximately 130 years.1

Because of its prominence as a market measure, it is important that users fully understand the S&P index as it is conventionally reported, and the S&P index that could be reported. Confusion continues today over its composition relative to the history of the S&P 500 (weekly) index and the S&P 90 (daily) index and its relation to the work of Alfred Cowles that has been used to extend this index back in time. Although not used, a broad S&P weekly price index does exist for the period 1918 through 1956, and these data are among the highest‐quality measures of stock movements over this period. These weekly data were abandoned by S&P in 1957, and considerable confusion of stock price history has resulted. Failure to use a consistently defined time series of S&P 500 returns back to 1871 has resulted in misstatements of how well the “market” has actually performed over long periods and the inability to compare results across studies.

Consider the historical description by Siegel, which has been quoted by other researchers: “In 1939, Alfred Cowles … constructed a stock price index back to 1871. … The Cowles index became the basis of what is currently the most important benchmark among portfolio analysts, the S&P 500 index … . The Standard & Poor’s stock price index was inaugurated on March 4, 1957” (Siegel 1998, pp. 60–61). In fact, the Standard Statistics Company began estimating stock price indexes on a weekly basis in 1923 for an “all stocks” index, with subsector indexes, which were carried back to 1918. This series is the basis of what was later to be called the S&P 500 Index. The Cowles index made an important contribution by extending existing S&P data backward in time from 1918, but it most certainly was not the basis of the S&P Composite Index.

This article clarifies the historical record of stock prices and returns using the Standard & Poor’s and Cowles monthly indexes from January 1871 to the present, providing precise details of the coverage of the index as well as making some improvements. It will probably surprise many users of the S&P Composite Index to learn that, to date, studies reporting S&P composite returns prior to 1957 (e.g., the Ibbotson Associates data) have not used more than 90 stocks for the time period 1926–56. In fact, it is impossible because prior to March 1957 Standard & Poor’s never collected data for 500 companies on a consistent basis, although data for more than 400 S&P stocks were collected and are available.

Studies using Cowles’s and the early S&P historical monthly data have failed to recognize the biases inherent with time‐averaged data. Percentage changes in such data have a downward‐biased variance and an upward‐biased measure of autocorrelation relative to month‐end data. Also, many studies have combined returns from various indexes when reporting stock returns over long periods, thereby distorting the record with regard to the S&P Composite—in other words, the results reported are not S&P Composite data plus Cowles extensions but rather a mixture of indexes. In contrast, we report a consistent and broad index of “S&P” stock returns over the entire time period for which these data are available, with improved data for the Cowles extensions. The S&P weekly data for the broadly defined Composite index from January 1918 through January 1957 have been improved by our estimation of month‐end prices. We refer to this improved price index as the Standard & Poor’s Weekly and Cowles (SPWC).

II.  Standard & Poor’s Indexes

As noted, the Standard Statistics Company began estimating stock price indexes on a weekly basis in 1923 for an “all stocks” index. This weekly index was carried back to January 1918. The indexes were capitalization weighted and based on Fisher’s “ideal index” formula (1927). These stock indexes underwent a major change in 1926 by placing the index on a base, which was later changed to , and still later was changed to , which continues as the base today. Standard Statistics Company merged with Poor’s in 1940, but the indexes were maintained.

It is important to understand clearly that there were actually two different S&P “Composite” Indexes.2 First, there is the weekly index (the “all stocks” index) as described above. Second, a daily index of 90 stocks, consisting of 50 industrials, 20 rails, and 20 utilities, was initiated in 1928 (and carried back weekly to December 1925). Going forward, this index of 90 stocks continued until March 1957, when the daily index was shifted to the 500 stocks. In March 1957, S&P abandoned its historical weekly series with the statement, “To avoid confusion, Standard & Poor’s has standardized on its former daily price index … for the back record.”3 This caution by S&P was repeated in the biennial editions of their historical data (the “Blue Books”) for many years, but is not mentioned in recent editions. Simultaneously, S&P ceased estimating the S&P 90 Index. The decision to abandon the historical record of the broad‐based weekly index in favor of the S&P 90, which ceased, has been the cause of much confusion.

Figure 1 shows the time periods covered by the various indexes that deal with S&P data, and is useful in sorting out the alternatives. As this figure shows, and using the labels in that figure, the S&P 90 (the daily index) covers the time span 1925–56, while the S&P Weekly Index covers the time period 1918–56. The S&P 500 Index with 500 stocks in it begins in 1957, while the NYSE CRSP begins in 1926. Going back historically, the Cowles index covers the period 1871–1940 (including his updates). The Standard & Poor’s Weekly Composite with Cowles extensions, with our modification with month‐end price estimates (labeled as SPWC), covers the period 1871–2000. Our reconstructed data based on Cowles’s monthly estimates extend from January 1871 and are linked to the S&P weekly data in January 1918, and further spliced to the S&P 500 daily price index in February 1957, thereby providing coverage for the entire time period.

(27 KB)

Fig. 1.— Time coverage of the major value‐weighted indexes, 1870–1999. A: SPWC (modified S&P Weekly/Cowles) Index, January 1871–December 1999. B: Cowles Monthly Averaged Index, January 1871–December 1940; 1 (1938), January 1871–December 1937, 2 (1939), January 1871–December 1938, 3 (1941), January 1939–December 1939, and 4 (1942), January 1940–December 1940, and conversion of the complete monthly series to the base period. C: Standard & Poor’s Weekly Index, January 1918–February 1957; (1941) January 1918–December 1940, converted to the 1935–39 base, which continued through February 1957. D: Standard & Poor’s Daily Index, December 1925–February 1957 (Standard & Poor's Statistical Service 1957). E: Standard & Poor’s 500, March 1957–December 1999 (Standard & Poor's Statistical Service 1999). F: NYSE Index, Chicago Research in Security Prices, December 1925–December 1999 (Center for Research in Security Prices 2000).

Open New Window

Thus, what is commonly referred to as the S&P 500 Composite Index actually consists of 90 stocks for the period 1926–56 and not 500 stocks as most people assume. Although the broader weekly coverage was continued by S&P through March 1957, it basically was lost to researchers because S&P itself did not maintain the data nor preserve it. This study restores these weekly data, making possible for the first time research using the broader (weekly) S&P Composite Index.

We have recovered most of the biennial yearbooks published by the S&P Statistical Service and have compiled a record of the number of stocks, by major sector, that were included from 1918 to the present in the broader weekly index. Table 1 provides an estimate of the number of stocks included in the S&P Composite Index based on the broader weekly coverage from 1918 to the present.4 For reference, the last column of the table shows the number of NYSE stocks that are included in the CRSP NYSE index (2000).5 What is important to note is that the S&P Composite Index, using the weekly data up to 1957, included a large number of companies, in excess of 400 by 1928. Thus, the weekly index data abandoned by S&P were potentially very valuable as a measure of the broad market and were comparable to what is used today by S&P, the 500 Composite Index, to measure the market.

Table 1 Estimated Number of Stocks Covered by Major Industry Group by S&P and CRSP/NYSE, 1918–99

Open New Window

When the Standard Statistics Company began estimating the “all stock” index in 1923, 233 stocks were included. Of this total, 184 were industrials, 31 were railroads, and 18 were utilities.6 When these indexes were carried back weekly to January 1918, the number of stocks available decreased to 198 firms at the beginning of the series (Cowles 1939, p. 448). In 1927, the company made major changes in their sector classifications and established the base of the index as . The number of firms covered decreased slightly to 228, with 176 industrial, 31 rails, and 21 utilities. The capitalization of these major sectors, as a percentage of total capitalization, was approximately 60% industrial, 30% railroads, and 10% utilities.7

As table 1 shows, from December 1927, the number of stocks included in the weekly S&P Composite Index increased sharply to over 400 within 2 years, remained between 404 and 421 stocks through 1951, and reached 480 stocks during the period 1952–56.8 Over the same time period, the number of stocks on the NYSE, as shown by the CRSP NYSE data in the last column, increased from 539 in December 1927 to 1,059 by the end of 1956.

In March 1957, S&P reconstituted their indexes, making the base period and fixing the composite at 500 stocks with a pattern of fixed allocation by sector for extended periods, as shown in table 1.9 Finance was added as a major sector in 1977 with 40 stocks initially. Fixed allocation within sector was discontinued in 1988, although the total number of stocks has remained fixed at 500. As table 1 also shows, the number of NYSE stocks covered by CRSP increased from 1,056 in 1957 to 2,895 by 1998.

III.  Cowles’s Extension of the S&P Stock Indexes

In the mid‐1930s, Alfred Cowles and Associates undertook the task of carrying monthly estimates of stock prices back as far as feasible at the time, ending in January 1871. Their objective was stated as follows: “The purpose of the Cowles Commission common‐stock indexes is to portray the average experience of those investing in this class of security in the United States from 1871 to 1938. The indexes of stock prices are intended to represent, ignoring the elements of brokerage charges and taxes, what would have happened to an investor’s funds if he had bought at the beginning of 1871 all stocks quoted on the New York Stock Exchange, allocating his purchases among the individual issues in proportion to their total monetary value, and each month up to 1938 had by the same criterion redistributed his holdings among all quoted stocks” (Cowles 1939, p. 2). This was an ambitious project. After reviewing all of the various measures of U.S. stock indexes and methods for calculation, Cowles chose to use Fisher’s “ideal index” with the outstanding shares as weights for the computation of the index values.

Stock price data could have been secured from daily or weekly quotes, but a more convenient (and less costly) source for data on monthly stock prices was the midpoint of the high and low prices available from the Commercial and Financial Chronicle.10 Cowles used the monthly arithmetic averages of the weekly indexes of the Standard Statistics Company from 1918 through 1938. He carried monthly data back from 1918 to January 1871, using, to the full extent possible, the same definitions for inclusion and the same technical construction.11

Table 2 provides the number of stocks included in Cowles's early indexes for selected years. Cowles was skeptical about the accuracy of “curb” prices, and relied only on NYSE stocks. Prior to 1871, few stocks had consistent price data; only 48 stocks in January 1871 had reliable quotes. The number of stocks included increased for all three sectors, with rails increasing more slowly than utilities, which increased more slowly than industrials. Cowles estimated that his indexes “include approximately 97 per cent of the market value of all stocks quoted.”12

Table 2 Number of Stocks Covered by the Cowles Commission by Major Sector and Composite, 1870–1915

Open New Window

In addition to carrying the monthly stock price indexes back to 1871, Cowles undertook the estimation of monthly dividends and annual earnings for the major sectors as well as for subsector indexes. The Standard Statistics Company had not estimated any “all stock” or sector dividends or earnings for their indexes from 1918, only beginning such estimates for the S&P 90 indexes in 1928 (and carried back to December 1925). Therefore, without the Cowles data it would not be possible to construct total returns for the S&P Composite Index for the period prior to December 1925.

These indexes were first published in 1938 (Cowles 1938), but were soon updated “to correct such errors as have been discovered and to bring the indexes down through 1938” (Cowles 1939, p. viii). In two later mimeographed updates, Cowles carried the data through 1939 and 1940, as well as incorporating the revised base from 1926 through 1935–39.13 The price index data of Cowles added nothing to the information available from the Standard Statistics Company from 1918, and by averaging its weekly indexes to obtain monthly prices may have subtracted from its value because of the bias in averaged data (examined below). However, the monthly dividend and annual earnings information added great value for the returns and valuation series for the period 1918 through 1940. As figure 1 shows, the Cowles data, including extensions, cover the time period 1871–1940. They overlap with S&P’s weekly data during the period 1918–40.

IV.  An Analysis of the Coverage of Stocks Using the S&P Composite Data

Given the computational technology of the 1920s and 1930s, calculation of a value‐weighted index was a tedious, costly, and time‐consuming task. In addition, obtaining timely data for a number of stocks was simply not possible. Indeed, S&P attributes the delay of including financial stocks in the composite index until 1976 to “the difficulty of including Over‐the‐Counter stocks in our computerized calculations.”14 Therefore, it is not surprising that the daily index, which is the index officially reported by S&P for 1926–56, contained only 90 stocks, whereas the S&P indexes including the expanded list of weekly stocks were calculated less frequently.

Regardless of difficulties, the broader weekly S&P composite data provide a better representation of the overall market during the early period. Cowles’s estimate of the capitalization value of the Standard Statistics Company using the broader weekly index was about 90% of the NYSE subsequent to 1917, 73% of the market value of common stocks listed on the 24 various exchanges at that time, and 77% of those stocks that were actively traded (see Cowles 1939, pp. 6–7). The Standard Statistics Company itself estimated in 1927 that the capitalization of the stocks in the weekly index was about 92% of the total of stocks traded on the NYSE. As a comparison, in the biennial editions of their yearbooks S&P estimates capitalization values, as a percentage of the NYSE total capitalization, at 76.5% for 1980. This percentage remains relatively steady through 1997, where the estimate is 81%.15

Although total capitalization has been a focus of the Standard & Poor’s Index Committee, there has always been an interest in industrial sector representation. Early in the life of the “all stocks” index, the selection process was described: “The stocks were selected largely from the common stocks of over 700 corporations listed regularly by groups in the Standard Earnings Bulletin. The value of the issue … was computed for each stock. Then a sufficient number of stocks were selected to obtain a fair representation, varying from about 70 to 100 per cent of total market value for each group of stocks” (Standard & Poor's Statistical Service 1930, p. 63).

For 1918, Cowles counted 32 groups, increasing to 66 in 1937 (Cowles 1939, p. 448). From S&P reports, the number of industrial sectors stood at 68 in 1941, increasing to 84 in 1957. The number of sectors has remained relatively constant since 1957, although many sectors have been added, dropped, or redefined.16 The goal has been to include the major firms in representative sectors of the market. Over time, the index has not been solely an index of large company stocks, but a collection of representative companies in “broad industry groups that approximates the distribution of … the NYSE stock population,” taken as the assumed model. A major use of the group indexes is as a standard for comparison of performance of individual stocks within each group (Standard & Poor's Statistical Service 1984, p. 3).

A major development in measuring stock prices and returns occurred with the introduction of the Chicago Research in Security Prices (CRSP), which was enabled by a grant from Merrill Lynch, Pierce, Fenner & Smith in 1960. The data covered all firms traded on the NYSE, included month‐end prices and monthly dividends, and were combined into a composite value‐weighted price and a total returns index. These data were chronicled by Fisher and Lorie (1964, 1968, and 1977). This effort provided the inspiration for Ibbotson and Sinquefield in 1974 to construct a similar stock and returns index using the data from Standard & Poor’s. As shown in figure 1, the CRSP NYSE data begin in 1926 and continue to the present.17

The historical data used by Ibbotson and Sinquefield (1976), and updated on a continuous basis by Ibbotson Associates (1999), are based on the S&P 90 daily data from December 1925 through 1956 rather than the more broadly defined S&P “500” weekly data. Therefore, the historical data reported by Ibbotson Associates are not, in actuality, the S&P 500 Composite Index in the sense of a broad market index of hundreds of stocks. As table 1 shows, from 1928 on there were always in excess of 400 stocks available for the broader S&P weekly index. However, Ibbotson and Sinquefield (1976) would have found it very difficult to locate those data. Therefore, they did the only reasonable construction of the S&P index possible at that time and cannot be faulted for presenting the data as they have. Furthermore, Ibbotson Associates clearly states in their publications that their S&P 500 Index is based on the S&P 90 during the period 1926–56, and Ibbotson clearly uses month‐end prices, thereby avoiding the use of averaged data.

Standard & Poor’s reports a historical series for its index, but it is not based on the best data available. First, the data for 1926 through March 1957 are based on the S&P 90 Index. Second, although S&P reports their monthly prices as both end‐of‐month and averages, they continue to report their historical series, through all versions, using averaged data. Finally, for the back record prior to 1926, S&P uses Cowles’s data, but from the first edition (1938) instead of the revised (corrected) data from his second edition (1939); in addition, the Cowles data are averaged data. Therefore, the data used by Shiller and by Siegel from 1871 through 1925, as described below, are based on inferior estimates. Similarly, the dividends and earnings for the S&P historical series are based on the same sources as their stock price data and therefore contain errors that were later corrected by Cowles.

Research by Shiller (1989) utilizes the January values of the S&P historical index and the calendar year dividends and earnings adjusted to the index. Shiller’s data are based on the S&P historical series, for January only, which are averaged data based on the S&P 90 from 1926 through 1956. Shiller’s earlier data are also averaged.

Siegel (1992, 1998), in the construction of his index of stock prices and returns, uses Shiller’s data from 1871 to 1925, then splices to the Chicago Center for Research in Security Prices (CRSP) from 1926 to the present. Therefore, while Siegel’s data and analysis may offer useful insights into the history of stock returns, these data cannot directly be compared to other studies that use only S&P data. Furthermore, for the early period 1871–1925, the data are based on the flawed S&P historical data.

It is apparent that the historical record of stock prices and returns based on S&P data, with or without the Cowles extension backward in time, consists of a mixture of measures that lacks consistency of definition. To date, the following observations emerge: (1) No study to date has presented a “true” composite of S&P prices and returns for the “modern” period from 1926 based on the S&P weekly data, which contain more than 400 stocks. The 30‐year period from 1926 to 1956 contains only 90 stocks, and the performance of the 90 stocks was quite different from the broader market representations (as shown below). (2) No study to date has presented the definitive record of stock returns since 1871 based on the Cowles data and the broad S&P Index, using corrected end‐of‐month data.

V.  Reconstruction of a Monthly Composite Price Index

Cowles’s monthly data are flawed for some research purposes because they are “averaged.” Cowles used monthly arithmetic means of the S&P weekly indexes from 1918 through 1940 and the midpoint of the monthly high and low prices of stocks in constructing monthly indexes back to January 1871. Working (1960) pointed out that changes in time‐averaged data have a downward‐biased variance and a built‐in‐first‐order autocorrelation. In fact, Cowles was instrumental in Working’s article. Cowles and Jones (1937) did an analysis of stock index and stock price movements in which they concluded that there was an element of predictability in stock price movements. Working’s analysis was in response to this article (and others). This led Cowles to revisit his analysis and to revise his conclusions (Cowles 1960).

Schwert (1990) showed that the cross‐correlations of averaged data with other series are downward biased as well. Although the problem of stochastic bias has focused on first‐order conditions, there now is evidence that autocorrelation problems with monthly changes can persist through at least the second order, and the downward biases in variances and covariances can persist through higher orders, at least up to the twelfth.18

We are not the first to attempt to correct for the averaged data inherent in the Cowles series. Schwert (1990) constructed a monthly index of stock prices and returns using Cowles’s price and dividend data from 1871 to 1885. He was well aware of the biases in averaged data, and in an attempt to deal with this issue Schwert shifted to the Dow Jones Industrial Averages through 1925, where he then shifted to the CRSP monthly estimates. This dealt with the averaged‐data problem because the Dow Jones data use month‐end prices. However, the appreciation change in that measure over the December 1885 through 1925 period was 296.75%, compared to an appreciation change in Cowles’s monthly averaged composite price index of 139.38%, a difference of 1.27% per annum compound. Thus, the trade‐off in accuracy of estimating stock returns relative to having desirable stochastic properties in the returns series seems excessive.19

These problems can better be addressed with the new data we have constructed. We have reconstructed the S&P weekly data that Cowles averaged over the period 1918–40 and estimated month‐end prices based on a “companion” series of stock prices from February 1885 through February 1957, splicing into the S&P end‐of‐month prices that continue to the present.20 This provides, for the first time, a consistent long‐term series for the S&P Composite Index based on month‐end prices (the SPWC series).

With the S&P weekly data, between January 1918 and February 1957 there are 470 monthly closing prices, with 84 (18%) of the S&P weekly price quotations being on the last trading day of the month.21 There were 384 months when the S&P Index was calculated near the last trading day and required interpolation from a companion series.22 Candidates for a companion daily series of stock prices for the S&P Weekly Index over the period 1918 through 1956 are few. After reviewing those few daily indexes available, we have chosen what we feel are the best alternatives for various time periods, as explained below.

For each of the 384 affected observations, we calculate a percentage adjustment of the S&P weekly price nearest the end of the month using the percentage change of the daily companion series from that date to month end, multiplied by the S&P weekly price. The S&P 90 prices were used from January 1928 through February 1957, the New York Times Industrial and Rail Averages from January 1921 through 1927, and the Dow Jones Average from January 1918 through 1920. This process maintains the basic average changes of the S&P weekly prices while eliminating the biases of the averaged data.23

For the period prior to the S&P weekly series in January 1918, we used as the companion series the Dow Jones Averages, which extend back to 1885. We calculated the monthly closing price as a percentage of the midpoint of the high and low prices, and applied that to Cowles’s monthly average. This produced an estimate with greater variability than predicted by Working, and geometric adjustments of the ratios were made to achieve reasonable month‐end estimates for Cowles’s data. These adjustments were applied back to the beginning of the Dow Jones Averages in February 1885. We have been unsuccessful in finding an appropriate companion series prior to 1885, and have used Cowles’s monthly averages back to 1871.24

Working estimated that the bias in averaged data would mean that the variance of end‐of‐period changes would be 50% greater than changes in the averaged data, and that the averaged changes would have first‐order autocorrelation of an amount approximating .25 greater than that of the end‐of‐period data. Schwert (1990) suggested that the covariance of changes in averaged data with other series would be downward biased by about two‐thirds relative to the covariance of period‐end changes. We determined that, if the method of using the best available companion stock price series with the Cowles averaged data for month‐end estimates achieved the 50% increase in the variance of first‐order changes, the autocorrelation and covariance problems would take care of themselves.

Table 3 presents the results of a comparison of variances, autocorrelation, and covariances between the month‐end estimates and the averaged prices. The time periods for these comparisons are based on dates of changes in companion series and/or S&P/Cowles estimates. The first row of data in table 3 relates to the approximate expected values for each of the three statistics. The variance differential is the variance of the month‐end percentage changes relative to the variance of averaged percentage changes, which Working suggested would be “about” 1.5. The autocorrelation differential is an arithmetic difference of the first‐order autocorrelation of the month‐end percentage changes minus the autocorrelation in the averaged changes, which Working suggested would be “about” 0.25. The covariance differential is the estimated covariance between the estimated month‐end percentage changes and the percentage changes in the averaged prices, which Schwert suggested would be around .60 or .70. For the different subperiods shown in table 3, the estimated values strongly suggest that the methodology used with the companion series has eliminated the biased stochastic percentage changes associated with averaged data.

Table 3 Statistical Results for Tests of the Variance Differential, the Autocorrelation Difference, and the Covariance of Changes in the Averaged and Period‐End Price Data, by Subperiods, Relative to the Expected Amount of the Bias

Open New Window

Our resultant S&P composite price index (SPWC) is that of Cowles's monthly averages from January 1870 through January 1885, a month‐end estimate of prices through 1917, and a weekly and month‐end price index from January 1918 to the present. This allows estimation, for the first time, of a continuous “S&P Composite Index” based on the broader weekly measure of the market and Cowles's data that have been corrected for the averaging problem.

VI.  Dividends and Earnings

Cowles provided monthly price index estimates, as well as cumulative monthly wealth indexes, for the composite and for the subindexes. We have used Cowles's estimates of the two indexes to estimate the dividend amounts adjusted to the index.25

For the estimates of monthly dividend amounts adjusted to the price index, we begin with the revised estimates from Cowles’s data by Wilson and Jones (1987).26 These monthly estimates are based on the arithmetic difference of percentage changes in Cowles’s total returns index and the price index changes, and cover the period from January 1871 through December 1940 (Cowles’s series P and series C, 1939, 1941, and 1942). In the early years, prior to 1900, this procedure sometimes led to estimates of dividend returns that were negative or unrealistically large, usually for adjacent months. These yields were further smoothed and our estimated annual dividend levels checked against annual dividend amounts that were estimated by Cowles.27 This provides 70 years of monthly dividend data that are consistently defined.

Standard & Poor's provides 12‐month moving dividend estimates for the S&P 90 for the period 1926 through 1956.28 Alternative monthly dividend yields can be calculated from the CRSP NYSE files for the period 1926 to the present. Moody’s has estimated monthly dividend amounts and yields for industrial, transportation, and utilities averages for the period 1929 through 1974, and annual estimates for a composite average, when their estimates ceased (Moody's Investment Service 1976).29 For available overlap periods prior to 1940, or between 1957 and the present, the strongest relation was between the Cowles–S&P Weekly and the CRSP NYSE, followed by estimates based on Moody’s, with the weakest relation with the S&P 90 estimates.

Standard & Poor's publishes quarterly 12‐month moving dividend estimates for the composite (as well as the major subindexes) from March 1957 to the present, which leaves a 16‐year gap in dividend estimates for the index from January 1941 through February 1957.30 We used the CRSP NYSE yields multiplied by the S&P weekly prices to estimate dividend levels from January 1941 through February 1957. From the S&P 500 quarterly dividend amounts from 1957 to the present, we have estimated monthly dividend amounts within each quarter based on the seasonal pattern of the CRSP NYSE yields.

Cowles, for the period 1870 through 1940, provides two different estimates for earnings: earnings yields for calendar years for stocks for which earnings estimates were available (Series R), and earnings amounts based on the prices of stocks for which earnings are available (Series E with price series P[Ea]). The earnings series are of lesser quality than the corresponding dividend estimates because of the complete absence of data on the earnings of some corporations prior to the end of World War I. With Cowles’s estimates an alternative series (downward biased due to the different price indexes P and P[Ea]) can be estimated by multiplying the earnings yields and the all‐stock price index. Between 1871 and 1917 we use the average earnings of the two estimates from Cowles’s data. Between 1917 and 1940, the two estimates differ little. The only other earnings series available are for the S&P 90 from 1926 through 1957, and those of Moody’s, which is annual for the composite, but quarterly for the subsector averages.

We use the annual earnings yields for the composite multiplied by the Cowles price index and use the Moody’s Industrial quarterly pattern to check the level of earnings for the overlap period 1930–40. Moody’s data were used to estimate quarterly and annual earnings for the period 1941 through March 1957. From March 1957, S&P estimates quarterly and four‐quarter moving total earnings. The S&P 90 earnings are available over the period 1926 through March 1957, but the composition of the S&P 90 places relatively too much weight on rails and utilities, and this earnings series is less correlated with the Cowles series during the overlap period than with Moody’s estimates.

VII.  Comparisons of Appreciation and Total Returns with Alternatives

Given the alternative estimates of aggregate U.S. stock returns available from Cowles, Ibbotson and Sinquefield, Shiller, Schwert, Siegel, and Wilson and Jones, we now consider the significance of the differences among these series. We review alternative results in terms of appreciation and total returns with reinvestment of dividends for the composite indexes.

Table 4 compares the total and appreciation returns of various indexes for selected time periods covering the life of the indexes. The Cowles (1939) index, with the updates (1941, 1942), extends through 1940, as do the Standard & Poor’s historical returns as used by Shiller and by Siegel. Beginning in December 1925, the S&P series used the S&P 90 prices and dividends, which continued through 1956, and are used by Ibbotson Associates.

Table 4 Total Percentage Appreciation and Cumulative Returns, and Annual Geometric Means and Geometric Standard Deviations, for Four Nonoverlapping Time Periods, 1871–1999, and for the Total Periods December 1925–December 1999 and January 1871–December 1999

Open New Window

Panel A in table 4 provides comparisons from January 1871 through December 1925 for the three series available: Cowles, the S&P historical returns, and our month‐end estimates of the S&P Weekly Composite and Cowles Index (SPWC). Included are the total percentage change and the per annum geometric means and standard deviations for both appreciation and cumulative returns. The returns for the S&P historical series are slightly less than either of the others. The differences are attributable to the S&P series being based on the values from the earlier edition of Cowles, rather than the updates that corrected some errors. Also, the dividends are based on reinvestment annually rather than monthly, as is the case for the other two series. The differences between Cowles’s returns and the weekly S&P Weekly Composite and Cowles (SPWC) returns are due primarily to our modification of monthly dividend estimates and are influenced only slightly by the use of our month‐end price estimates. Our month‐end price estimates produce a higher degree of accuracy of returns than the average prices used by Cowles, as well as the effects of monthly reinvestment and compounding.

Panel B of table 4 compares the returns from the end of 1925 through 1940. For this period, we can include the Ibbotson and the CRSP NYSE estimates that began in December 1925.31 During this volatile period of the Great Depression and the stock market crash, appreciation returns and cumulative returns differ dramatically. The S&P 90 stocks, as reported by Ibbotson and by the S&P historical series, fared better than the stocks in the broader S&P weekly index (SPWC) and in the CRSP NYSE index. The S&P 90, with heavier weighting in utilities and rails than in industrials, relative to the broader measures, had less of an appreciation loss and a greater total return. The CRSP NYSE index was more significantly affected than the Cowles index or the weekly S&P Composite Index because of its broader coverage, but the differences among these three indexes are less dramatic than the comparisons of all three with the S&P 90.

Panel C of table 4, covering the period December 1940 through 1956, is beyond the period covered by Cowles, which now drops out of the comparisons. The S&P historical data and Ibbotson’s measure, which are based on the S&P 90 for the period, are available for comparison with the broadly based S&P 500 index using weekly data (SPWC) and with the CRSP NYSE data. The narrower‐coverage indexes (S&P historical and Ibbotson Associates) outperformed the broader measures, suggesting once again that the standard measurement of returns during the period 1926–56 is influenced by the relatively narrow coverage of the S&P 90 stocks. The CRSP NYSE returns also suffered by comparison because of the broader coverage but outperformed the weekly S&P Composite Index.

Panel D of table 4 covers the period December 1956 through 1999 for S&P historical, Ibbotson, CRSP NYSE, and the S&P Weekly Composite and Cowles Index. For this period any differences between the returns of Ibbotson and the reported S&P Index are generally small. The same S&P 500 prices are being used by both for this later period, and capital appreciation returns will be identical, except for round‐off differences in measuring the indexes. The difference in the total returns, with dividends reinvested monthly, is a result of the source of the estimates of the dividend amounts, the allocation of dividend amounts by months within quarter, and the periodicity of dividend reinvestment.

Our S&P dividend estimates, as reflected in the S&P Index as reported here, are based on the S&P 500 quarterly and annual dividends since the first quarter of 1957, with some smoothing from the estimates from the earlier period. The S&P estimates of quarterly dividends lag the price data by from 2 to 4 months and are subject to revision periodically. We use the quarterly dividend estimates, allocated monthly within quarter according to the seasonality of dividends of the CRSP NYSE data. Ibbotson has used various estimates of dividends over this period, all based on S&P data, which may differ slightly from their “official” estimates.32 Our estimates for this period show a geometric mean for both appreciation and total returns only slightly higher than Ibbotson’s, and with geometric standard deviations that are very slightly lower. Over this period, the appreciation returns, as well as the total returns for the CRSP NYSE, are lower than for the other three indexes.

Panels E and F provide a long‐term perspective on appreciation and returns. As panel E shows, since 1925 Ibbotson reports a geometric mean for total returns on the S&P 500 of 11.3469%. However, this geometric mean exceeds the other three measures shown. The broader S&P Composite Index shows 10.9952%, while the CRSP NYSE Index is even lower at 10.7039%. Therefore, our analysis clearly suggests that Ibbotson’s data, based as they are on 90 stocks for a 30‐year period, overstate the rate of return on the S&P 500.

Panel F provides the longest‐term comparison, from 1871 through 1999. The S&P historical series produces the greater geometric means for both appreciation and total returns. We believe that these means overstate the returns, for reasons presented earlier.

For the complete period, January 1871 through 1999, our newly reconstructed S&P Weekly plus Cowles Index, with one dollar invested in January 1871, and with dividends reinvested monthly, reaches a value of $107,508.834 at the end of 1999, for a geometric mean return of 9.3965% per annum. The appreciation return for the 129 years totaled 288.0916%, or 4.4907% per annum.

VIII.  Cowles and S&P Composite Data with Basic Valuation Ratios

The alternative series of Ibbotson Associates, Schwert, and CRSP do not provide earnings estimates for their indexes. Siegel presents graphs of earnings, with dividends, from 1870 through 1996 (1998, p. 79, table 5‐1) but does not provide a source. Since the CRSP price and dividend data that he uses from 1926 do not include earnings estimates, we conclude that he must be using the estimates from the S&P historical series, which in turn are based on the S&P 90 estimates from 1926 through 1957 and the (uncorrected) Cowles estimates from his first edition. The data that we have reconstructed include earnings estimates, adjusted to the index, which are described below.

Table 5 Summary Values of Valuation Ratios for the SPWC Index (1870–1999) and Two Subperiods (1870–1935 and 1936–99)

Open New Window

The reconstructed and revised annual data for the broadly defined S&P composite are presented in appendix table A1. The prices are year‐end using the base, and the dividend and earnings levels, adjusted to the index, are for the indicated calendar year. The dividend and earnings levels from 1871 through 1940 are based on Cowles (1939, 1941, 1942), and for 1957 through 1999 are from S&P. The levels from 1941 through 1956 have been estimated by the authors, with dividend levels based on the CRSP NYSE estimates and earnings based on Moody’s estimates.

The dividend yields provided are defined as the dividend in the current year divided by the price at the end of the previous year, expressed as a percent. The dividend payout rate is the indicated year’s dividend divided by the earnings for that year and expressed as a percent. The Price‐Earnings (P/E) ratio represents the price at year end divided by the earnings for that calendar year, or the one‐year trailing ratio.

Table 5 summarizes the valuation ratios for the Standard & Poor’s Weekly plus Cowles (SPWC) price, dividends, and earnings for the complete 129‐year period. For comparison, the summary valuation ratios are provided for two subperiods that are split at December 1935. Because of the volatility in these ratios and large skewness and kurtosis measures, the summary statistics are given in terms of quartile values as opposed to means, with variance measured as the interquartile range. The interquartile range includes 50% of the values, with 25% of the values being above the upper quartile and 25% below the lower quartile.

The median value of the dividend yield for the complete period is 4.57%, and for the period 1871–1935 is 5.12%, whereas for the 1936–99 period the median is 4.10%. The interquartile range for the complete period is 1.57%, with a lower quartile of 3.92% and an upper quartile of 5.49%, with the range being similar for the early and the late period. These yields are in stark contrast to the decade of the 1990s, where all of the yields are less than the lower quartile for the period. In the 1990s, dividend yields range between a high of 3.69% in 1991 and a low of 1.36% in 1999. The dividend yield in 1999 was the lowest level over the 129 years.33

The median payout rate for the complete period is 58.77%, with the lower quartile at 51.90% and the upper quartile at 70.52%, with an interquartile range of 18.62%. The highest payout ratio was 929.12% in 1932 when earnings were near zero. The payout rate was more than 100% in 8 of the 129 years. For the 1871–1935 period the median rate was 67.42% and for 1936 through 1999 was 54.43%. The decrease in the payout rate between the early and late period is similar to the decrease in dividend yields between the two periods and again is in contrast with the 1990s where the payout rate in 1999 was 34.65%, the smallest over the complete period.

The median P/E ratio over the 129‐year period is 14.28, with a lower‐quartile value of 11.33 and an upper‐quartile value of 17.81. The lowest was 5.53 in 1917, and the highest, 136.50, occurred in 1932 when earnings were near zero. For the early and the late periods the median was 14.28 and 14.29, respectively.34 The ratio in 1933 of 33.66 is the second highest level, and the ratio in 1998 of 32.60 is the third highest, followed closely by the 1999 ratio of 30.50.

Appendix table A1 provides the annual appreciation returns, total returns with dividends reinvested monthly, and the total returns index with monthly reinvestment of income. The summary statistics for these returns were presented in table 4 above.

IX.  Conclusion

In this article we reexamine carefully the entire history of the S&P Composite Index as well as develop a corrected and improved composite series from 1871 through the present. The S&P 500 Index price and return data have often been misrepresented, and confusion still continues about their historical coverage. Furthermore, the extensions back in time using Cowles data that have been published in numerous sources have been confusing, based on averaged monthly data rather than month‐end prices and often failing to incorporate Cowles’s own corrections to his data.

We have been able to reconstruct the S&P weekly index, which, unlike their “official” index containing only 90 stocks for the period 1926–56, contained more than 400 stocks for most of that period. There is no S&P composite index of 500 stocks until 1957, regardless of any reconstructions that can be accomplished.

Cowles made an important contribution to the study of returns over long periods of time by extending the original S&P data backward in time to the beginning of 1871. In subsequent reports Cowles corrected errors in his data, which have not been picked up by most sources reporting this historical series. For example, although Standard & Poor’s reports their historical series based on Cowles’s data, S&P has consistently failed to use the subsequently corrected Cowles data. The uncorrected S&P series, in turn, has been used by Shiller and by Siegel, as well as others, in studying stock returns over long periods.

We have reconstructed the Cowles data based on his updated and corrected series. We have also attempted to reconstruct the data to deal with time‐averaging biases, going beyond Schwert’s attempt to do so. Averaged data have a downward‐biased variance and built‐in first‐order autocorrelation. Extensive analysis and adjustment were required to produce a series that avoids the averaging problem back to February 1885.

Standard & Poor’s did not estimate dividends and earnings for their price series prior to 1925, and then only for the 90 stocks, but Cowles did so for the period 1871–1940. We have used Cowles's estimates of the monthly price index and cumulative monthly wealth index to estimate the dividend amounts adjusted to the index, and carry the dividend and earnings estimates through March 1957, where we pick up the S&P 500 estimates.

Our analysis of the various series dealing with S&P composite data indicates that significant differences do exist when measuring returns over long periods. In particular, the use of only 90 stocks by Standard & Poor’s, as duplicated by Ibbotson Associates and others, causes significant differences in both appreciation returns and cumulative total returns. The S&P 90 stocks fared better than stocks in the broader weekly index and in the CRSP NYSE index during the period 1926–56. As a result, the narrower coverage indexes of the S&P historical series and of Ibbotson Associates outperformed the broader measures over a long period of time and effectively overstate the returns to a broader composite index of S&P stocks.

Researchers interested in the long time series offered by S&P composite data, as extended by the Cowles data, should be aware of the nature of the time series and the differences in results that can occur between the narrow coverage of the 1926–56 period and the broader coverage of other series. They should also understand the limitations of the data. In particular, they should understand that several well‐known reported series using Cowles's data are based on the earlier edition of Cowles’s work rather than Cowles’s own corrected data, and that the Cowles data are averaged data, which have potential statistical problems.

Our alternative stock return data provide new information and a new time series that can be analyzed in connection with long‐term studies involving stock returns. Hopefully we will see improvements in the data that are being analyzed and reported on, and better recognition of the possible limitations in these data.