2.8 Confidence bounds and statistical tests - based on an eight-State, paired-comparisons estimator

The Appendix of this report offers alternative procedures for computing point estimates of CHMSL effectiveness as well as their standard errors, by calendar year. As described in the Appendix, the method is to aggregate the data for all eight States, estimate the effectiveness of CHMSL (controlling for vehicle age) by a simple ratio-of-ratios of four cell counts from two adjacent CY, and, where necessary, use only half of the available data for the estimate in each CY in order to assure that no crash case is used in more than one of the estimates. In contrast to the preceding section, every estimate is based on a standard estimation formula and can be assumed (in view of the large sample size) to have normal properties, and every estimate is statistically independent from all the other ones. Thus, we can be sure that the customary z statistics will not understate the true confidence bounds. On the contrary, because the data were used quite inefficiently, the level of uncertainty is in a sense overstated. But there is "room to spare": even with these "relaxed" confidence bounds, CHMSL effectiveness is shown to be significant throughout 1986-95, and significantly lower averaged over 1989-95 than in 1987-88.

The strategy in this section is to apply the standard errors from the Appendix to the point estimates of this chapter. Alternatively, it would have been possible to use the point estimates from the Appendix, (which are quite similar to those from this chapter); however, it is believed that the point estimates in this chapter still represent the "best" estimates (most accurate, least biased) of CHMSL effectiveness.

The first finding of the Appendix was to accept the null hypothesis that CHMSL effectiveness remained unchanged throughout 1989-95, and to calculate a standard error for the pooled 1989-95 result. The standard errors for CHMSL effectiveness in the Appendix, by calendar year, are:

Calendar Year Standard Error
1986 .01319
1987 .01211
1988 .01187
1989-95 combined .00729

Application of these standard errors, using the customary critical value of 1.96 for z, generates the following confidence bounds, that will also be used in the Executive Summary of this report:

CY Point

Estimate

Std

Dev

Confidence Bounds
Lower Upper
1986 .0506 .01319 .0247 .0765
1987 .0853 .01211 .0616 .1090
1988 .0716 .01187 .0483 .0949
1989-95 .0433 .00729 .0290 .0576

In other words, the effect of CHMSL is statistically significant in 1986 (z = 3.84), 1987 (z = 7.04), 1988 (z = 6.03) and 1989-95 (z = 5.94). Moreover, effectiveness is significantly higher in 1987 than in 1989-95 (the difference in the effectiveness is .0420, the standard error of this difference is .01413, and z = 2.97). The average effectiveness in 1987-88, .07845 (with standard error .00848), is also significantly higher than the effectiveness in 1989-95 (z = 3.14).


2.9 Comparison with earlier analyses

Table 2-6 shows that, over the full 1986-95 time frame, CHMSL reduced police-reported rear impacts in eight States by an average of 5.11 percent. This result is remarkably consistent with Farmer's principal finding that CHMSL reduced insurance-reported rear impacts by 5.1 percent during the 1986-91 time frame [9].

Farmer was noncommittal on whether the effectiveness of CHMSL changed over time. On the one hand, he observed that "except for calendar year 1991, there is a ... steady decrease in effectiveness." The observed decreases, however, were not large relative to the sampling error in his study (which was based on substantially fewer crash records than this report) when the data were disaggregated by calendar year. "Thus, if there is a novelty effect, it is not strong enough to be detected [i.e., conclusively, with statistical significance] from [his] data." Nevertheless, the observed calendar-year trends in his results are consistent with the trend here.

This report's 8.5 percent reduction for CHMSL in CY 1987 is somewhat lower than the 11.3 percent reduction found in NHTSA's Evaluation of Center High Mounted Stop Lamps Based on 1987 Data [19] (16.95 percent reduction of "relevant" rear impacts = 11.3 percent reduction of all rear impacts). As discussed in Section 2.3 of this report, NHTSA's earlier study assumed a linear vehicle age effect, and thus overcorrected for vehicle age. With a more accurate vehicle age correction, the results of NHTSA's earlier study would have been quite close to this report's findings for CY 1987.

The only finding in this study that differs from expectations is the relatively low 5.06 percent effectiveness for CHMSL in CY 1986. In general, the intuition is that CHMSL effectiveness may have decreased, over time, as an ever increasing proportion of the vehicle fleet was equipped with them: from the very high effectiveness seen in the pre-1986 test fleets, to fairly high levels immediately following the effective date of CHMSL as standard equipment, to the "long-term" effectiveness in 1989 and later years. Yet, in this study, the observed effect in 1986 is significantly lower than in 1987. Also, at first glance, this study produces a substantially lower estimate than NHTSA's preliminary evaluation of CHMSL, which was based on a relatively small but nationally representative sample of police-reported crashes during the summer of 1986 and attributed a 14.8 percent reduction in rear-impact crashes to CHMSL [18], p. 11. The two studies use similar types of data and similar analysis methods. However, the current study is based on about 25 times as many crash records, and has far less sampling error. In fact, confidence bounds on the estimate in NHTSA's preliminary evaluation would extend down to 7 percent or less - comparable to the results of this chapter. Additionally, Farmer's results for CY 1986 seem to be more in line with this report than with the preliminary NHTSA evaluation. It must be concluded that the estimate of this chapter is more precise than NHTSA's preliminary assessment of CHMSL effectiveness in CY 1986.