Evaluation is like regular exercise. We all know that exercise is good for us. And we all try it from time to time. But the majority of Americans fall far short of the recommended exercise requirements. Why is that?
In most cases, we don’t exercise regularly because we have convinced ourselves that exercise requires too much effort, that it will hurt, and that it probably won’t give us a perfect body anyway.
It’s the same way for evaluation. Everyone acknowledges that it is always a good idea to evaluate any program that uses taxpayer dollars, but when it comes time to build an evaluation into a program plan, dozens of excuses are offered as to why it just can’t or shouldn’t be done, in this particular case. So why do many people shy away from conducting program evaluations regularly?
Most excuses for not doing an evaluation are variations on the following four themes.
Do some of these sound familiar? Have you found yourself thinking these thoughts? Let’s go through them one by one to show you why they aren’t true and may be standing in the way of your success in traffic safety.
“Evaluation is too complicated.”Many are intimidated by the whole concept of evaluation. A mystique has built up that program evaluation is very complicated with a hundred ways to do it wrong and only one, very difficult, way to do it right. And in some cases this mystique is justified. When you are talking about establishing a direct cause and effect relationship between a specific traffic safety countermeasure and a reduction in traffic deaths, you need a solid research design with random assignment to experimental and control groups and sophisticated statistical analyses. More importantly, you need large numbers of cases in order to detect any real change in traffic deaths.
In reality, however, traffic safety evaluation need not involve “Ivory Tower” laboratory science. Traffic safety evaluation is an applied science that works within the constraints of state and local program implementation. Most local communities simply do not have the volume of traffic deaths and injuries to conduct that kind of countermeasure effectiveness evaluation. Instead, these communities can focus their evaluation efforts on determining if the particular program they implemented achieved its specific objectives.
If you are implementing an occupant protection program, your evaluation dollars would be better spent demonstrating an increase in safety belt use rather than proving, once again, that safety belts save lives. Documenting an increase in safety belt use over baseline levels involves a much simpler evaluation and should not deplete your program resources.
“I may not like the answer so I better not ask the question.”
When you implement a program, you put a lot of yourself into the effort. You believe in your heart that it is a good program, and you do everything you can to make it work. But you never know what might happen.
Some people shy away from evaluations because they don’t want their good idea to be proven wrong. The mistake they are making is viewing evaluation as the last step in a process, like a final exam. If evaluation is tacked on at the end of the project, you may very well come up with answers that you don’t like.
The key to successful evaluation is to build evaluation in from the start, so it can help you frame the questions you are asking, and even clarify the problem that you are trying to solve. A well-planned evaluation should not yield last minute unpleasant surprises. Instead, it will provide useful information that helps fine-tune the program at every step of the way. It also tells you what’s working and what can be made better.
“I have a limited budget; I prefer to spend my dollars on implementation.”
These days, everyone is expected to do more with less. Project directors struggle to stretch every program dollar to the limit. When asked to choose between delivering more high school presentations or conducting a program evaluation, many choose more presentations because they believe that their impact will be greater if they can reach more kids. There are two errors in this logic.
Another point to keep in mind when thinking about conserving program dollars is satisfying your funding source. When it is time to ask your funding source to extend your program for another year, your proposal will be much more persuasive if it is backed up with solid data demonstrating that you accomplished your objectives.
“Evaluation is too much work.”
Evaluation can be labor-intensive (note that we didn’t say expensive) and very tedious. This is why evaluation is one of those areas that, as a project director, you should delegate to someone else. This could be an employee from another department in your agency, a faculty member or graduate student at a local university, or a private evaluation consultant. In addition to doing the time-consuming work involved in data collection, you should expect an evaluation specialist to be able to explain the results in language you and everyone else can understand. There is no place for gobbledygook in an evaluation report. (Section V of this Guide talks about what to look for in evaluators and how best to work with them.)
Now that we have eliminated all your old reasons for not doing an evaluation, we can concentrate on some new reasons for doing one.
Let’s go back to our exercise analogy, for a moment. We all know some people who are fully committed to getting regular exercise. They don’t have to think about if they will exercise or how they will fit it into their schedule. Exercise is an essential part of their daily existence, just like eating and sleeping. These individuals report that exercise improves every aspect of their life, not just their physical conditioning. They have more energy, they are more productive at work, and they are less prone to depression. What separates them from the rest of us is their exercise mentality.
To get the full range of benefits from evaluation, you need to have an evaluation mentality. This means that you never even consider implementing a program without first thinking like an evaluator. You approach each new problem with the same set of questions:
With the answers to these questions in hand, you are prepared to convince any funding source that you know what needs to be done and that your ideas have a high probability of success. Your evaluation mentality will also ensure that at the end of this project, you can report back to these same funding sources with solid information on what you accomplished.
An evaluation mentality cannot ensure that every project you implement will be a resounding success, but it can ensure that you fully understand what you tried to do and why things turned out as they did.
If you have an evaluation mentality, you design your program and your evaluation at the same time. The benefits of this approach are substantial. An evaluation mentality will enable you to:
You do not need to be an evaluation specialist to have an evaluation mentality, just as you don’t have to be body builder to have an exercise mentality. You just have to recognize that evaluation, when built in from the beginning, provides benefits throughout the life of your program.