You’re probably thinking that this Guide is glossing over the really hard parts of evaluation just to trick you into taking the plunge. Well you are right—but not because we are trying to trick you. There are hard parts to evaluation and they can create real nightmares if they are not done right. But there is no point in trying to teach you how to do them right in a little guide that you can fit in your pocket.
Evaluation is one area where it makes sense to bring in outside expertise. It will save you considerable time and effort, and will increase your probability of success significantly. Ah, but what about the opening scenario when the manager delegated responsibility for the evaluation to a volunteer from another office and it was a disaster? Obviously, you can’t be so eager to delegate the evaluation tasks that you hire the first person that comes along.
What Can (and Should) an Evaluation Specialist Do for You?
You want your evaluation specialist to:
- Design the evaluation
- Recruit and train the data collectors
- Collect the data
- Provide interim feedback to you during the conduct of the program
- Analyze the data and present the findings
- Provide input to you as you draw conclusions
Since the evaluation should be designed right along with implementation, you want your evaluation specialist working with you at the beginning, when you establish your program objectives. Your evaluation specialist can help you focus on what can be measured and what evaluation questions you will be able to answer.
The evaluation specialist will also be able to counsel you about the problems you might encounter gathering or analyzing particular types of data. For example, the evaluator will recognize the potential effect of seasonal differences in driving levels, or the impact political changes might have on enforcement levels. Beyond recognizing the potential problems, however, the evaluator will know how to deal with them.
With an experienced evaluator on board, you will not have to worry about the design of surveys, sampling plans or data collection forms, or the training of your collectors , or the appropriate analytical procedures that should be followed. You of course, need to stay in touch with the evaluation to make sure that the processes and products are in line with your expectations. You will also need to listen to your evaluator when she points out the problems with your carefully thought out plans. An evaluator is trained to be objective and you should take her concerns seriously.
What to Look for in an Evaluator
If you are like most managers, you get a little nervous delegating responsibility for a critical activity to someone you don’t know very well. The anxiety increases significantly if the activity involved is highly technical and is outside your area of expertise, because you know you can’t just step in and take over data analysis if there is a problem. The way to overcome this anxiety is to have a very clear understanding of what to look for in an evaluation specialist. The following is a list of criteria you can use for selecting an evaluation specialist.
When you are hiring an evaluation specialist, look for
- Someone who explains things in terms you can understand
- If you can’t understand him or her in the interview, you won’t understand the final report. Even if someone gets high marks on all the other criteria, pay attention to this one.
- Someone who understands characteristics and limitations of traffic safety data
- Some evaluators waste time and resources trying to make traffic safety data do the impossible. You also don’t want your evaluator to become educated on your nickel. You should give your evaluator a copy of the Compendium of Traffic Safety Research Projects 1987-1997 (NHTSA, Document DOT HS 808599) for more detailed information about what has been learned about traffic safety evaluations.
- Someone with previous evaluation experience, particularly in use of behavioral observations, public opinion questionnaires, and analysis of archival data
- Read reports your evaluator has written to assess writing skills.
- Talk to program managers for these projects to assess the evaluator’s performance.
- Look for examples that resemble the type of work you will be doing.
- Someone who fully understands research design and statistical techniques and when they should and should not be applied
- Good research design is always needed, even on simple evaluations.
- Statistical analyses may or not be appropriate, depending on what you are trying to measure. Describe your situation and ask the evaluator what he or she would recommend. Ask for as description of situations which would not require statistical analyses. If the answer is “Statistical analyses are always required.” that may indicate a problem.
- Someone who can present results clearly, both verbally and in writing
- Ask for a variety of writing samples
- Ask the evaluator to briefly explain a recent project to you. Watch for the use of technical jargon. Ask a lot of questions to measure the evaluator’s patience. This individual may need to represent you someday in a meeting. How will he or she do?
- Someone who is skilled in presenting quantitative information graphically so that it highlights key issues
- Ask for a report with a variety of graphs and charts. Are they easy to understand? Do they make things clearer or more confusing? Ask the evaluator to explain them to you to see how the he or she can communicate complex information in simple terms.
- Someone who can get access to data
- You want your evaluator to make your job easier for you, not harder. A good evaluator will already know how to get access to a variety of data sources. He should not be relying on you to locate the most appropriate sources and to negotiate access to the best sources. If a candidate has never worked with any of the data sources listed below, does he or she at least seem capable of figuring things out independently?
— Archival data (FARS, NASS, State data files)
— Police Reports
— Court Records
— Medical Records
- Someone who can get access to data collectors
- You want your evaluator to have experience dealing with data collectors and all the challenges they can provide. Potential data collectors could include students, volunteers from the community, and temporary workers.
- When a manager is intimidated by the whole concept of evaluation, he or she tends not to ask the nagging questions for fear of looking uninformed. This can be disastrous! While you are delegating the tasks associated with evaluation, you cannot delegate your own responsibility for managing all aspects of your program. Therefore, it is critical that you hire an evaluation specialist with whom you are completely comfortable. You should feel free to ask any questions that occur to you, no matter how fundamental, and you should understand every answer that is given. If you don’t have that relationship with a potential evaluator, keep looking!
Where Should You Look for Evaluation Assistance? (see Links to Sources)
With your criteria for selecting an evaluator in hand, you can begin your search for an evaluator right in your own backyard. You should explore the resources in other departments of your own agency and in other agencies within your jurisdiction. However, don’t make the mistake the manager in the opening scenario did. Examine an in-house evaluator with the same rigor that you would an outside consultant. A few college statistics courses do not qualify someone as an evaluator.
There are a variety of other sources of evaluation expertise that you can explore as well. They include:
- Local colleges and universities
- Check with the Education, Psychology, Business, Public Health, and Epidemiology departments at nearby universities. The whole discipline of program evaluation originated in the field of education. You should be cautious recruiting in a Math department because of their focus on theoretical statistics rather than applied statistics.
- You should expect to pay for the evaluation services you obtain from local universities, but they may be an excellent source of low-cost data collectors and graduate students. You will need to review the procurement procedure in your community to determine how long it might take to contract with the university. Be sure to clearly specify milestones and due dates so that your project doesn’t get lost in the shuffle.
- Private consultants
- There are numerous consultants in the private sector with evaluation expertise. The firm you hire should have traffic safety evaluation experience, and should be familiar with the archival data sources that you will be using.
- Depending on your procurement requirements, you may need to conduct a competition in order to hire a private consultant. This may take several months to complete. Be sure to allow enough time for this process. You don’t want your evaluator coming on board several months into implementation.
- Private consultants do not have the schedule limitations caused by the semester breaks and summer vacations at universities.
Working with an Evaluator
Before you even start looking for an evaluator, you should prepare a clear statement of work with the specific tasks to be accomplished, a description of all deliverables, and a schedule for their completion. This document will serve as the foundation for your relationship with your evaluator. It should be as specific as possible so that there are no misunderstandings down the road. You will also need to prepare a budget for your evaluation, but this should be negotiated with your evaluator.
It is reasonable for you to expect your evaluator to complete all tasks on time, or to notify you in advance if there will be any delays. Likewise your evaluator should be able to expect you to complete any reviews of deliverables on time. (The schedule for your project should include feasible review periods.) If you have changes that you want, you need to be very specific.
Meet with your consultant regularly, either in person or over the phone. This demonstrates that you are interested in what she is doing and that you want to be involved in any major decisions that need to be made. At the same time, avoid the temptation to micro-manage. You should be focusing your attention on the overall implementation, not the details of the training for data collectors. If there is a problem, your evaluator should tell you about it. Think about how many reports you want your consultant to provide you. Frequent progress reports are essential if they are the only form of communication you have with your consultant, but if you can meet face to face every week, progress reports become just an extra burden that takes time away from your evaluation effort.