Welcome to part two of Evaluation for RFAs, RFPs and RFQs. In this section I’ll be discussing considerations for designing an evaluation to improve the performance or impact of the project you’re working on. The process for designing an evaluation is not unlike the process I’ve suggested you use to develop the proposed solution you are currently implementing for your client. There is, indeed, a method to the madness. As with any achievement-oriented endeavor, it’s best if you start with the objective in mind. As a side note, designing your evaluation really began at the moment you started putting pen to paper on your proposal for the project. If you have built your proposed technical solution correctly, you should already have the pieces in place necessary to evaluate the work.
There are six things you want to make sure to consider when designing an evaluation. First, determine what questions you want the evaluation to answer. For example, if you’re working on an agriculture extension project with smallholders, what are you most interested in learning about? Did the total number of small holdings increase? Did the aggregate output of existing smallholders increase? Was there a shift in the variety of goods produced? Have there been unintended externalities because of smallholder growth? As you can see, the types of questions you can ask about any particular program are almost endless. This is why you need to determine what you’re interested in learning specifically from this “focused research project” that you’ll be undertaking. I recommend limiting your major questions to five or less.
Second, you need to decide which type of evaluation will answer the questions you have posed. Performance evaluations are designed to identify accomplishments, performance issues, and constraints in the implementation of your project. They identify results and lessons learned in project implementation. An impact evaluation, in contrast, measures the change a program has had in the target population in terms of knowledge, attitudes, beliefs or behaviors. This change is measured against something called a counterfactual, which is an example of a similar situation not covered by your program.
Third, you need to decide what types of data you will collect and how you will collect it. This again, is dependent upon the type of evaluation you put together, and the data collection tools you use. As an example, some evaluations are skewed toward quantitative measures and look at large data sets such as statistics produced by government bodies. Other evaluations focus more on qualitative data, which is collected through methods such as interviews and focus group discussions. At ISG, the work we perform usually includes what’s called a mixed methods implementation that combines both qualitative and quantitative methodologies.
A fourth aspect to consider is how you will analyze the data you have collected. This is also known as your analytical framework. Again, if you’ve put your program together properly, it’s highly likely that you have one or more analytical frameworks already prepared because you have a good understanding of the levers you are using to affect the change in the target population.
The fifth area you need to consider are the practical aspects: how much time, money and what types of resources will you need to perform the evaluation properly? This will be greatly affected by your decision about whether or not the evaluation will be conducted internally, using staff and resources already on the project, or if it will be an independent evaluation conducted by external consultants or staff.
The sixth area to consider is how the evaluation will be used. For example, if you’re performing a midterm evaluation on a three or four year project, it’s likely the evaluation will be used to make adjustments and shifts in strategy on the program. Another example might be an end of term evaluation for the same project, which will be used to inform future programs of similar nature, contribute to the greater humanitarian aid and development literature or some other output.
Once you have designed your evaluation, you need to implement and manage it so the results can be used in some way that improves performance and impact. This is the topic of the next section of this training. Thank you for watching this training on evaluation for RFAs, RFPs and RFQs. Remember, if you have any questions at all, please contact us at training@Aidpreneur.com.