Can Qualitative Research be Rigorous? Part 2: The Value of Qualitative Research Methods for Impact Evaluation
For
the remaining segments of this series, I’m going to specifically focus on the
role of qualitative research in the context of impact evaluation. It is often
the case that qualitative methods go under-appreciated in this context – after
all, the focus is usually on outcomes – and it’s not always obvious how
qualitative research can contribute. Yet, qualitative research methods can be
incredibly useful for impact evaluations. In this section, I’ll discuss a few
ways in which qualitative research can help make an impact evaluation stronger.
Briefly,
let’s consider why qualitative information is useful in the first place.
Anyone
who has had the dissatisfaction of receiving a single score on a written exam,
essay or performance evaluation, with no further explanation, can probably
understand the value of qualitative information. Let’s say this score was high,
for example 98 out of 100. Not so bad, you think to yourself, and maybe you
don’t worry too much about it. But still, some small part of you may wonder
“Why were those 2 points deducted? And why was it that I did so well anyway?”
If the score was lower than you expected, the situation is more troublesome.
You don’t know what expectations you failed to meet and it isn’t clear how you
can do better next time. You may even feel the score was unfair. By including a
few sentences in addition to an exam or test score, a teacher can convey: why
he assigned the score, what he think went well (and should be continued) and
what went wrong (and needs to be improved.)
A similar logic applies to impact evaluation. Sometimes impact evaluations measure a change in outcomes, without providing contextual information. For example, a study might measure the impact of self-help groups on monthly earnings and report that self-help groups caused members to increase their monthly earnings by 25 percent. This would be great news and valuable from a policy perspective, but maybe we want to know more than simply whether or not the program was effective at improving outcomes. Maybe we want to expand the program so it reaches more people and would like to understand the conditions under which the program was successful. We might want to know the program’s staff structure, its outreach methods, the beliefs and motivations of participants who join the program, components of the program that participants found particularly valuable, or challenges the program faced and how these were overcome. Qualitative research methods provide a structured means for obtaining this sort of information.
A similar logic applies to impact evaluation. Sometimes impact evaluations measure a change in outcomes, without providing contextual information. For example, a study might measure the impact of self-help groups on monthly earnings and report that self-help groups caused members to increase their monthly earnings by 25 percent. This would be great news and valuable from a policy perspective, but maybe we want to know more than simply whether or not the program was effective at improving outcomes. Maybe we want to expand the program so it reaches more people and would like to understand the conditions under which the program was successful. We might want to know the program’s staff structure, its outreach methods, the beliefs and motivations of participants who join the program, components of the program that participants found particularly valuable, or challenges the program faced and how these were overcome. Qualitative research methods provide a structured means for obtaining this sort of information.
Consider
another scenario – this time a study which was sufficiently powered to detect
small effects of a very promising program. Despite high confidence on the part
of program developers, staff and stakeholders, the study found no impacts on measured
outcomes for program participants. Should we conclude that the program was a
bad idea and shut it down? That’s a big decision. Perhaps first, we’d like to
consider other possible explanations for why the study found no program related
impacts. For example, perhaps the program model itself was promising, but
complications arose in implementation – maybe lack of funding or situational
constraints caused the program to remove or modify some important components,
or perhaps external factors prevented the program from reaching all of its
targeted participants. Another explanation might be that program participants
did benefit from the program – but in ways that were unexpected and therefore
weren’t measured in the study. Qualitative research can help provide these
kinds of explanations for impact study results, which are particularly helpful
when no significant impacts were found.

Keep
in mind that this discussion is not intended to show all the ways in which
qualitative research can be utilized or incorporated into impact evaluation
studies. It also does not argue for qualitative methods to replace quantitative
analysis or hypothesis testing. Rather, it highlights common examples of how these
two research forms can work together as complementary components of a
comprehensive impact evaluation.
Together,
the “quantitative” and “qualitative” approaches can provide a more thorough
understanding of a program’s operations and impact than either can by itself. Sometimes
it is necessary, or even more appropriate to use either a qualitative or
quantitative research approach instead of a combined, “mixed-methods” approach;
however, that is often the exception rather than the norm. Usually, research –
evaluation, in particular – that combines both quantitative and qualitative
approaches is able to tell a more complete, comprehensive and ultimately more
valuable story than research that favors a particular approach over the other.
Comments
Post a Comment