Skip to main content

Can Qualitative Research be Rigorous? Part 2: The Value of Qualitative Research Methods for Impact Evaluation

For the remaining segments of this series, I’m going to specifically focus on the role of qualitative research in the context of impact evaluation. It is often the case that qualitative methods go under-appreciated in this context – after all, the focus is usually on outcomes – and it’s not always obvious how qualitative research can contribute. Yet, qualitative research methods can be incredibly useful for impact evaluations. In this section, I’ll discuss a few ways in which qualitative research can help make an impact evaluation stronger.

Briefly, let’s consider why qualitative information is useful in the first place.

Anyone who has had the dissatisfaction of receiving a single score on a written exam, essay or performance evaluation, with no further explanation, can probably understand the value of qualitative information. Let’s say this score was high, for example 98 out of 100. Not so bad, you think to yourself, and maybe you don’t worry too much about it. But still, some small part of you may wonder “Why were those 2 points deducted? And why was it that I did so well anyway?” If the score was lower than you expected, the situation is more troublesome. You don’t know what expectations you failed to meet and it isn’t clear how you can do better next time. You may even feel the score was unfair. By including a few sentences in addition to an exam or test score, a teacher can convey: why he assigned the score, what he think went well (and should be continued) and what went wrong (and needs to be improved.)


A similar logic applies to impact evaluation. Sometimes impact evaluations measure a change in outcomes, without providing contextual information. For example, a study might measure the impact of self-help groups on monthly earnings and report that self-help groups caused members to increase their monthly earnings by 25 percent. This would be great news and valuable from a policy perspective, but maybe we want to know more than simply whether or not the program was effective at improving outcomes. Maybe we want to expand the program so it reaches more people and would like to understand the conditions under which the program was successful. We might want to know the program’s staff structure, its outreach methods, the beliefs and motivations of participants who join the program, components of the program that participants found particularly valuable, or challenges the program faced and how these were overcome. Qualitative research methods provide a structured means for obtaining this sort of information.

Consider another scenario – this time a study which was sufficiently powered to detect small effects of a very promising program. Despite high confidence on the part of program developers, staff and stakeholders, the study found no impacts on measured outcomes for program participants. Should we conclude that the program was a bad idea and shut it down? That’s a big decision. Perhaps first, we’d like to consider other possible explanations for why the study found no program related impacts. For example, perhaps the program model itself was promising, but complications arose in implementation – maybe lack of funding or situational constraints caused the program to remove or modify some important components, or perhaps external factors prevented the program from reaching all of its targeted participants. Another explanation might be that program participants did benefit from the program – but in ways that were unexpected and therefore weren’t measured in the study. Qualitative research can help provide these kinds of explanations for impact study results, which are particularly helpful when no significant impacts were found.

Qualitative research methods are helpful for providing context and explanations for our findings, but they can also help us at the start of an impact evaluation study. Often, we are inspired by a program idea or model, but it is not until we spend a few days observing the program, observing participants and speaking with program staff and participants that we begin to fully comprehend the program that we want to study. This initial period of getting to know and understand the program is also qualitative research. During this period, we not only come to understand the program better, but we also learn the right questions to ask and how to ask them. We learn what outcomes the program might reasonably be expected to change during the study period and how to adapt our outcome measures to fit the local context. We may even identify new impacts to study that weren’t previously considered. When we take the time to incorporate qualitative research methods in the initial stages of an impact evaluation, we can feel more confident that our study is appropriately designed, not just adequately powered, to detect changes related to the program we are studying. 

Keep in mind that this discussion is not intended to show all the ways in which qualitative research can be utilized or incorporated into impact evaluation studies. It also does not argue for qualitative methods to replace quantitative analysis or hypothesis testing. Rather, it highlights common examples of how these two research forms can work together as complementary components of a comprehensive impact evaluation.

Together, the “quantitative” and “qualitative” approaches can provide a more thorough understanding of a program’s operations and impact than either can by itself. Sometimes it is necessary, or even more appropriate to use either a qualitative or quantitative research approach instead of a combined, “mixed-methods” approach; however, that is often the exception rather than the norm. Usually, research – evaluation, in particular – that combines both quantitative and qualitative approaches is able to tell a more complete, comprehensive and ultimately more valuable story than research that favors a particular approach over the other. 

Comments

Popular Posts

Vocationalisation of education in India: Current Scenario, Key Challenges and New directions

“Every handicraft has to be taught not merely mechanically as is done today, but scientifically. This is to say, the child should learn the why and wherefore of every process.” - Gandhi’s Philosophy of Education

The greatest challenge in Indian education system today is to provide skill based education to the youth. This is exacerbated by a mismatch in demand and supply for the skilled workforce. The penetration of vocational education and training remains poor not only in rural areas, but also in urban regions where there is a higher installed capacity to impart the same. This post is an attempt to make the readers understand the need of vocational education in India. Also, this is an attempt to summarise a few recommendations on the same. 
A recent survey (61st round) conducted by the NSSO found that:

1. The percentage of population that completed primary education was 70%, but less than 10% went on to complete a graduation course and above. Almost 97% of individuals in the age bracket…

Rockstar of Financial Inclusion: Business Correspondent Model of India

About Author:  Jatinder Handoo is a social business enthusiast and a branchless banking practitioner. Currently works at FINO PayTech Ltd and is based out of Mumbai. He is reachable at jatinder.handoo@fino.co.in
India is a hot bed of financial exclusion. A country which houses nearly 16% of the global population  has more than 65% of its people outside the formal financial system (Global Findex 2012). The Indian banking system has adopted multiple approaches to make universal financial inclusion a reality right from early days Indian post-independence banking system. Be it bank nationalization in 1969 or formation of Regional Rural Banks. Formation of NABARD or fostering microfinance through Bank-SHG linkage programme in early 90’s. A shimmering ray hope was rekindled with the growth of JLG based microfinance, however later studies made it clear that the model is credit led, concentrated predominately in the southern region of India thus could not be seen as painting complete financial…

A Platform for Knowledge - Enabling people to learn ..

I received a rather interesting link/website via my email today. The link read as MR University and all I could think of was, "Ok, this must be another website portal of some university or college". Well, on clicking the link and looking through the contents of the site, I was pleasantly surprised. The site http://www.mruniversity.com/ is an online education portal or platform that allows users or teachers to upload short videos on topics or lessons they wish to impart. First topic that I come across is Development Economics.
The intent of the website is eloquently put out by the two economists, Tyler Cowen and Alex Tabarrok in the intro video. What started as a blog focusing on economics and its various implications in understanding why things are the way they are around us, has now an interesting addition. A video portal titled MRUniversity or Marginal Revolution University that focuses on online education with subjects pertaining to economics. It brought back to my mind,…