Skip to main content

Can Qualitative Research be Rigorous? Part 3: Sample size

Since the last post, I’ve had the opportunity to witness a couple of dialogues on the quantitative vs. qualitative debate; some were more encouraging than others. The first was during a panel session at the fall conference in Manila, where I witnessed one presenter’s harangue against the perpetrators of quantitative research, particularly those involved in RCTs. The main point of her talk was, I believe, to advocate for the value of qualitative research and point out that RCTs often miss subtleties that qualitative research is well-positioned to address. I agreed with the main point, but was disappointed with her approach. Rather than convey a way to bridge the two sometimes disparate fields, she pitted them against each other, pronouncing qualitative the winner. I observed more than one person enthusiastically rallying against RCTs after this session and I thought that was an unfortunate turn of events. 

To those who still see qualitative and quantitative methods as fundamentally different and opposed to each other, I’d like to offer two examples of research that have integrated qualitative and quantitative methods in a unique and meaningful way. In the first case, researchers used qualitative research to assess the measurement error in surveys of illicit behavior among street youth in Liberia. These researchers found that they could verify the results of self-reported data through observations and informal interviews, for relatively little cost. In the second case, researchers developed comprehensive life trajectories through qualitative interview techniques, to understand how major life events coincide with financial well-being. Researchers used this qualitative research to provide the more nuanced and detailed insights that their larger panel study failed to achieve.

The second discussion I witnessed was among a much smaller group of researchers clustered together for an entirely different purpose. Here, we listened to the representative of one prominent development research funding institution discuss the realities of mixed methods research. According to the representative, while an increasing number of evaluations are including qualitative research, often times the qualitative component is given less attention. Frequently, the qualitative work is done to help draft a survey better (a valuable use!), but isn’t used in the larger study to answer more complex or nuanced questions. Moreover, the qualitative research is often not well designed or thought through. I found this discussion particularly relevant to the theme of this series. 

It is not always obvious how to make qualitative research rigorous, how to make it meaningful or how to do it well. Truthfully, like all other types of research, qualitative research is an art and one generally improves with experience. Still, it is helpful to have some guidelines. I will cover some main points that I find helpful for keeping qualitative research rigorous. Keep in mind that I am not an expert, just someone who values qualitative work and has learned a bit along the way. I should also point out that most of what I talk about will refer to interviews, because that is what I do. But I think it can be applied to other methods as well. This post starts with a discussion of sample size and the next post will cover techniques for data collection and analysis. For those who are interested in learning more, I’ll provide resources along the way and at the end of each post.




Picking the Right Sample Size 
Qualitative research has some ambiguous qualities. One of them has to do with sample size. Unlike studies that capture changes in quantities, qualitative studies have no clear formula for picking the “appropriate” number of respondents. Indeed, there is not even a universally agreed upon method for selecting a sample. Some say sample size doesn’t matter; others say 30 is the magic number; while an entirely different approach suggests one should keep going until the responses start repeating each other. Sample size, considered so crucial to the construction of a sound quantitative study, immediately proves itself to be a stumbling block for establishing “rigor” in qualitative research.

Typically, we engage in qualitative research because we want to understand a phenomenon better. Our questions can’t be answered with a yes or no or multiple choice response; we want to know “what?”, “why?” or “how?” The trick here is that each observation is unique. One person or program site or village is not going to provide perfect information about the next. (This is, of course, the same variation between observations that we consider when we conduct power calculations or think of external validity.) People differ in how they understand and perceive the world around them, they differ in opinions, they differ in so many ways on the very things we want them to help us understand.  If you conduct one lengthy interview with one person, you will surely understand the phenomenon as applied to this one person, but you won’t have the full range of responses. In fact, you may have just accidentally interviewed the one outlier in the community.

The key question is: how much does the full range of responses matter? If it doesn’t matter at all, then the number of observations you choose for your qualitative research is probably not very important. If you are hoping to capture the full range of responses, or very close to it, then the situation becomes more tricky. If you use the “keep going until responses start repeating” approach, you just may miss out on the full range. Given what we know about probability, there’s a good chance that you may interview several people in a row who give similar responses due to…well, chance. So can you be sure that the repetitiveness is an indication that you’ve got the full range – or at least most of it?

The best bet is to avoid “rules of thumb” and to make a thoughtful, well-reasoned justification for your sample size, which is clearly explained upfront. Selection of sample size should match your goals and the type of research you are doing. There is a lot of guidance out there for identifying the appropriate sample size, which may vary according to your project’s goals and approach. One enterprising marketing consultant offers a formula for calculating sample size, based on the probability of missing infrequent phenomena. Although this approach has not yet made its way to academic circles, it certainly can be appealing to those who crave the satisfaction that only mathematical formulas can provide.

Note: Two researchers from the National Centre for Research Methods in the UK tried to address this sample size question by reaching out to leading qualitative scholars and soliciting their input. They obtained a variety of responses.  It is an interesting read and provides a nice foundation for thinking about sample sizes in qualitative research.

Further reading:
Patton, Michael Quinn (2002). Qualitative evaluation and research methods. Thousand Oaks, CA: Sage Publications.


Anwuegbuzie, Anthony J. & Leech, Nancy L., “A Call for Qualitative Power Analysis,” Quality and Quantity 41 (2007): 105-121.

Marshall, Bryan, et al., “Does Sample Size Matter in Qualitative Research? A Review of Qualitative Interviews in IS Research,” Journal of Computer Information Systems (2013): 11-22.

Can Qualitative Research be Rigorous? Part 1: What is Qualitative Research?

Can Qualitative Research be Rigorous? Part 2: The Value of Qualitative Research Methods for Impact Evaluation



Comments

Popular Posts

Vocationalisation of education in India: Current Scenario, Key Challenges and New directions

“Every handicraft has to be taught not merely mechanically as is done today, but scientifically. This is to say, the child should learn the why and wherefore of every process.” - Gandhi’s Philosophy of Education

The greatest challenge in Indian education system today is to provide skill based education to the youth. This is exacerbated by a mismatch in demand and supply for the skilled workforce. The penetration of vocational education and training remains poor not only in rural areas, but also in urban regions where there is a higher installed capacity to impart the same. This post is an attempt to make the readers understand the need of vocational education in India. Also, this is an attempt to summarise a few recommendations on the same. 
A recent survey (61st round) conducted by the NSSO found that:

1. The percentage of population that completed primary education was 70%, but less than 10% went on to complete a graduation course and above. Almost 97% of individuals in the age bracket…

Rockstar of Financial Inclusion: Business Correspondent Model of India

About Author:  Jatinder Handoo is a social business enthusiast and a branchless banking practitioner. Currently works at FINO PayTech Ltd and is based out of Mumbai. He is reachable at jatinder.handoo@fino.co.in
India is a hot bed of financial exclusion. A country which houses nearly 16% of the global population  has more than 65% of its people outside the formal financial system (Global Findex 2012). The Indian banking system has adopted multiple approaches to make universal financial inclusion a reality right from early days Indian post-independence banking system. Be it bank nationalization in 1969 or formation of Regional Rural Banks. Formation of NABARD or fostering microfinance through Bank-SHG linkage programme in early 90’s. A shimmering ray hope was rekindled with the growth of JLG based microfinance, however later studies made it clear that the model is credit led, concentrated predominately in the southern region of India thus could not be seen as painting complete financial…

A Platform for Knowledge - Enabling people to learn ..

I received a rather interesting link/website via my email today. The link read as MR University and all I could think of was, "Ok, this must be another website portal of some university or college". Well, on clicking the link and looking through the contents of the site, I was pleasantly surprised. The site http://www.mruniversity.com/ is an online education portal or platform that allows users or teachers to upload short videos on topics or lessons they wish to impart. First topic that I come across is Development Economics.
The intent of the website is eloquently put out by the two economists, Tyler Cowen and Alex Tabarrok in the intro video. What started as a blog focusing on economics and its various implications in understanding why things are the way they are around us, has now an interesting addition. A video portal titled MRUniversity or Marginal Revolution University that focuses on online education with subjects pertaining to economics. It brought back to my mind,…