Are RCTs Enough?

I’ve noticed that the gold standard in Microfinance research seems to be the “randomized control trial”, offering an unsurpassed level of experimental sophistication. From a statistical viewpoint, a randomized trial of a sufficiently large population allows researchers to ascribe causality. This is important in the impact evaluation of a policy.

While RCTs may be new to economics, they are nothing new to the natural sciences which have for a long time emphasized the importance of randomization. However, trials in the natural sciences (especially medicine and theoretical physics) control two one other crucial biases – the placebo effect and observer expectation. This is accomplished through the “double-blind trial” in which neither the sample nor the researcher know whether they belong to the trial or the experiment.

This is incredibly easier in the natural sciences (after all, the whole treatment is contained in a pill, and easy to replicate). However, recent speculation, suggests that the placebo effect might be quite as present and important for economic issues. The economist Tyler Cowen even suggests that the placebo effect could be employed to overcome issues of macroeconomic importance such as consumer confidence.

Is there a chance (researchers please comment) that this effect applies somehow to microfinance trials as well? Placebo effect, after all, is a result of expectation – do these trials somehow form an expectation that is self-fulfilling as it is in medicine?

Double-blind trials achieve one other thing – remove observer-expectation bias. Modern physics lends a beautiful example of how elegant research design can overcome this bias. In neutrino experiments, researchers have a bias of the total number of neutrinos, N, that they expect to see. Therefore, even if many different researchers are shown the whole sample, the observation would systematically conform to the expectation. Rather, each researcher is only shown a theoretically meaningless N’, which is a random fraction of the total sample. Because this is a meaningless value, there exists no expectation and hence the overall value (an aggregate of each answer) is not biased.

I wonder if something similar can be done in microfinance. I am not familiar enough with the actual research methodology to comment, but would be interested to hear from researchers regarding the above two points. And, if theoretically it is possible to conduct a double-blinded RCT – is it practically possible, or would it take too much effort. Inevitably a double-blinded trial has a large financial burden, but in medicine I would never trust to be reliable anything that has not been rigorously tested in a double-blinded setting. Would you say the same about economics?
social science methods 6449133078381778455

Post a Comment

  1. It is not "trial or experiment" it is placebo group or treatment group.
    The Placebo effect is no simple matter, it subdivides in different effect e.g. Hawthorne effect (the effect that being observed changes the state of the matter - even true in natural sciences).

    You cannot do a double-blind trial in economics.
    How do you expect to do that? People know whether they get a subsidy or not, etc.

    There are also a bunch of biases you should know about:
    -attrition bias
    is normal in economics or medicine (particles in physics don't leave your experiment because they are fed up, people do!).
    You need to plan for this and make sure it does not happen more in one group than in the other.
    -skewed data
    Sometimes your outcome data is skewed and researchers like to report what is convenient for them. The mean and the median might differ quite a bit if the data is skewed.
    You should carefully think whom you consider!
    Microfinance might be beneficial for the farmer or the village receiving the credit but destroy economics of scale and therefore be bad for the whole society.

    Overall there is only so much what can be written in a comment box. You definitely should read about RCTs and their procs and cons, microfinance and economics and statistics in general.

    1. I wrote a few days back on this blog that researchers in their quest of knowledge or to prove something new, tend to over-theorize. It is up to them to clearly set out their objectives, whether their quest is for statistical robustness or to do a more "realistic" impact evaluation. Whether they want a journal publication and conference audience or they want to communicate to a wider non-academic audience as well. There is this piece of blog by a CMF Program Head (not me)which may be worth a read.




Home item

Sign Up for Updates

Twitter Updates


Flickr Photostream