Skip to main content

Are RCTs Enough?

I’ve noticed that the gold standard in Microfinance research seems to be the “randomized control trial”, offering an unsurpassed level of experimental sophistication. From a statistical viewpoint, a randomized trial of a sufficiently large population allows researchers to ascribe causality. This is important in the impact evaluation of a policy.

While RCTs may be new to economics, they are nothing new to the natural sciences which have for a long time emphasized the importance of randomization. However, trials in the natural sciences (especially medicine and theoretical physics) control two one other crucial biases – the placebo effect and observer expectation. This is accomplished through the “double-blind trial” in which neither the sample nor the researcher know whether they belong to the trial or the experiment.


This is incredibly easier in the natural sciences (after all, the whole treatment is contained in a pill, and easy to replicate). However, recent speculation, suggests that the placebo effect might be quite as present and important for economic issues. The economist Tyler Cowen even suggests that the placebo effect could be employed to overcome issues of macroeconomic importance such as consumer confidence.

Is there a chance (researchers please comment) that this effect applies somehow to microfinance trials as well? Placebo effect, after all, is a result of expectation – do these trials somehow form an expectation that is self-fulfilling as it is in medicine?

Double-blind trials achieve one other thing – remove observer-expectation bias. Modern physics lends a beautiful example of how elegant research design can overcome this bias. In neutrino experiments, researchers have a bias of the total number of neutrinos, N, that they expect to see. Therefore, even if many different researchers are shown the whole sample, the observation would systematically conform to the expectation. Rather, each researcher is only shown a theoretically meaningless N’, which is a random fraction of the total sample. Because this is a meaningless value, there exists no expectation and hence the overall value (an aggregate of each answer) is not biased.

I wonder if something similar can be done in microfinance. I am not familiar enough with the actual research methodology to comment, but would be interested to hear from researchers regarding the above two points. And, if theoretically it is possible to conduct a double-blinded RCT – is it practically possible, or would it take too much effort. Inevitably a double-blinded trial has a large financial burden, but in medicine I would never trust to be reliable anything that has not been rigorously tested in a double-blinded setting. Would you say the same about economics?

Comments

  1. It is not "trial or experiment" it is placebo group or treatment group.
    The Placebo effect is no simple matter, it subdivides in different effect e.g. Hawthorne effect (the effect that being observed changes the state of the matter - even true in natural sciences).

    You cannot do a double-blind trial in economics.
    How do you expect to do that? People know whether they get a subsidy or not, etc.

    There are also a bunch of biases you should know about:
    -attrition bias
    is normal in economics or medicine (particles in physics don't leave your experiment because they are fed up, people do!).
    You need to plan for this and make sure it does not happen more in one group than in the other.
    -skewed data
    Sometimes your outcome data is skewed and researchers like to report what is convenient for them. The mean and the median might differ quite a bit if the data is skewed.
    -viewpoint
    You should carefully think whom you consider!
    Microfinance might be beneficial for the farmer or the village receiving the credit but destroy economics of scale and therefore be bad for the whole society.

    Overall there is only so much what can be written in a comment box. You definitely should read about RCTs and their procs and cons, microfinance and economics and statistics in general.

    ReplyDelete
    Replies
    1. I wrote a few days back on this blog that researchers in their quest of knowledge or to prove something new, tend to over-theorize. It is up to them to clearly set out their objectives, whether their quest is for statistical robustness or to do a more "realistic" impact evaluation. Whether they want a journal publication and conference audience or they want to communicate to a wider non-academic audience as well. There is this piece of blog by a CMF Program Head (not me)which may be worth a read.

      http://www.indiadevelopmentblog.com/2011/07/impact-evaluations-in-real-world.html

      Delete

Post a Comment

Popular Posts

Vocationalisation of education in India: Current Scenario, Key Challenges and New directions

“Every handicraft has to be taught not merely mechanically as is done today, but scientifically. This is to say, the child should learn the why and wherefore of every process.” - Gandhi’s Philosophy of Education

The greatest challenge in Indian education system today is to provide skill based education to the youth. This is exacerbated by a mismatch in demand and supply for the skilled workforce. The penetration of vocational education and training remains poor not only in rural areas, but also in urban regions where there is a higher installed capacity to impart the same. This post is an attempt to make the readers understand the need of vocational education in India. Also, this is an attempt to summarise a few recommendations on the same. 
A recent survey (61st round) conducted by the NSSO found that:

1. The percentage of population that completed primary education was 70%, but less than 10% went on to complete a graduation course and above. Almost 97% of individuals in the age bracket…

Rockstar of Financial Inclusion: Business Correspondent Model of India

About Author:  Jatinder Handoo is a social business enthusiast and a branchless banking practitioner. Currently works at FINO PayTech Ltd and is based out of Mumbai. He is reachable at jatinder.handoo@fino.co.in
India is a hot bed of financial exclusion. A country which houses nearly 16% of the global population  has more than 65% of its people outside the formal financial system (Global Findex 2012). The Indian banking system has adopted multiple approaches to make universal financial inclusion a reality right from early days Indian post-independence banking system. Be it bank nationalization in 1969 or formation of Regional Rural Banks. Formation of NABARD or fostering microfinance through Bank-SHG linkage programme in early 90’s. A shimmering ray hope was rekindled with the growth of JLG based microfinance, however later studies made it clear that the model is credit led, concentrated predominately in the southern region of India thus could not be seen as painting complete financial…

A Platform for Knowledge - Enabling people to learn ..

I received a rather interesting link/website via my email today. The link read as MR University and all I could think of was, "Ok, this must be another website portal of some university or college". Well, on clicking the link and looking through the contents of the site, I was pleasantly surprised. The site http://www.mruniversity.com/ is an online education portal or platform that allows users or teachers to upload short videos on topics or lessons they wish to impart. First topic that I come across is Development Economics.
The intent of the website is eloquently put out by the two economists, Tyler Cowen and Alex Tabarrok in the intro video. What started as a blog focusing on economics and its various implications in understanding why things are the way they are around us, has now an interesting addition. A video portal titled MRUniversity or Marginal Revolution University that focuses on online education with subjects pertaining to economics. It brought back to my mind,…