Skip to main content

The Art of Process Evaluation

What is process evaluation?
The purpose of Process Evaluation is to assess whether the implementation of the process in a project, is in accordance with the overall project objectives and project plan. PE analysis assesses the strength and structure of; (1) operational processes – service delivery, methodology used and resource usage; and (2) program component processes – service usage and participant experiences. PE therefore seeks to evaluate overall project achievements, and evaluate why and how changes occurred in the population being studied. Criteria such as feasibility, efficiency, effectiveness, relevance and sustainability are used to evaluate the processes. PE is usually conducted primarily using qualitative methods.
Findings from a PE analysis can help program managers better understand and evaluate issues arising during implementation of a process – operational or programmatic. In addition, PE can provide measurable statistics about what the challenges experienced during project implementation were.

How is PE conducted?
A feasible way to approach the PE analysis is by first identifying all critical project processes and sub-processes. Often due to resource and budgetary constraints only critical processes are selected for the PE analysis.  Once the processes are identified, PE is conducted for each of these processes to either determine whether the process was feasible, efficient, effective, relevant or sustainable. Other relevant criteria can also be set up to conduct PE of critical processes.

What is measured?
It is critical to use measurable indicators when evaluating any process. Following must be kept in mind when deciding what changes the PE is trying to identity and how:

Objective
 Determine what the team seeks to learn from the PE analysis
Audience
 Determine who will benefit or who will be the audience of the PE  
 analysis
What to measure
 Identify what changes you are trying to measure and what the 
 unit of measurement will be
Target population
 Identify the target population in which the change is being 
 measured
Indicators
 Use measurable indicators to determine the change. Use 
 qualitative and qualitative indicators for measuring the change. 
 When selecting indicators to measure change, the following 
 features about indicators must be kept in mind:
 ·        Targeted – Indicators must measure what is changing   (elements of change), who is involved (target group), where the change took place (place), and when the change will/has happened (timeframe).
 ·        Measureable – Indicator must measure units that have changed (units of measurement), capture the measurement at beginning (baseline measurement), and capture the quality of the change (whether change is effective, appropriate etc).
 ·        Reliable – Indicator must be credible, assumptions about its usage should minimal, and connections between indicator and what is being proven should be clear
 ·        Feasible – Using the indicator must be feasible and viable. Measurement must be doable so that information can be obtained


An Example
IFMR-Research recently conducted Process Evaluation for the EKO[1] Financial Literacy Project in Bihar. The project was funded by the Russian Trust Fund, as a part of the World Bank funded Financial Literacy projects across the world. The objective of the project was to measure the impact of Financial Literacy on savings behavior of EKO clients. EKO clients receive a 1-hour Financial Literacy Training about the importance of saving in formal financial institutions, developing savings goals, and conducting financial management. PE was performed to determine the effectiveness, relevance, and feasibility of the training sessions. Critical processes that encompassed the Financial Literacy training process were identified.  

The following Process Map of the overall training process indicates the sub-process that comprising the training process:


  1.  Trainers Training Process - The CMF staff provided trainers with a 10 day training (also called TOT – Training of the Trainer). The TOT process was evaluated by conducting anonymous phone interview of trainers. Questions about trainer experience during CMF training sessions, relevance of the training materials, effectiveness of training materials used by CMF, feasibility of training locations and timings were asked.
  2.  Client Training Process – Once the Trainers acquired knowledge about financial literacy through CMF hosted TOT sessions, and had sufficient experience conducting mock trainings sessions, trainers were sent to the field to conduct client trainings. Clusters of 10 clients were formed and the trainers provided 1-hour interactive Financial Literacy training to these clusters. This process was evaluated by conducting interviews with 10% of clients randomly selected from each cluster. The clients were asked about their experience during the training, feasibility of training locations and timings, relevance of training sessions in their day-to-day lives, and suggestions for improvement of the overall training delivery mechanism.
  3. Client Knowledge Acquisition Process – Clients were provided comprehensive financial literacy during the sessions; topics such as importance of savings with formal institutions, developing savings goals based on their long-term and short-term life needs, understanding the range of savings products available for disparate savings needs, general expenses and income management. Clients’ knowledge about these various topics was evaluated by conducting interviews with 10% of the clients, randomly selected from each cluster. To determine if trainings had a longer-term effect, clients’ knowledge was reassessed through a subsequent survey, conducted approximately 6-8 weeks after the FE trainings.

When conducting PE, it is important to determine what process is being evaluated and why. Each process implemented during a project has sub-process that can be deemed as critical or non-critical based on the criteria used by evaluators. To ensure PE can be conducted within a reasonable time and within budget, it is essential that critical sub-processes be evaluated first; rather than evaluating the entire sample study participants, it may be effective to evaluate a sub-set of the sample. PE may be conducted during various phases of the project –mid-term, upon completion and after completion to ensure the true issues surrounding the implementation of critical processes is well understood. To  ensure PE is unbiased, it may also be a good idea to have an independent 3rd party conduct the PE.

Recommended Reading:
Cheyanne Church and Mark M. Rogers, DESIGNING FOR RESULTS: Integrating Monitoring and Evaluation in Conflict Transformation Programs, Search for Common Ground, 2005



[1] EKO provide mobile banking services to its clients via Customer Service Points (CSPs). CSPs are typically local kirana or convenience stores that provide banking services such as savings and withdrawal services through EKO banking channels in rural and urban areas. 

Comments

Popular Posts

Vocationalisation of education in India: Current Scenario, Key Challenges and New directions

“Every handicraft has to be taught not merely mechanically as is done today, but scientifically. This is to say, the child should learn the why and wherefore of every process.” - Gandhi’s Philosophy of Education

The greatest challenge in Indian education system today is to provide skill based education to the youth. This is exacerbated by a mismatch in demand and supply for the skilled workforce. The penetration of vocational education and training remains poor not only in rural areas, but also in urban regions where there is a higher installed capacity to impart the same. This post is an attempt to make the readers understand the need of vocational education in India. Also, this is an attempt to summarise a few recommendations on the same. 
A recent survey (61st round) conducted by the NSSO found that:

1. The percentage of population that completed primary education was 70%, but less than 10% went on to complete a graduation course and above. Almost 97% of individuals in the age bracket…

Rockstar of Financial Inclusion: Business Correspondent Model of India

About Author:  Jatinder Handoo is a social business enthusiast and a branchless banking practitioner. Currently works at FINO PayTech Ltd and is based out of Mumbai. He is reachable at jatinder.handoo@fino.co.in
India is a hot bed of financial exclusion. A country which houses nearly 16% of the global population  has more than 65% of its people outside the formal financial system (Global Findex 2012). The Indian banking system has adopted multiple approaches to make universal financial inclusion a reality right from early days Indian post-independence banking system. Be it bank nationalization in 1969 or formation of Regional Rural Banks. Formation of NABARD or fostering microfinance through Bank-SHG linkage programme in early 90’s. A shimmering ray hope was rekindled with the growth of JLG based microfinance, however later studies made it clear that the model is credit led, concentrated predominately in the southern region of India thus could not be seen as painting complete financial…

A Platform for Knowledge - Enabling people to learn ..

I received a rather interesting link/website via my email today. The link read as MR University and all I could think of was, "Ok, this must be another website portal of some university or college". Well, on clicking the link and looking through the contents of the site, I was pleasantly surprised. The site http://www.mruniversity.com/ is an online education portal or platform that allows users or teachers to upload short videos on topics or lessons they wish to impart. First topic that I come across is Development Economics.
The intent of the website is eloquently put out by the two economists, Tyler Cowen and Alex Tabarrok in the intro video. What started as a blog focusing on economics and its various implications in understanding why things are the way they are around us, has now an interesting addition. A video portal titled MRUniversity or Marginal Revolution University that focuses on online education with subjects pertaining to economics. It brought back to my mind,…