The Art of Process Evaluation
http://www.developmentoutlook.org/2013/08/the-art-of-process-evaluation.html
What is process evaluation?
The purpose of Process Evaluation is to assess whether the
implementation of the process in a project, is in accordance with the overall
project objectives and project plan. PE analysis assesses the strength and
structure of; (1) operational processes – service delivery, methodology used
and resource usage; and (2) program component processes – service usage and
participant experiences. PE therefore seeks to evaluate overall project
achievements, and evaluate why and how changes occurred in the population being
studied. Criteria such as feasibility, efficiency, effectiveness, relevance and
sustainability are used to evaluate the processes. PE is usually conducted
primarily using qualitative methods.
Findings from a PE analysis can help program managers better
understand and evaluate issues arising during implementation of a process –
operational or programmatic. In addition, PE can provide measurable statistics
about what the challenges experienced during project implementation were.
How is PE conducted?
A feasible way to approach the PE analysis is by first
identifying all critical project processes and sub-processes. Often due to
resource and budgetary constraints only critical processes are selected for the
PE analysis. Once the processes are
identified, PE is conducted for each of these processes to either determine
whether the process was feasible, efficient, effective, relevant or
sustainable. Other relevant criteria can also be set up to conduct PE
of critical processes.
What is measured?
It is critical to use measurable indicators when evaluating
any process. Following must be kept in mind when deciding what changes the PE
is trying to identity and how:
Objective
|
Determine what the team seeks to learn from the PE analysis
|
Audience
|
Determine
who will benefit or who will be the audience of the PE
analysis |
What to
measure
|
Identify
what changes you are trying to measure and what the
unit of measurement will be |
Target
population
|
Identify the
target population in which the change is being
measured |
Indicators
|
Use
measurable indicators to determine the change. Use
qualitative and qualitative indicators for measuring the change. When selecting indicators to measure change, the following features about indicators must be kept in mind:
· Targeted – Indicators
must measure what is changing (elements of change), who is involved (target group), where the change took
place (place), and when the change will/has happened (timeframe).
· Measureable – Indicator must measure units that have
changed (units of measurement), capture the measurement at beginning
(baseline measurement), and capture the quality of the change (whether change
is effective, appropriate etc).
· Reliable – Indicator
must be credible, assumptions about its usage should minimal, and connections
between indicator and what is being proven should be clear
· Feasible – Using the
indicator must be feasible and viable. Measurement must be doable so that
information can be obtained
|
An Example
IFMR-Research recently conducted Process Evaluation for the
EKO[1]
Financial Literacy Project in Bihar. The project was funded by the Russian
Trust Fund, as a part of the World Bank funded Financial Literacy projects
across the world. The objective of the project was to measure the impact of
Financial Literacy on savings behavior of EKO clients. EKO clients receive a 1-hour
Financial Literacy Training about the importance of
saving in formal financial institutions, developing savings goals, and
conducting financial management. PE was performed to determine the
effectiveness, relevance, and feasibility of the training sessions. Critical
processes that encompassed the Financial Literacy training process were
identified.
The following Process Map of the overall training process
indicates the sub-process that comprising the training process:
- Trainers Training Process - The CMF staff provided trainers with a 10 day training (also called TOT – Training of the Trainer). The TOT process was evaluated by conducting anonymous phone interview of trainers. Questions about trainer experience during CMF training sessions, relevance of the training materials, effectiveness of training materials used by CMF, feasibility of training locations and timings were asked.
- Client Training Process – Once the Trainers acquired knowledge about financial literacy through CMF hosted TOT sessions, and had sufficient experience conducting mock trainings sessions, trainers were sent to the field to conduct client trainings. Clusters of 10 clients were formed and the trainers provided 1-hour interactive Financial Literacy training to these clusters. This process was evaluated by conducting interviews with 10% of clients randomly selected from each cluster. The clients were asked about their experience during the training, feasibility of training locations and timings, relevance of training sessions in their day-to-day lives, and suggestions for improvement of the overall training delivery mechanism.
- Client Knowledge Acquisition Process – Clients were provided comprehensive financial literacy during the sessions; topics such as importance of savings with formal institutions, developing savings goals based on their long-term and short-term life needs, understanding the range of savings products available for disparate savings needs, general expenses and income management. Clients’ knowledge about these various topics was evaluated by conducting interviews with 10% of the clients, randomly selected from each cluster. To determine if trainings had a longer-term effect, clients’ knowledge was reassessed through a subsequent survey, conducted approximately 6-8 weeks after the FE trainings.
When conducting PE, it is important to determine what
process is being evaluated and why. Each process implemented during a project has
sub-process that can be deemed as critical or non-critical based on the
criteria used by evaluators. To ensure PE can be conducted within a reasonable
time and within budget, it is essential that critical sub-processes be
evaluated first; rather than evaluating the entire sample study participants,
it may be effective to evaluate a sub-set of the sample. PE may be conducted
during various phases of the project –mid-term, upon completion and after
completion to ensure the true issues surrounding the implementation of critical
processes is well understood. To ensure PE is unbiased, it may also be a good
idea to have an independent 3rd party conduct the PE.
Recommended Reading:
Cheyanne Church and Mark M. Rogers, DESIGNING
FOR RESULTS: Integrating Monitoring and Evaluation in Conflict Transformation
Programs, Search for Common Ground, 2005
[1]
EKO provide mobile banking services to its clients via Customer Service Points
(CSPs). CSPs are typically local kirana or convenience stores that provide
banking services such as savings and withdrawal services through EKO banking
channels in rural and urban areas.