Interview: Quality and effectiveness

Structured monitoring and evaluation are a basic element of ongoing project management at Malteser International. Detailed analysis of data from previous and current projects helps us to measure their success, and ensure that they provide the greatest possible benefit to the people at whom they are aimed. Dr. Marie T. Benner is Malteser International’s Senior Health Adviser – responsible for maintaining the high standards of quality in Malteser International‘s work. In this interview, she discusses the challenges of measuring the quality and effectiveness of highly complex humanitarian interventions.

Why is it important for Malteser International to implement good projects?

Marie Benner: Our mission as an organization is to help people in need, and that is my own personal motivation for the work that I do as well. When we implement good projects that means our aid is reaching people, that we are helping to improve their situation, and that we are fulfilling our mission. On top of this, we have to make sure that our donors, sponsors and partners can rely on us to carry out good quality work, and be a responsible organization to work with.

What do you mean by “quality” in this context?

Marie Benner: We judge the quality of our work based on a range of indicators. There are a number of questions that are important for us. Is the project relevant? That is, does it really cover the needs of the people that we are trying to help? Are long-term developments and other external factors, for example, the political context, taken into account? Does our help also reach people in outlying regions? Is the project effective, and do the resources used achieve an appropriately successful result?

What are the criteria that you apply when you make these assessments?

Marie Benner: We determine whether the results indicate that the project is successful on three levels: output, outcome, and impact. The output refers to the sum total of the results achieved – for example, a number of training sessions for health workers, or a certain amount of medication delivered. At the outcome level, we look at the results of our work in relation to the goal of the project. That means we determine, for example, how many people have been given access to better medical care because of some particular training, or because we delivered a certain medication, or what is the proportional reduction of malaria cases that we can attribute to our work, for example.

Where it is meaningful, we also take into account the impact level – this means the overall effect of our measures in a wider, superregional context. There are often a range of factors that need attention in order to bring about an improvement. The help of our local partners, health authorities, and civil society is a great support in this. When we treat malaria in our projects, we contribute to the broader fight against the disease. These kinds of overarching effects go beyond the specific project goals, and this is something that we assess in the context of a specific evaluation, and not our regular monitoring.

The Malteser International project cycle — Monitoring and evaluation form an essential part of our project cycle.

What is the difference between evaluation and monitoring? How does Malteser International ensure the quality of its projects on a regular basis?

Marie Benner: We need well-sourced information about any given situation, and the services that we have provided, as the basis for our project planning and risk management, and in order to be able to make informed decisions as an organization. We ensure the quality of our projects by making monitoring and evaluation activities an integral part of our project cycle.

Although monitoring and evaluation are often mentioned in one breath, they each have a very specific meaning. We have had intensive discussions in order to produce a valid and purposeful definition of what they mean for Malteser International. We understand monitoring as the regular and systematic collection, analysis and interpretation of qualitative and quantitative data in order to measure the progress of a project. Evaluations, on the other hand, are objective and systematic examinations of ongoing or concluded projects that are undertaken according to a predefined set of criteria. While monitoring is a routine and ongoing process, evaluation takes place at certain points – usually at the midpoint and at the end of a project’s life cycle.

Monitoring and evaluation are not just about collecting reliable data. They also help us to establish crucial learning processes within our organization, which allow us to assimilate the information that we have gathered and use these insights to improve our future projects.

The first important step in evaluating a project is defining goals that can be judged against objective criteria. What kind of goals are used?

Marie Benner: You do indeed need measurable and realistic goals to be able to perform a meaningful evaluation. That means that in the first place we have to define clear structural units that we can use for measurement. We do this by establishing goals directly on the project level. For example, when we work to improve the health system in the DR Congo that means we concentrate on the seven health zones supported by Malteser International. Doing this allows us to evaluate whether the 1.3 million people that live there are able to see a sustainable improvement in their access to basic healthcare.

Secondly, the goal has to be defined as concretely as possible. To continue with the example of the Congo, this means that we evaluate improved access to basic healthcare by assessing whether, for example, the existing facilities are being used more often than before, whether medication and medical equipment is reliably available in local health stations, or whether the system for financing medical infrastructure has improved.
These are measurable indicators for us.

How do you determine if a project has been successful?

Marie Benner: By referring to these clearly defined goals, it is relatively straightforward to determine whether a project has been a success by asking whether the goals have been reached or not, and whether the processes were effective or not. We then apply our predefined quality criteria to the indicators established by the project goals. To make this process public and transparent, we aggregate this information into our so-called performance figures, which give an overview of our work, and we publish these on our website and in our annual report.

Learning processes and effective knowledge management are extremely important for our organization. This means that if a project produces information that we can use to improve our work locally, or in other projects, it is a success for us in the sense that it has helped to make us better and more efficient at what we do.

You travel a lot, and perform many evaluations yourself. What makes a project successful for you personally?

Marie Benner: I have seen quite a number of projects in my 24 years working for Malteser International. For me personally, whether or not a project is successful is not something limited to the objective criteria that I apply for an evaluation. Beyond these, a good contextual analysis, a good integration of local partners and community structures, and above all, of the people who we want to support are all highly important. Listening to these people, making a clear assessment, understanding problems and their causes, and then drawing the right conclusions are the basic ingredients of a successful project. I have very high standards, but of course I understand that we work in a very difficult field, and that not everything can run smoothly. It is our job to find solutions for people in the extremely complex environments of disasters, wars, and epidemics. Most of the time we succeed, but sometimes the situation is simply too complex. If we can use the resources placed at our disposal to help people in a significant way then I am satisfied.