Graphics & Design

An evidence-based approach to benchmarking the fairness of health-sector reform in developing countries

The Benchmarks of Fairness instrument is an evidence-based policy tool developed in generic form in 2000 for evaluating the effects of health-system reforms on equity, efficiency and accountability. By integrating measures of these effects on the
of 8
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
  534  Bulletin of the World Health Organization |  July 2005, 83 (7)   Abstract  The Benchmarks of Fairness instrument is an evidence-based policy tool developed in generic form in 2000 for evaluating the effects of health-system reforms on equity, efficiency and accountability. By integrating measures of these effects on the central goal of fairness, the approach fills a gap that has hampered reform efforts for more than two decades. Over the past three years, projects in developing countries on three continents have adapted the generic version of these benchmarks for use at both national and subnational levels. Interdisciplinary teams of managers, providers, academics and advocates agree on the relevant criteria for assessing components of fairness and, depending on which aspects of reform they wish to evaluate, select appropriate indicators that rely on accessible information; they also agree on scoring rules for evaluating the diverse changes in the indicators.In contrast to a comprehensive index that aggregates all measured changes into a single evaluation or rank, the pattern of changes revealed by the benchmarks is used to inform policy deliberation about which aspects of the reforms have been successfully implemented, and it also allows for improvements to be made in the reforms. This approach permits useful evidence about reform to be gathered in settings where existing information is underused and where there is a weak information infrastructure.Brief descriptions of early results from Cameroon, Ecuador, Guatemala, Thailand and Zambia demonstrate that the method can produce results that are useful for policy and reveal the variety of purposes to which the approach can be put. Collaboration across sites can yield a catalogue of indicators that will facilitate further work. Keywords  Health care reform; Benchmarking; Social justice; Social responsibility; Health services accessibility; Evidence-based medicine; Program evaluation/methods; Developing countries; Cameroon; Ecuador; Guatemala; Thailand; Zambia ( source: MeSH, NLM  ). Mots clés  Réforme domaine santé; Banc mesure performance; Justice sociale; Responsabilité sociale; Accessibilité service santé; Médecine factuelle; Evaluation programme/méthodes; Pays en développement; Cameroun; Equateur; Guatemala; Thaïlande; Zambie ( source: MeSH, INSERM  ). Palabras clave  Reforma en atención de la salud; Benchmarking; Justicia social; Responsabilidad social; Accesibilidad a los servicios de salud Medicina basada en evidencia; Evaluación de programas/métodos; Países en desarrollo; Camerún; Ecuador; Guatemala; Tailandia; Zambia ( fuente: DeCS, BIREME  ). Bulletin of the World Health Organization 2005;83:534-540. Voir page 539 le résumé en français. En la página 539 figura un resumen en español. 1  Professor of Ethics and Population Health, Harvard School of Public Health, Department of Population and International Health, Building 1, Room 1104C, 665 Huntington Avenue, Boston, MA 02115, USA (email: Correspondence should be sent to this author. 2  Research Associate, Universidad de San Carlos de Guatemala, Guatemala City, Guatemala. 3  Professor of Community Medicine, Centre for Health Equity Monitoring, Faculty of Medicine, Naresuan University, Phitsanulok, Thailand. 4  Dean, Faculty of Medicine, University of Yaounde, Yaounde, Cameroon. 5  Professor Emeritus, Aga Khan University, Karachi, Pakistan. 6  Director, Centre for Health, Science, and Social Research, Lusaka, Zambia. 7  Associate Professor, Kunming Medical College, Kunming, Yunnan, China.Ref. No. 04-016428 ( Submitted: 15 July 2004 – Final revised version received: 4 January 2005 – Accepted: 31 January 2005  ) An evidence-based approach to benchmarking the fairness of health-sector reform in developing countries Norman Daniels, 1  Walter Flores, 2  Supasit Pannarunothai, 3  Peter N. Ndumbe, 4  John H. Bryant, 5  T. J. Ngulube, 6  & Yuankun Wang 7 accountability. The Benchmarks address the complaint that “it is unfair” when the system treats some patients differently from others with similar needs, when some needs are not met because of administrative inefficiency, or when people have no say in how the system treats them. Fairness involves various claims about what people are owed as a matter of justice ( 1–3 ). Policy and Practice Introduction The Benchmarks of Fairness instrument is a method for evaluat-ing the fairness of health-sector reforms. The concept of fairness in health systems is broad, integrating the goals of equity in access and financing, clinical and administrative efficiency, and  535 Bulletin of the World Health Organization |  July 2005, 83 (7)   Policy and Practice Norman Daniels et al. Benchmarking the fairness of health-sector reform   Box 1. The nine main Benchmarks of Fairness and their corresponding key objective of fairnessBenchmark   Objective of fairness B1 Intersectoral public health EquityB2 Financial barriers to equitable access B3 Non-financial barriers to access B4 Comprehensiveness of benefits and tiering B5 Equitable financing B6 Efficacy, efficiency and quality improvement EfficiencyB7 Administrative efficiency B8 Democratic accountability and empowerment AccountabilityB9 Patient and provider autonomy be various fair ways to weight and trade off changes within and across benchmarks, we propose that the different benchmarks and their component criteria should not be weighted in terms of their relative importance. The Benchmarks instead aim to reveal the complex pattern of the effects of reforms on different aspects of fairness. Understanding this pattern is more informa-tive for local decision-makers than compressing information into an index. Changes in the various dimensions of fairness measured by the Benchmarks can still be evaluated. Methods: developing evidence to guide reform The generic benchmarks and criteria must be adapted to serve a specific purpose by an interdisciplinary team. The team refines the generic criteria, specifies indicators appropriate to local con-ditions, and seeks agreement on how to evaluate changes in these indicators. Planners or community groups can then evaluate health policies in light of the evidence they have agreed is relevant. Under ideal conditions, policy-makers assessing reforms would benefit from systematically reviewed evidence (from natural or controlled social experiments, were they available) (  22  ,  23 ). Developing countries implementing reforms need good evi-dence, based on local information, about the varied effects of actual reforms (  24  ), and this is what the Benchmarks provide. An ideal interdisciplinary team consists of policy-makers, academics, health-systems personnel, clinicians and civil society groups. The breadth of the Benchmarks compels people who have different training and work at various levels in the system to cross disciplinary boundaries and reconcile their perspec-tives. The adaptation process encourages stakeholders to take ownership of the results.The team must consider the purpose of the application: for example, whether it is evaluating comprehensive reforms (e.g., Mexico (  25  )), measuring district variation in the imple-mentation of reforms (e.g., Cameroon (P. Ndumbe, unpublished data, 2005) and Thailand (S. Pannarunothai et al., unpublished data, 2005)), measuring the impact of decentralization and financing reforms on delivery of public health services (Ecuador and Guatemala), evaluating the equity effects of rural insurance programmes (Yunnan, China), or the effects on the health sec-tor of scaling-up AIDS treatment and prevention programmes (Zambia). The purpose determines how the team will modify the generic benchmark criteria.The Benchmarks ask by how much reforms improve or  worsen aspects of fairness within the health sector nationally or subnationally. They combine an ethical framework with familiar methods from operations research. Using appropriate indicators, changes are measured and evaluated relative to a baseline (the status quo at the time reforms are introduced). This evidence and evaluation enhances deliberations about improving reforms. The Benchmarks serve a different purpose than indices that aggregate all these changes into a single num-ber for ranking health systems comparatively.The Benchmarks aim to fill two crucial gaps. First, inter-nationally supported reforms over two decades have failed to integrate the key goals of fairness. During the 1980s and early 1990s, international agencies such as the World Bank and In-ternational Monetary Fund pushed reforms that involved priva-tization, user fees, and decentralization, thus sacrificing equity ( 4–7  ) and other goals of fairness to the market-oriented pursuit of efficiency. Recently, the World Bank’s attention has focused on health inequalities ( 8  , 9  ) and better governance ( 10  , 11 ) but has still failed to integrate these goals. Second is the lack of capacity in developing countries to undertake evidence-based policy analysis which is the result of weak information infra-structures and a lack of tools to maximize the use of existing information. Development and rationale Teams from Colombia, Mexico, Pakistan and Thailand, using their own recent reforms as case studies, adapted a matrix for assessing the fairness of proposed American health insurance reforms ( 1 , 12  , 13 ) into a generic developing-country framework ( 14  ). Despite different cultural and social histories and levels of development in the collaborating sites, teams were able to agree on a generic matrix that included nine main Benchmarks (Box 1). Each Benchmark specifies a key objective of fairness through criteria that capture important elements and means of achieving these objectives (Box 2).The nine generic Benchmarks integrate the goals of fair-ness as follows: Benchmarks 1–5 address equity, 6 and 7 consider efficiency, and 8 and 9 concern accountability (Box 1). Improv-ing clinical and administrative efficiency can make a system fairer by allowing it to meet otherwise unmet needs. Though not all conflicts between efficiency and equity disappear, a fair process (included in Benchmark 8) can resolve disputes about them ( 15  ). Accountability is valued both intrinsically, as a mat-ter of fairness in governance, and instrumentally, since it helps achieve efficiency or equity. Contrast with WHO framework for health-systems performance The Benchmarks serve a different purpose from WHO’s index of health systems performance ( 16  , 17  ). The WHO index com-bines measures of health outcomes, system responsiveness, and the distribution of financial contributions into an overall index that ranks countries’ performance comparatively. The dimen-sions measured are assigned weights by surveying appropriate experts. A commission evaluating the index proposed various methodological improvements ( 18–21 ).In contrast, the Benchmarks make no cross-country com-parisons, avoiding some methodological issues about weighting and aggregation that face the WHO index. Because there may  536  Bulletin of the World Health Organization |  July 2005, 83 (7)   Policy and Practice Benchmarking the fairness of health-sector reform Norman Daniels et al.   Box 2. Examples of main criteria and their corresponding benchmarksBenchmark   Selected main criteria B1 Intersectoral public health 1. Degree to which reform increases percentage of population (demographically differentiated) benefiting from basic nutrition, clean water, literacy 2. Development of information infrastructure for monitoring inequalities in health status 3. Reform engages in intersectoral effortsB3 Non-financial barriers to access 1. Reduction in geographical maldistribution of services, supplies, etc. 2. Gender issues affecting access: degree of reproductive autonomy, access to resources, authority in family regarding decisions, sensitivity of services 3. Cultural barriers: language, attitudes towards disease 4. Discrimination (race, religion, class, sexual orientation, disease stigma)B8 Democratic accountability and empowerment 1. Explicit, public procedures for evaluating services 2. Explicit, deliberative procedures for resource allocation with transparency and stakeholder involvement in agreement on rationales 3. Fair appeals procedures 4. Adequate protections for privacy and patients’ rights 5. Measures to strengthen civil society In our case, the Guatemalan team ignored some bench-marks, combined features of others, and developed criteria that concentrated on the delivery of public health services. Familiarity with the specifics of local reform mechanisms helps in selecting criteria. Developing criteria for improvements in efficiency or accountability requires knowing how commu-nity structures are involved in health-unit management (as in Cameroon or Zambia) (Box 3).The team must pay attention to the sources and quality of information used for different indicators. There often is good data for traditional indicators bearing on some measures of health outcomes and some kinds of utilization, but indica-tors bearing on accountability, intersectoral cooperation, and quality measures, require non-standard sources of information or the use of qualitative techniques.Teams must define an approach to evaluating changes from a measured baseline. Most of our sites have not completely addressed this evaluative task, concentrating instead on develop-ing an instrument capable of measuring a baseline. Most teams plan to develop scoring rules that assign a qualitative value to a degree of measured change. This evaluation might be explicitly qualitative (a scale of poor, fair, good, excellent or colour-coded stars); alternatively, a team may use ordinal numbers on a scale (e.g., from –5 to 5), provided it avoids using these ordinals for arithmetic operations like averaging or aggregating.Evaluation requires deliberation. To illustrate: some team members might think that if 50% of the population lacked ac-cess to clean water, then each additional 10% of the population that gained access should count for an additional point on an ordinal scale ranging from 0 to 5. Others might want to give  weight to the difficulty of reaching remote populations and propose a different scoring rule. Whatever rule the team agrees on operationalizes its view of fairness on this dimension. Some other team or group may disagree. There is no effort to capture a general public or expert consensus since the goal is not a uni-versal ranking of fairness. Nevertheless, the result is this kind of objectivity: disagreements focus on the evidence about change and the reasons for considering some amount of change as be-ing more or less fair. This approach provides a proper basis for evidence-based evaluation and promotes better deliberation.In Thailand (see below) the problem of evaluation was ap-proached differently. Instead of developing scoring rules, diverse focus groups were informed about changes in selected indica-tors and judgements about the fairness of these changes were elicited. This approach engages a wider group in evaluation than the use of scoring rules, but it is less clear how evaluations are connected to specific changes. As country-level work proceeds,  we shall have to grapple with the strengths and weaknesses of these different approaches to evaluation. Applying the locally-adapted benchmarks In some middle-income developing countries, the application of the Benchmarks made robust use of locally available informa-tion of reasonable quality that had not been used previously to inform ongoing reform efforts (Ecuador, Guatemala, Mexico, Thailand). In low-income developing countries, they provide useful evidence even if crude proxies have to be substituted for better quality indicators that are not locally available (for exam-ple, in Cameroon, Yunnan Province in China, and Zambia). Guatemala  A team of academics, members of the ministry of health, and a nongovernmental organization (NGO) (CARE) adapted the Benchmarks to assess reforms aimed at decentralizing, financ-ing and improving access to public health services. To measure equity, the team developed three indices that draw on good quality data available at the district level. An Index of the Pri-ority of Health-Care Services combines the coverage for three basic services (immunization, antenatal care and supervised deliveries). The larger the gap between actual and ideal cover-age — that is, the greater the need for resources — the higher the coefficient (between 0.01 and 0.99). The Index of Resource Distribution (IRD) is a weighted measure of health expenditure for primary care, health personnel and health facilities, using districts with the most resources for comparison with other districts. The higher the score (between 0.01 and 0.99) on the IRD, the more resources per capita are available in a district.  An index of human resources measures the availability of com-munity volunteers at the district level.  537 Bulletin of the World Health Organization |  July 2005, 83 (7)   Policy and Practice Norman Daniels et al. Benchmarking the fairness of health-sector reform   Box 3. Examples of locally-developed indicators for adapted benchmark criteriaBenchmark and criterion a   Country that developed indicator and definition of indicator  B1 1. Degree to which reform increases percentage of the Cameroon: percentage of health districts monitoring iodized salt in markets population (demographically differentiated) benefiting from basic nutrition, clean water, literacy, etc.B3 1. Reduction in geographical maldistribution of services Cameroon: percentage of health units with more than 10% of population living more than 1 hour’s walk from health unitB3 1. Reduction in geographical maldistribution of services Guatemala: district ratio of index of priority for immunization coverage (IPHS) to index of resource distribution (IRD) b B3 3. Language barriers Guatemala: percentage of health personnel by category who speak an indigenous languageB8 5. Measures to strengthen civic society Cameroon: percentage of health units with budget approved through community dialogue a  See reference 14   for a full listing of the Benchmarks and their criteria. b  See discussion of Guatemala in text for further details. Thailand  A team of Thai researchers applied an adapted version of the Benchmarks to examine how different groups judged the fair-ness of reforms (S. Pannarunothai, unpublished data, 2005). They presented the results from 81 health-outcome and process indicators that measured the geographical variations in the ef-fects of reform to eight focus groups in each of 10 provinces. The groups, differing by urban and rural location and further divided into groups concerned with health care and its delivery (managers, providers, advocates) and comparable groups not primarily concerned with health, were asked to evaluate the fairness of the changes. Non-health groups tended to give lower estimates of fairness than health groups. The approach revealed the instrument had a reasonable sensitivity to different levels of performance in the system, moving the ministry of health to plan to develop this approach for further use in evaluating health-sector reforms under decentralization. Mexico  A team in Mexico used the Benchmarks to evaluate retro-spectively the fairness of changes from 1995, when reforms  were introduced, until 2000. They had to restrict their study to indicators available at both times (  25  ). As in other middle-income countries, the benchmarking approach used good quality, underutilized data in a novel way. A newer Mexican adaptation allows for ongoing monitoring and evaluation of cervical cancer screening programmes (  26  ). Cameroon In 2002, a workshop of district medical officers, ministry of health officials and academics adapted the Benchmarks to assess recent national reforms, assigning the task of refining indicators to a team of district medical officers. The adaptation focused on seven benchmarks and 70 criteria, for which 73 indicators  were selected (see examples in Box 3). The restrictions were that indicators had to involve information that could be collected by medical students during their sixth-year district rotations, could not require special surveys, and could not put the students at risk. After training the students, two cycles of data collec-tion were undertaken involving 70 medical students in 20 of 155 districts.These field tests revealed errors in some indicators. Stu-dents could not, for example, collect information about testing Equity requires that districts with greater unmet need receive more resources. Fig. 1 (web version only, available at: shows the mismatch in coeffi-cients for districts. The ministry hoped to rely on community health workers to compensate for the lack of resources in higher priority districts, but the results showed that high priority dis-tricts also had worse scores on the index of human resources. Other indicators showed variations among districts in the de-gree to which patients had access to personnel who spoke their native language (Box 3), revealing inequities in a non-financial barrier to services. Ministry participants, impressed with these results, plan to use a benchmarking approach to monitor the implementation of recent reforms. Ecuador Learning from the experience in Guatemala, the Ecuador team built a coalition that included representatives from the provin-cial level of the ministry of health, local government, NGOs and civil society organizations, in order to better sustain pres-sure on the ministry and local government to pursue fairness in reforms. Drawing on and adding to the Guatemalan indicators, the team concentrated on evaluating the ministry of health’s policy of providing free care to mothers and children younger than 5 years of age by reimbursing public facilities for their care. The Benchmark on financial barriers to care showed that health facilities in 4 of 6 districts surveyed charged patients US$ 1–4.00 for treating an acute respiratory infection. Acute respiratory infections were used as an indicator for services that  were supposed to be free for the targeted women and children. These findings were presented to health authorities. Some of the civil society groups requested an investigation into the charges and that immediate action be taken to stop them.To evaluate improvements in the quality of services for target groups, the team used the ratio of administrative staff to medical staff in health facilities, which is admittedly a crude indicator of efficiency and quality. On average, public facilities have 7 administrative staff for every 10 medical staff whereas facilities run by NGOs have a ratio of 3 to 10. Public hospitals varied considerably, with ratios ranging from 1 to 5 to 1 to 2. The coalition secured agreement from provincial authorities to study public staffing issues and modify policy where necessary.  538  Bulletin of the World Health Organization |  July 2005, 83 (7)   Policy and Practice Benchmarking the fairness of health-sector reform Norman Daniels et al. Adapting to local purposes and local conditions One important lesson is that interdisciplinary teams can agree on what changes make a system more or less fair in specific  ways and adapt the generic Benchmarks to meet specific pur-poses and local conditions. Because such agreement involves deliberation about values and trade-offs among them, it is a key step towards using evidence to evaluate policy. Whereas the generic Benchmarks show that considerable agreement can be reached across cultural differences, the country-specific projects allow different sites to emphasize aspects of fairness in a way that reflects local deliberation.Teams are resourceful in respecting constraints on local information yet using it fruitfully. Mexico’s retrospective evalu-ation required restricting the analysis to data available at the inception of reforms. Cameroon selected indicators that could be collected at the district level by medical students. Zambia also restricted indicators to those that could be collected locally. Guatemala and Ecuador, middle-income countries with better information sources, developed more complex indices of avail-able information that had not been used previously to evaluate reforms. One obstacle to the process is ensuring that teams know what information can be accurately collected locally. Cameroon’s field tests revealed that some indicators had to be revised. Zambia proposed exit surveys that were too difficult.  A strength of the adaptation process is that it builds capacity to understand the process of monitoring and evaluating reform: community members in Zambia and civic society groups in Ecuador mastered the task of deciding what should be improved and determining how to tell if it had improved. Process and use of results  Another central lesson of these applications is that stakeholders — the ministry of health, providers and civic society groups — take ownership of the adaptation primarily through their involvement in the process. When this is achieved, some of the political tensions that arise during evaluation can be overcome. For example, although the ministry of health staff involved in the Guatemalan project were initially resistant to making certain information available, when they realized that it would enable them to improve the delivery of services, they produced the information, participated in the analysis, and later took the results seriously. The team in Ecuador, aware of the Guatema-lan experience, included representatives of civic society groups from the start in order to sustain pressure to use the results.The adaptation process requires a significant investment of labour by a diverse team; it can take months, even a year, before it yields results. This factor is further complicated by the difficulty of securing funding for early-stage projects as well as for sustained monitoring and evaluation.One compensating strength of the process is that it devel-ops capacity to analyse and deliberate about improving reform efforts. It is also possible to learn across sites, as Ecuador did, improving on the process and using indices already developed in Guatemala. The work in Zambia was enhanced by having district medical officers from the Cameroon project help con-duct the workshops. Sharing experiences from Zambia in the Malawi regional meeting advanced the work in several sites. Evaluation  Although these sites are at a relatively early stage of using the evidence acquired from benchmarking for evaluation, most teams plan to develop scoring rules. Thailand is exploring the for the presence of iodized salt in local markets (Box 3) because it is available only at the provincial level. Field tests neverthe-less showed that students, assisted by properly trained district medical officers could gather reliable evidence with the bench-marking approach, although appropriate surveys were also needed. The full dataset is being analysed to provide a baseline against which national reforms can be monitored. The use of students is not only low cost, but has educational value, since it produces a large number of doctors trained to assess the fair-ness of reforms. (A project in Pakistan also incorporates the Benchmarks into medical school curriculum.) Zambia In June 2003, a workshop involving some members of parlia-ment, representatives from the ministry of health and com-munity members from four districts, proposed adapting the Benchmarks for use in monitoring the equity of new antiretro-viral treatment and prevention programmes for HIV/AIDS.In March 2004, a team of community representatives and health providers from four districts adapted the benchmarks to that purpose by constructing a table with four key questions about the effects of scaling-up treatments: what do you want to see get better? How would you judge or tell it was getting better? What information would you need to make that judge-ment? Where would you get that information? For example, the team wanted a fairer process to be used in selecting treat-ment sites and patients. They would be able to tell whether the process was fair if appropriate stakeholders were involved, and this information could be obtained from minutes of meetings at different levels. A regional meeting in Malawi, drawing on the results of the Zambian workshop as well as on other proposals about monitoring the introduction of antiretroviral treatments in the southern African region, proposed additional indicators that  will be incorporated into the Zambian work. These indicators included the percentage of treatments delivered at different facility levels within the health system (to measure integra-tion in the system) and the percentage of treatments by gender compared with prevalence (to measure gender equity). Shar-ing benchmarking ideas across the region will accelerate the adaptation process. Yunnan Province, China  A team including academics from Kunming Medical School, various medical professionals, and health authorities used a benchmarking approach to measure the effects of a new rural insurance programme in selected districts and municipalities in Yunnan Province. They found significant financial barriers to enrolment of the poorest farmers, who could not afford a contribution of 10 yuan per year and who had limited access to subsidies that were supposed to reach the very poor. They found lower enrolments among better-off farmers, who thought the care being provided would not meet their needs. They found overcharging for drugs in some districts. This initially promis-ing effort encountered an obstacle we have faced in a number of settings, namely, inadequate funding to conduct monitoring and evaluation on a sustained basis. Discussion: lessons learnt Several important lessons can be drawn from these illustra-tions of the benchmarking approach despite the fact that it is a work in progress.
Similar documents
View more...
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!