A decade of investments in monitoring the HIV epidemic: how far have we come? A descriptive analysis

Abstract

Background

The 2001 Declaration of Commitment (DoC) adopted by the General Assembly Special Session on HIV/AIDS (UNGASS) included a call to monitor national responses to the HIV epidemic. Since the DoC, efforts and investments have been made globally to strengthen countries’ HIV monitoring and evaluation (M&E) capacity. This analysis aims to quantify HIV M&E investments, commitments, capacity, and performance during the last decade in order to assess the success and challenges of national and global HIV M&E systems.

Methods

M&E spending and performance was assessed using data from UNGASS country progress reports. The National Composite Policy Index (NCPI) was used to measure government commitment, government engagement, partner/civil society engagement, and data generation, as well as to generate a composite HIV M&E System Capacity Index (MESCI) score. Analyses were restricted to low and middle income countries (LMICs) who submitted NCPI reports in 2006, 2008, and 2010 (n =78).

Results

Government commitment to HIV M&E increased considerably between 2006 and 2008 but decreased between 2008 and 2010. The percentage of total AIDS spending allocated to HIV M&E increased from 1.1% to 1.4%, between 2007 and 2010, in high-burden LMICs. Partner/civil society engagement and data generation capacity improved between 2006 and 2010 in the high-burden countries. The HIV MESCI increased from 2006 to 2008 in high-burden countries (78% to 94%), as well as in other LMICs (70% to 77%), and remained relatively stable in 2010 (91% in high-burden countries, 79% in other LMICs). Among high-burden countries, M&E system performance increased from 52% in 2006 to 89% in 2010.

Conclusions

The last decade has seen increased commitments and spending on HIV M&E, as well as improved M&E capacity and more available data on the HIV epidemic in both high-burden and other LMICs. However, challenges remain in the global M&E of the AIDS epidemic as we approach the 2015 Millennium Development Goal targets.

Background

Since the 2001 United Nations General Assembly Special Session (UNGASS) Declaration of Commitment on HIV/AIDS [1], countries and the international community have channeled unprecedented political and financial support to the fight against AIDS. Investments increased eight-fold during the 10 years that followed the Declaration to over $16.8 billion in 2011 and $18.9 billion in 2012 [2]. There has been a parallel increase in focus on accountability to ensure that this funding is spent responsibly and to enable programmatic improvement, policy change, and a greater understanding of the evolving nature of the AIDS epidemic. This is dependent on high quality, representative data, which is in large part gathered from national monitoring and evaluation (M&E) efforts. These data, and the related data systems, are essential to enable countries to have greater ownership of their response by allowing them to plan for, collect, analyze, disseminate, and use HIV-related data to inform the national response [3]. Strengthening national M&E system capacity is therefore a critical component of combating the AIDS epidemic. Country-owned HIV M&E systems without duplication are supported by the “Three Ones” principles [4], which sought to create a nationally unified data system to support the information needs of a dramatically escalating HIV epidemic.

Creating, maintaining, and improving national health information systems requires sustained investment of political, human, and financial resources [5]. General guidance states that 5% to 10% of the national AIDS budget should be used for HIV M&E activities [6]. There has been substantial investment in HIV M&E systems in the last 10 years [7], as well as several efforts to define the basic components of a functional HIV M&E system [3, 8]. However, to our knowledge, there has not been a quantitative assessment of M&E investments and national HIV M&E system capacity on a global scale.

The goal of this paper is to quantify HIV M&E investments, commitments, capacity, and performance in the decade following the Declaration of Commitment to inform the current global discussion on expanding access and use of development data, and ensure increased support for accountability systems in the context of the post-2015 agenda.

Methods

Data source

Data on M&E spending, systems, and performance were derived from UNGASS country reports submitted in 2006 [9], 2008 [10], and 2010 [11]. Country Progress Reports, including a core set of indicators, have been submitted to the Joint United Nations Programme on HIV/AIDS (UNAIDS) every second year since 2004 as part of the UNGASS reporting system [11]. Information on government commitment, government engagement, partner/civil society engagement, data generation, and data dissemination/utilization at the national level was taken from the National Composite Policy Index (NCPI) questionnaire [11], which is an integral part of the UNGASS process and the most routinely collected data globally on M&E system capacity. The NCPI is a two-part questionnaire, the first section of which solicits government feedback on the HIV/AIDS strategic plan, political support, prevention, treatment, care and support, and M&E. The second section is administered to civil society organizations and bilateral and multilateral agencies, and assesses human rights, civil society involvement, prevention and treatment, and care and support. Country response rates over the 2006, 2008, and 2010 reporting cycles ranged from 50% to 90% (Table 1).

Table 1 UNGASS and NCPI reporting response rates, 2006–2010

Sample

These analyses were restricted to low- and middle-income countries (LMICs) who submitted NCPI questionnaires in each of the 2006, 2008, and 2010 reporting cycles (n =78) in order to assess trends over time. The 2004 NCPI reporting was excluded as it did not include a M&E section.

The 78 LMICs who submitted NCPI questionnaires in 2006, 2008, and 2010 were subdivided into the 22 LMICs with the highest number of people living with HIV, hereafter known as the “high-burden countries”, compared with 56 other LMICs (hereafter known as “other LMICs”). The high-burden countries are Angola, Botswana, Brazil, Burundi, Cameroon, Côte d’Ivoire, India, Indonesia, Kenya, Lesotho, Malawi, Mozambique, Namibia, Nigeria, Russian Federation, Rwanda, South Africa, Swaziland, Uganda, Ukraine, Vietnam, and Zambia. The other LMICs are Algeria, Argentina, Armenia, Bangladesh, Belarus, Belize, Benin, Bulgaria, Burkina Faso, Cambodia, Chile, Colombia, Comoros, Congo, Costa Rica, Democratic Republic of the Congo, Dominica, Ecuador, El Salvador, Ethiopia, Fiji, Gabon, Georgia, Guatemala, Guinea, Guinea-Bissau, Islamic Republic of Iran, Kyrgyzstan, Lao People’s Democratic Republic, Madagascar, Malaysia, Mali, Marshall Islands, Mauritius, Mongolia, Morocco, Nepal, Nicaragua, Pakistan, Palau, Panama, Paraguay, Philippines, Republic of Moldova, Romania, Saint Kitts and Nevis, Saint Lucia, Saint Vincent and the Grenadines, Senegal, Somalia, Sri Lanka, Tajikistan, the former Yugoslav Republic of Macedonia, Togo, Turkey, and Uruguay.

This analysis focuses on four primary areas: government commitments to HIV M&E systems, HIV M&E expenditures, HIV M&E system capacity, and HIV M&E system performance.

Government commitments

Government commitment was measured using three questions in the NCPI, outlined in Additional file 1. The scores for each question were averaged to create a domain score for each country, as explained further below and detailed in Additional file 1. There were two questions that could not be included in the domain score as they were not asked in all three reporting cycles, but are displayed in Table 2.

Table 2 Summary of government commitment to HIV monitoring and evaluation systems among low- and middle-income countries who completed the NCPI questionnaires in 2006, 2008, and 2010

HIV M&E expenditures

HIV M&E expenditures were drawn from UNGASS Indicator 1 which measures levels of HIV/AIDS spending and is collected through a standard reporting form [11]. This national funding matrix is a two-dimensional matrix showing how funds are spent at the national level and where those funds are sourced. This is accomplished by relating financing sources (funding entities that disburse money to agents) with AIDS spending categories (goods, services, and activities delivered as part of the HIV response).

Domestic spending can be classified in the National Funding Matrix under 89 specific spending categories. The spending categories follow a functional classification described in the National AIDS Spending classifications and definitions [12, 13]. We analyzed the spending trends and funding sources of M&E expenditures which are categorized under AIDS spending category 04.03 “Monitoring and evaluation” [12]. In addition, analyses included a broader sub-set of spending categories which we considered important components of an adequate M&E framework, referred to as “M&E+” which includes the following: 04.04 (Operations research), 04.05 (Serological surveillance), 04.06 (HIV drug-resistance surveillance), 04.08 (Information technology), and 04.09 (Patient tracking) in addition to 04.03 “Monitoring and evaluation”.

A descriptive analysis of HIV and health expenditures in the time span from 2007 to 2010 of the subset of 43 LMICs with reported data for at least two of these four years was conducted. Spending data prior to 2007 was not available. Comparable data was used to reveal the trend of the share of M&E and M&E + out of total spending on HIV/AIDS. When necessary, the quantity of data points was broadened by conducting a linear interpolation between two known points in the dataset.

HIV M&E system capacity

HIV M&E system capacity was calculated as a composite of three domains: government engagement, partner/civil society engagement, and data generation. All NCPI questions related to M&E from both government and civil society sections administered as part of the 2006, 2008, and 2010 UNGASS reporting cycle were reviewed. Questions that remained consistent over the three cycles and were relevant to HIV M&E system capacity were grouped into three domains: government engagement, partner and civil society engagement, and data generation. Percentage scores for each question were calculated using the scoring rubric outlined in Additional file 2.

Index construction and analysis

The three domains, (i) government engagement, (ii) partner and civil society engagement, and (iii) data generation, were also aggregated to form an index score, the HIV M&E System Capacity Index (MESCI) (see Additional file 2 for individual components and scoring calculations). While information on data utilization was collected through the NCPI, only one question was asked in all three reporting cycles, and is therefore not included in domain or index calculations.

To calculate the MESCI, an arithmetic mean was generated for each of the three domains in the MESCI using the scoring methodology outlined in Additional file 2. Means included only questions that were asked in 2006, 2008, and 2010 NCPI cycles. The three domain scores were then averaged to generate the overall HIV M&E System Capacity Index, which ranges from 0% to 100%, for each country and each time-point. MESCI component, domain, and overall scores, including 95% confidence intervals, are presented for each round of UNGASS reporting, stratified by high-burden countries vs. other LMICs. Countries that did not answer a particular question were given a missing value for that question, rather than imputing a response.

HIV M&E performance

UNGASS indicator reporting from 2006 to 2010 was used to represent HIV M&E system performance. Reported UNGASS indicators were analyzed to assess levels of country reporting for three data streams, namely indicators measured from nationally representative, population-based surveys (UNGASS indicator numbers 7, 13 15, 16, 17), behavioral surveillance for key populations at higher risk of HIV, i.e., men who have sex with men, sex workers, and people who inject drugs (UNGASS indicator numbers 8, 14, 18, 19, 20, 21, 23), and service delivery metrics were derived from program data (UNGASS indicator numbers 4,5,6,24) (Additional file 3) [9–11]. Countries reporting any of the indicators in each data stream were counted as having reported in that particular data stream. The scores for each of the three streams were averaged to create an HIV M&E performance score. We also examined the number of countries able to report program data on antiretroviral therapy by sex; this metric was not available in 2008 and is therefore not included in the performance score calculations.

All analyses were conducted using SAS v. 9.2.

Results

Government commitment to HIV M&E

Levels of government commitment to HIV M&E generally increased between 2006 and 2008 (Table 2). This was particularly true for countries reporting having an HIV M&E plan in place, which significantly increased from 64% to 95% in high-burden countries, and from 61% to 83% in other LMICs. The percentage of countries reporting secured funding for a HIV M&E plan budget was stable between 2006 and 2008, but decreased between 2008 and 2010. The overall government commitment domain score in high-burden countries increased considerably between 2006 and 2008, but decreased slightly from 2008 to 2010 (65% to 88% to 81%, respectively).

HIV M&E expenditures

Over time, reporting on HIV M&E expenditure increased from 2004 until 2009 when the number of countries reported peaked, since slightly fewer countries reported in 2010. The total AIDS spending for the 43 LMICs studied grew from 2007 to 2010, reaching a total of $3.9 billion (Table 3). In these same countries, the percentage of that money that was spent on both M&E and M&E + grew steadily from 2007 to 2010 (from 1.16% to 1.38%, and from 1.55% to 1.91%, respectively). The percentage of total AIDS spending allotted to M&E and M&E + generally increased over time among high-burden LMICs. However, high-burden LMICs reported a lower percentage of funds going to M&E + than that seen in other LMICs in all years analyzed; the same was true for three of four years of M&E spending (2007, 2008, and 2010).

Table 3 Total AIDS spending and HIV M&E spending as a percent of total AIDS spending for low- and middle-income countries (n =43), 2007–2010

HIV M&E system capacity

Government engagement

Overall, there were improvements in government engagement from 2006 to 2008 in both high-burden and other LMICs, though these gains decreased by 2010 in high-burden countries (Table 4). However, government engagement was higher over time in high-burden countries (84% in 2006 and 2010) compared to other LMICs (56% in 2006 to 70% in 2010), though the gap did lessen over time. While the presence of a functional national HIV M&E unit was reported by 98% of all high-burden countries and 76% of other LMICs in 2010, only slightly more than 60% of countries in both groups reported that these groups meet regularly (representing a more than 20% decrease from 2008 in high-burden countries).

Table 4 Summary of government and partner/civil society engagement to HIV M&E systems, data generation, and data dissemination/utilization for HIV M&E systems, and the HIV M&E System Capacity Index among low- and middle-income countries who completed NCPI questionnaires in 2006, 2008, and 2010

Partner and civil society engagement

In high-burden LMICs, there was a significant increase in overall partner and civil society engagement from 2006 (66%) to 2008 and 2010 (97% and 94%, respectively) (Table 4). The percentage of high-burden countries reporting mechanisms to ensure major partners submit M&E data/reports to the national HIV M&E unit nearly doubled, from 53% in 2006 to 90% in 2010. Reporting the existence of an HIV M&E plan developed in consultation with civil society also increased in high-burden countries from 75% in 2006 to 95% in 2010. The perceived extent of civil society inclusion (61% in 2010 for high-burden countries) and the alignment of partners’ HIV M&E requirements with national M&E plans (a key component of the “Three Ones”; 68% in 2010) were both lower relative to other scores in this domain for high-burden countries.

Data generation

Overall, high-burden countries had very high scores in the data generation domain, with improvements from 2006 (81%) to 2008 (97%) that were maintained into 2010 (Table 4). Other LMICs did not report as many improvements over time, and in fact showed decreases between 2008 and 2010 in the number of countries reporting that their national HIV M&E plan included i) guidelines on tools for data collection (88% to 85%), ii) a strategy for assessing data quality (71% to 59%), iii) a data dissemination and use strategy (84% to 79%) and iv) the presence of a functional national and subnational HIS (91% to 85% and 80% to 78%, respectively).

Data utilization

While no domain score was calculated for data utilization, the percentage of countries producing annual HIV M&E reports increased steadily from 2006 to 2010 across all LMICs (Table 4).

HIV M&E system capacity index

Overall, the index improved between 2006 and 2008, though this improvement was greater in high-burden LMICs (a significant shift from 78% to 94%) than in other LMICs (70% to 77%) (Table 4). High-burden LMICs saw a slight decrease between 2008 and 2010. Among high burden LMICs, the trajectories of government commitment and HIV M&E system capacity map closely onto one another, while in other LMICs, government commitment decreases from 2008–2010 as HIV M&E system capacity increases (Figure 1).

Figure 1

Government commitment score and HIV Monitoring and Evaluation System Capacity Index Score from 2006 to 2010. Note: Solid lines represent high-burden LMICs, dotted lines represent other LMICs.

HIV M&E system performance

M&E system performance generally shows improvements over time among all LMICs regardless of HIV burden (Table 5). For high-burden countries, the overall progress summary score significantly increased from 52% in 2006 to 89% in 2010 and from 37% to 89% for other LMICs. In 2006, only half of the high-burden countries and just one in five of the other LMICs could report on indicators related to the population based indicators. By 2010, this had significantly increased to 95% of high-burden and 84% of other LMICs. The same trend is also seen for indicators related to key populations, with increases from 23% to 73% for high-burden countries and from 50% to 82% for other LMICs, as well as for the service delivery indicators (from 82% to 100% for high-burden and 41% to 100% for other LMICs).Among high burden countries, both MESCI and HIV M&E performance increase from 2006–2008, and level or slightly decrease by 2010 (Figure 2). In contrast, among other LMICs, HIV M&E performance increases much faster than the MESCI between 2006–2008 (Figure 2).

Table 5 Monitoring and evaluation system performance 2006–2010, as measured through country-reported UNGASS data

Figure 2

HIV Monitoring and Evaluation System Capacity Index Score and HIV monitoring and evaluation performance score from 2006 to 2010. Note: Solid lines represent high-burden LMICs, dotted lines represent other LMICs.

Discussion

The rapid expansion of available resources to fight the global AIDS epidemic since the new millennium has brought with it commensurate expectations for accountability at subnational, national, and international levels. Our analysis shows that government commitment to HIV M&E, particularly among high-burden LMICS, has increased considerably since 2006, as has the percentage of AIDS spending allocated to HIV M&E. As government commitments and spending have increased, so have government and partner engagement, data use, and data generation. Over this same time period, HIV M&E system performance has increased by over 70% in high-burden LMICs, and by 140% in other LMICs.

Reported HIV M&E expenditures were low across the board when judged against the 5% to 10% of total AIDS spending as a recommended benchmark [6], never being higher than 1.4% in high-burden countries. While it is likely that there were additional sources of funding for HIV M&E systems beyond those reported in UNGASS reporting, it seems unlikely that they would reach recommended levels. However, the suggested level may also need to be reconsidered at this time as evidenced by the improved performance of the M&E systems at current funding levels.

Many countries still lack the systems to actually track spending on M&E, a barrier that is not confined to the AIDS response. Recognizing this challenge, overall support to national statistical services from international agencies and donor countries has grown considerably, from $US1.0 billion in 2006 to $US2.3 billion in 2010–2012 [14].The relationship seen between the MESCI score and HIV M&E systems performance (Figure 2) indicates that among high-burden LMICs, countries with high capacity should be better able to produce HIV M&E data to meet national and international needs. This relationship does not seem to hold true for other LMICs, where system performance is higher than system capacity. One potential explanation is that countries with lower burdens of HIV are able to monitor their epidemic with a lower system capacity than that required for high-burden countries.

The mandate for national and international HIV M&E provided in the 2011 Political Declaration on HIV/AIDS [15], to further strengthen and support HIV data systems while working to integrate them with broader national health systems, is a significant shift from the previous strategy of often parallel systems. There has also been a call to use HIV resources more strategically, to encourage greater efficiencies and higher performance [16]. The decrease seen in the number of high-burden countries reporting a National HIV M&E Committee or Working Group that meets regularly may be of concern, as that is one of the main bodies that will be able to guide the integration agenda that has been set for HIV M&E. However, it may also be the evolution of M&E systems from the early and intensive development stages to the current and more established systems maintenance stage. Folding the resources and experience from the HIV response in with those of other health sectors offers the opportunity for synergies that could be used to leverage the maximum benefit out of all systems and resources. Neither the NCPI M&E section nor the MESCI as they are currently structured are able to measure integration between HIV M&E systems and other national M&E systems, but this is an important modification that should be considered in future years.

Since the beginning of the AIDS epidemic, civil society has played a key role in the response in countries all over the world. The engagement in the monitoring of the AIDS epidemic has also been shown to be important [17]. Civil society engagement in the development of M&E plans was reported to be high already in 2008 (around 80% of countries) and increased to more than 90% of countries in 2010. However, the perceived extent of civil society inclusion was lower, at only 60% in high-burden countries and 50% in other LMICs, leaving room for considerable improvements.

This study has several key limitations. Data on government commitments and system capacity is derived from the NCPI, which is self-reported data representing both national government representative’s opinions of the status and performance of their HIV M&E systems and civil society/bilateral/multilateral opinion on perceived inclusion in HIV M&E activities. The nature of response to some of these questions is therefore dependent to some extent on the respondent and their personal measures of performance. Calculations of domain scores and the MESCI were limited to questions that appeared in all three reporting cycles analyzed – as NCPI reporting continues, the contents of each domain may be enhanced by a greater breadth of questions asked over time. Indicator performance and patterns of reporting over time was used as a proxy for M&E system performance in countries. Although there are limitations in this approach, consistent increases in reported data, from varying sources and data collection systems (program data, population-based surveys, behavioral surveillance), indicate that there has been progress in strengthening national M&E systems in recent years.

The most important reason for conducting M&E is to provide the data needed for guiding policy and program implementation. A functional M&E system collates and presents the data in a way that facilitates data use at all levels, including the general public and beneficiaries of HIV services [3]. Countries rated their data utilization for the development/revising of national AIDS strategies, resource allocation and program improvement in 2010 above average in most countries, though there is room for improvement. Accurate and regular data utilization is critical to ensuring epidemiologically appropriate and evidence-based resource allocation; gaps in this process may be in part responsible for the variable resource allocation strategies currently seen across countries [18].

Through the UNGASS reporting mechanism, countries were asked to submit nationally representative data. However, for an effective response it is not sufficient to collect data at the national level but data is also needed at the subnational level, e.g., to identify geographic areas where localized HIV epidemics or specific populations most affected by the epidemic are not being reached by services. In recent years, more focus has been put on these efforts [19].

The gathering of reliable data on the AIDS epidemic and the response to it is of increasing importance for donors such as the Global Fund to Fight AIDS, Tuberculosis, and Malaria, for whom performance-based funding is a core principle [20]. M&E is a cornerstone of this approach and allows donors to make funding decisions against proven measurable results. More recently, the Global Fund to Fight AIDS, Tuberculosis, and Malaria’s New Funding Model has put greater emphasis on impact measurement and allocative efficiency [21]. Further examples include the Investment Framework at a national level [16] and the PEPFAR reporting requirements at a donor level [22].

Conclusions

This analysis demonstrates increases in all domains of HIV M&E systems assessed between 2006 and 2010, as well as in HIV M&E funding, though the latter remained well below the recommended benchmarks. In this same window, HIV M&E system performance increased substantially. Building national M&E systems requires sustained efforts over long periods of time with effective leadership and coordination [7]. Ongoing investments into effective HIV M&E systems and data utilization are required. However, with the increased resource constraints in many countries, these investments must be spent strategically and according to the specific needs of each country [16]. There is no single M&E system that fits all countries; the systems should be based on an efficient and effective mix of standardized methods of data collection and analyses that meet country and international needs [23]. The 2011 Political Declaration on HIV/AIDS calls for the integration of HIV M&E systems into broader health information systems, a major repositioning for many countries who have invested heavily in parallel M&E systems [4]. This alignment of vertical interventions with the broader health system has gained momentum in light of the health-related Millennium Development Goal targets for which effective scale-up of services for individual diseases relies on the strength of the health system as a whole. In June 2013, a report published by the High-Level Panel on the Post 2015 Development Agenda called for a “New Data Revolution” to strengthen existing data collection systems in order to provide timely data for decision making, monitor delivery gaps, and ensure greater accountability [6]. The mandate to integrate HIV M&E systems within broader health information systems provides new opportunities to leverage and strengthen existing infrastructures. Analyses such as this offer key insights into this process, and should be a routine component of M&E system evaluation.

Authors’ information

TA, LF, and CA are employees of UNAIDS and were involved in the collection of data used in this study. WG is a former employee of UNAIDS, and was also involved in the collection of data used in this study.