This article is part of the supplement: SUPPORT Tools for evidence-informed health Policymaking (STP)

Open Access Guide

SUPPORT Tools for evidence-informed Policymaking in health 11: Finding and using evidence about local conditions

Simon Lewin1*, Andrew D Oxman2, John N Lavis3, Atle Fretheim4, Sebastian Garcia Marti5 and Susan Munabi-Babigumira6

Author Affiliations

1 Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway; Health Systems Research Unit, Medical Research Council of South Africa

2 Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway

3 Centre for Health Economics and Policy Analysis, Department of Clinical Epidemiology and Biostatistics; Department of Political Science, McMaster University, 1200 Main St. West, HSC-2D3, Hamilton, ON, Canada, L8N 3Z5

4 Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway; Section for International Health, Institute of General Practice and Community Medicine, Faculty of Medicine, University of Oslo, Norway

5 Institute for Clinical Effectiveness and Health Policy, Viamonte 2146, 3rd floor, C1056ABH, Buenos Aires, Argentina

6 Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway

For all author emails, please log on.

Health Research Policy and Systems 2009, 7(Suppl 1):S11  doi:10.1186/1478-4505-7-S1-S11

The electronic version of this article is the complete one and can be found online at: http://www.health-policy-systems.com/content/7/S1/S11


Published:16 December 2009

© 2009 Lewin et al; licensee BioMed Central Ltd.

This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers.

Evidence about local conditions is evidence that is available from the specific setting(s) in which a decision or action on a policy or programme option will be taken. Such evidence is always needed, together with other forms of evidence, in order to inform decisions about options. Global evidence is the best starting point for judgements about effects, factors that modify those effects, and insights into ways to approach and address problems. But local evidence is needed for most other judgements about what decisions and actions should be taken. In this article, we suggest five questions that can help to identify and appraise the local evidence that is needed to inform a decision about policy or programme options. These are: 1. What local evidence is needed to inform a decision about options? 2. How can the necessary local evidence be found? 3. How should the quality of the available local evidence be assessed? 4. Are there important variations in the availability, quality or results of local evidence? 5. How should local evidence be incorporated with other information?

About STP

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. The series is intended to help such people ensure that their decisions are well-informed by the best available research evidence. The SUPPORT tools and the ways in which they can be used are described in more detail in the Introduction to this series [1]. A glossary for the entire series is attached to each article (see Additional File 1). Links to Spanish, Portuguese, French and Chinese translations of this series can be found on the SUPPORT website http://www.support-collaboration.org webcite. Feedback about how to improve the tools in this series is welcome and should be sent to: STP@nokc.no.

Additional file 1. Glossary

Format: DOC Size: 79KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Scenarios

Scenario 1: You are a senior civil servant and have responsibility for putting forward a proposal for a new health reform. You want to ensure that the proposal clearly states the number of people likely to benefit from the health reform as well as the views of stakeholder groups regarding the new initiative.

Scenario 2: You work in the Ministry of Health and the Minister has decided on a new health reform. You have been instructed to write a background document for the reform and need to find information on the availability of resources to implement the planned changes and possible barriers to implementation.

Scenario 3: You work in an independent unit that supports the Ministry of Health in its use of evidence in policymaking. You have been commissioned to write a background document for a new health reform that may affect access to care. You need to find information on access to care for the elderly and for those with low incomes in your setting.

Background

This article suggests a number of questions that decision makers (Scenario 1) might ask their staff to consider regarding the finding and use of evidence on local conditions to inform health policy or programme options.

The article also suggests a number of questions that those who support decision makers (Scenarios 2 and 3) should consider both when guiding the identification and appraisal of evidence from their local setting to inform a decision on health policy or programme options, and when incorporating this evidence into health policymaking.

Options should always be informed by evidence about local conditions (hereafter referred to as local evidence) together with other forms of evidence. Global evidence - the best evidence from around the world - is the best starting point for judgements about the effects of options and factors that modify those effects [2], and for developing insight into ways in which problems can be approached and addressed. Local evidence is needed for most other judgements about what decisions and actions should be taken.

Local evidence is evidence that is available from the specific setting(s) in which a decision or action on an option will be taken. The word 'local' in this instance can refer to district, regional or national levels, depending on the nature of the policy issue being considered. Such evidence might include information on the presence of factors that modify the impacts of a policy (the modifying factors). Such modifying factors might include: the characteristics of an area and those who live or work in it; the need for services (prevalence, baseline risk or status); views and experiences; costs; political traditions; institutional capacity; and the availability of resources such as staff, equipment and drugs.

Local evidence may be obtained from a range of sources including: routine data (e.g. on the prevalence of diseases, healthcare utilisation, or service costs); survey data (e.g. on household conditions, health and demographics); and data from one-off studies (e.g. trials conducted locally, studies of consumers' views regarding a particular health issue, and cost-effectiveness evaluations). However, local evidence is often assessed only informally or not at all as part of policymaking processes. In some settings, such information may be difficult to locate or may be of poor quality. This article provides a systematic approach to finding, assessing, and incorporating local evidence into policymaking.

There are a number of ways in which local evidence may be useful (see Table 1 for a list of some of these). For example, policymakers may need local evidence on the prevalence or magnitude of a health issue in order to contextualise (and make relevant) the evidence available from global reviews or studies conducted elsewhere [3]. (See Table 2 for a discussion of this issue in the context of malaria treatment in Tanzania and Brazil.) Evidence based on information from the global, regional or national levels may not adequately describe a local situation. Local evidence may also be useful as part of a process of priority-setting for the development of evidence-informed policy and programme options [4]. Information on local delivery, financial or governance arrangements for healthcare may be needed to inform such decisions. The views and experiences of local stakeholders, such as health professionals or consumers, regarding a particular option constitutes another important form of local evidence [5,6]. (See Table 3 for examples of how local evidence has been used in Australia for assessing needs regarding general practice, and in South Africa and Mozambique regarding views about the use of insecticide-treated nets.) Finally, information on the local costs of an option and the availability of resources is essential in taking decisions regarding implementation and in planning the delivery of options [7-9]. (See Tables 4 and 5 for examples related to this issue in South Africa, Chile and the United States.)

Table 1. Uses of local evidence in informing decisions on options

Table 2. Using local evidence to estimate the magnitude of the problem or issue that an option aims to address

Table 3. Using local evidence to inform judgements about values and views regarding options

Table 4. Using local evidence to estimate the costs (and savings) of options

Table 5. Using local evidence to assess the availability of resources with a view to informing a decision regarding options

Local evidence may inform all stages of the policy process. For example, local evidence may place an issue on the policy agenda and so help to set policy goals. Local evidence may also be used by different stakeholders and interest groups to lobby for particular options. The Shack Dwellers Federation of Namibia, for example, provides support to local shack dweller associations for the collection of information on the socio-economic status of their members and other residents, and on the availability of local essential services. This information has been used to help identify local needs and also to provide local groups with a voice in government policy debates. Local groups are also able to use this information to lobby municipal officials and politicians in order to improve the quality of service provision in their areas and to make more land for housing accessible [10].

In addition to informing decisions about options directly, local evidence may be useful in monitoring the effects of a programme or policy over time in order to assess whether the anticipated impacts continue to be delivered [11]. (See Table 6 for a discussion of the use of local evidence in monitoring and evaluation in the context of antiretroviral treatment in South Africa.) Where data are collected routinely, some level of retrospective analysis may be possible and this can provide a baseline against which new programmes can be evaluated. Local evidence may also be useful in demonstrating trends in the effects of a programme across small geographic areas, such as neighbourhoods and districts, and in highlighting differences in implementation or uptake. Policymakers may also be concerned with the impacts of a programme on particular groups, such as vulnerable populations or minority groups. Local evidence may be useful in examining whether programme resources have been distributed equitably and if a programme is being implemented in ways that promote equity (see, for example, reference [12]).

Table 6. Using local evidence to monitor and evaluate policies

Policymakers should be cautious about using local evidence alone to assess the likely impacts of policy or programme options. Local evidence may be more directly relevant than studies conducted elsewhere, but it may also be less reliable due to important limitations in the studies that were done locally. In addition, even when reliable local evaluations are available, they may be misleading because of random errors. Judgements about whether to base a conclusion on a subset of the relevant evaluations (which happen to have been undertaken locally) or on the global evidence (including relevant studies undertaken in other settings) are better informed if made in the context of a systematic review of all of the relevant evaluations [2].

When a systematic review is unavailable and it is not feasible to conduct or commission one, local evidence alone may be used to inform decisions about options [13]. In these circumstances, policymakers should be aware of the risks of doing this, particularly if the local evaluation has important limitations (risk of bias) or is small (and therefore the results are imprecise). However, in (the relatively uncommon) circumstances where rigorous, directly relevant and large local impact evaluations are available [14], such evidence may be optimal for informing decisions.

Like all other forms of evidence, the reliability of local evidence needs to be appraised. In this paper we suggest five questions that can help to identify and appraise local evidence that is needed to inform a decision about options.

Questions to consider

The following five questions can be used to guide policymakers and others in identifying potential policy and programme options and finding related evidence. The relationship between these questions is shown in Figure 1:

thumbnailFigure 1. Finding and using evidence about local conditions to inform decisions about policy or programme options.

1. What local evidence is needed to inform a decision about options?

2. How can the necessary local evidence be found?

3. How should the quality of the available local evidence be assessed?

4. Are there important variations in the availability, quality or results of local evidence?

5. How should local evidence be incorporated with other information?

1. What local evidence is needed to inform a decision about options?

A range of local evidence may be needed to inform a decision about options (see Tables 1 to 8 for examples of the use of local evidence at different stages of the policy process). The evidence needed will depend on the nature of the option or question under consideration, the context, and the availability of different forms of local evidence.

Table 7. Using local evidence to diagnose the likely causes of a health issue

Table 8. Using local evidence to assess the likely impacts of options (i.e. the existence of modifying factors) and to identify barriers to implementing options

2. How can the necessary local evidence be found?

Local evidence may be obtained from routine health information systems, from larger surveys or studies that can be disaggregated, or from specific studies that have collected or analysed data on a local level. We discuss each of these in more detail below.

Like those processes related to global evidence of effects [15], the processes of searching for local evidence and making judgements regarding its inclusion and assessment should be systematic (i.e. systematic processes should be used to ensure that relevant research is identified, appraised and used appropriately) and should also be reported transparently. The selective use of local evidence (sometimes referred to as 'cherry picking') to demonstrate the usefulness of a particular option, should be avoided as it may result in important data or information being omitted or overlooked during the decision making process. For example, including only the largest estimates of the size of a problem, such as the proportion of children who do not complete their vaccination schedule, will result in a poor understanding of a problem such as incomplete vaccination. It may also result in scarce resources being allocated to interventions that are not needed, that do not respond to local needs, or that may not be needed at the extent to which they are provided. Using the largest estimates of the proportion of children who do not complete their vaccination schedule to inform a decision regarding options, for example, may result in more resources being allocated to the vaccination programme than are actually needed. Similarly, relying only on data on average immunisation coverage across a large population to inform policy may be inadequate. Such evidence may conceal large inequities in coverage across specific areas or groups.

While a wide range of sources of local evidence may be available, this evidence may not be available in a form that addresses the policy question under consideration. For example, data may be available from a survey on household access to different forms of sanitation, such as flush toilets or pit latrines. However, these data may not have been analysed at the level of aggregation needed, such as a specific health district or region, and may not indicate whether the sanitation facilities were operational. It may therefore be necessary to undertake further analysis of available data or to make assumptions regarding the applicability of the data to a particular policy question. We discuss this further in Questions 4 and 5 below.

Local collected data obtained from the routine health information system

National, district, or other local health authorities (or other organisations in the health system) often collect data routinely on a wide range of issues, including [16]:

Risk factors: Such as nutrition and blood pressure

Mortality and burden of disease: This includes health outcomes such as child mortality, TB treatment outcomes, peri-operative deaths, infectious disease and cancer notifications

Health service coverage:

∘ Coverage for clinical interventions or services such as childhood vaccinations or cervical screening rates

∘ Health service utilisation information such as length of hospital stay, number of outpatient visits for specific health conditions, and prescription drugs dispensed

∘ Routine surveys of patient satisfaction with care

Health systems resources:

∘ Healthcare expenditures according to various cost centres and programmes

∘ Human resource data such as numbers and grades of staff in different facilities and programmes, staff development programmes delivered, and staff absenteeism

∘ Clinical performance data such as post-surgical infection rates, time to treatment for people with myocardial infarctions

∘ Guidelines used for care delivery

∘ Adherence to guidelines for care delivery

Inequities in healthcare and health outcomes

For some of these sources, it may be possible to disaggregate data by specific groups, such as gender or age, or by specific local area, such as a neighbourhood or town [2]. Data from routine health information systems may not have been analysed systematically and considerable resources may be needed to undertake such analysis.

Good starting points for identifying local sources of routine data include the Health Information Departments of Ministries of Health, National Statistics Offices, and local health authorities. Increasingly, these departments publish lists of the range of data that they capture and analyse on the Internet. Many also regularly produce summary statistics. The City of Cape Town Health Department in South Africa, for example, publishes information on their website by sub-district for a small range of health indicators, such as number of live births, number of infant deaths, infant mortality rates, TB case loads and treatment outcomes (see: http://www.capetown.gov.za/en/cityhealth/Pages/CityHealth.aspx webcite). The Association of Public Health Observatories also provides data on key health indicators for each local authority in England (see: http://www.apho.org.uk/default.aspx?QN=P_HEALTH_PROFILES webcite). Local research institutions, health non-governmental organisations (NGOs), or the offices of bilateral or multi-lateral agencies, such as WHO country offices, may also be able to advise on local sources of routinely collected data. Some commercial databases may include useful local evidence, for example, related to local prices for drugs, their availability, and the use of other technologies. In general, local health authorities should maintain an overview of local sources of routinely collected data. Policymakers may want to familiarise themselves with these.

Data from larger surveys or studies that can be disaggregated to local level

Important data sources include large surveys or studies such as national censuses, regional surveys of access to basic facilities, and national demographic and health surveys. For some of these sources, disaggregation to the provincial or city level may be possible or may already have been conducted. For example, the Neighbourhood Statistics site of the United Kingdom Office for National Statistics (see: http://www.neighbourhood.statistics.gov.uk/dissemination/ webcite allows users to find statistics for an area by entering its name or postcode. Data on a wide range of topics are available, including access to services, crime and safety, general health, and teenage pregnancies. Similarly, the website of Statistics South Africa includes information on a wide range of topics disaggregated to a provincial level. For example, this includes information, based on data from a national household survey, on health insurance coverage and health service consultations by province (see: http://www.statssa.gov.za webcite).

For other datasets, analysis to the appropriate local level may not be conducted routinely. This may be feasible, though, if data are tagged by geographic area. The agency that conducted the survey or the agency housing these data should be able to advise on whether further disaggregation to the local level is possible. The process of further analysis is more complex and statistical support is therefore generally recommended. Some health data, such as the use of treatment services for sexually transmitted infections and HIV/AIDS, may be considered sensitive in nature. It may therefore not be possible to obtain data disaggregated to a local level if the agencies housing these data need to ensure that specific individuals cannot be identified from information placed in the public domain.

Specific studies that have collected and analysed data on a local area

Large numbers of research studies collect, analyse and report data focused on a local area such as a province of a country or a city. These studies may use a wide range of data collection and analysis methods. Studies that present data on a local area can be located in several ways:

• By searching (ideally with the help of an information specialist) global databases of published research papers, such as PubMed, the Cochrane Library or the WHO regional databases (e.g. the Latin American and Caribbean Health Sciences Database [LILACS]), using geographic terms such as 'Caracas' or 'Buenos Aires'. PubMed includes a hedge, or validated search strategy, that allows users to search for administrative databases studies, community surveys and qualitative studies (these may be helpful in providing information on utilisation patterns and on views and experiences, for example). This is available at: http://www.nlm.nih.gov/nichsr/hedges/search.html webcite

• By searching (ideally with the help of an information specialist) sources of 'grey' or unpublished literature, such as Google Scholar, the WHO Library Information System http://dosei.who.int/uhtbin/cgisirsi/Mon+May++4+21:00:46+MEST+2009/0/49 webcite, and OpenSIGLE (System for information on grey literature in Europe: http://opensigle.inist.fr webcite). Many local studies, such as operational research on health services, are published as reports on the web but may not be published in research journals. Grey literature is therefore a good source of such evidence

• By contacting local researchers in universities, research institutes or health departments or local research networks for relevant information, including unpublished study reports

• By contacting or searching the resources of health observatories such as the European Observatory on Health Care Systems http://www.euro.who.int/observatory webcite, the International Observatory on Mental Health Systems http://www.cimh.unimelb.edu.au/iomhs webcite, or the Africa Health Workforce Observatory http://www.afro.who.int/hrh-observatory webcite

3. How should the quality of the available local evidence be assessed?

Like all other forms of evidence, the quality of local evidence needs to be assessed. Where data quality is poor, interpretation can be difficult and there is a danger that faulty conclusions may be drawn. When considering local evidence, it may be useful to differentiate between data (i.e. the raw product of measurements or observations) and information (i.e. data that are organised or analysed in relation to a specific question or issue and are therefore more useful for decision making [17]). Some of the potential problems with local evidence relate to data (e.g. the ways in which measurement was done). Others relate to how these data are converted into information (e.g. as part of the analysis process).

A number of factors may compromise the quality of routinely-collected local data. Healthcare workers who collate and enter data, for example, may be poorly trained in this task. Similarly, if they do not receive timely feedback, they may not understand the usefulness of the data to informing service delivery. Data entry may also compete with a large number of other care tasks in clinics or hospitals and central quality control may be inadequate [18]. Problems related to the quality of data may be difficult to rectify once data have been collected. In contrast, it may be easier to rectify inadequacies in information by re-running an analysis. Systems for the collection of local data should ideally be designed to provide useful and timely feedback of information to those who collect such data.

Most local evidence that is used to inform decisions about options is descriptive (i.e. it includes simple summaries of the sample and measures or outcomes included in the data) rather than comparative (i.e. based on the comparison of one set of data with another, for example by area or over time). There are some exceptions, such as evidence about inequities which relies on comparisons.

The descriptive nature of most local evidence has implications for assessing its quality. In the case of comparative studies, the assessment of quality is focused primarily on the risk of bias (i.e. the risk of "a systematic error, or deviation from the truth, in results or inferences" [19]). In contrast, key questions in assessing the quality of local evidence include the following (adapted from [11]. Also see Table 9 for a summary of questions that can be used to guide assessments of the quality of local evidence):

Table 9. Questions to guide assessment of the quality of local evidence

Is the evidence representative? This question focuses on whether the evidence correctly represents the wider population from which it is drawn or to which the findings are generalised. There are several components to this question: firstly, is there a clear description of the source of the evidence? Secondly, if the evidence is drawn from a sample of the population of interest, is there a clear description of how the sampling was conducted, and was the sampling approach that was used appropriate? Thirdly, is there a description of how any inferences or generalisations were made to the wider population?

Is the evidence accurate? This question is concerned with whether the available data match, or are likely to match, the actual value of the outcome measured. When addressing this question, the user may want to consider whether there are clear descriptions of the processes through which the data were collected. Issues that should be addressed include: who collected the data and were they appropriately trained and supported in this task, what tools were used for data collection, when were the data were collected, was the quality of the collected data monitored, how was the analysis done (were the methods of analysis reported clearly), and were any data limitations discussed

Are appropriate outcomes reported? This question focuses on whether the measures reported in the data (such as treatment outcomes or health utilisation measures) are suitable for addressing the question for which the data will be used. When addressing this question, the user may want to consider whether there is a clear description of the outcome or outcomes measured, whether they are reliable, and whether these outcomes will provide a reasonable assessment of the health issue. If policymakers are considering, for example, how to improve the quality of care for people with TB, routinely-reported TB treatment outcomes may be a useful measure. This is because the completion of TB treatment is likely to be related to the quality of care received by patients

4. Are there important variations in the availability, quality or results of local evidence?

When assessing and using local evidence, it is important to be aware of variations in its availability, quality or results. Each of these issues is discussed below.

Availability

Large variations always occur in the range or depth of available local evidence across geographic areas, jurisdictions or population groups. In many instances, this variation may simply reflect differences in the policies or capacity of health authorities or other agencies across different jurisdictions or areas. In some cases, however, variations in the availability of local evidence across groups or areas may reflect other underlying inequities. These may include the poor access that certain groups have to health facilities, or the failure of surveys to include 'hard to reach' groups such as migrant populations, those speaking other languages, or those living in remote or poorly serviced areas. Groups that are stigmatised on the basis of ethnicity or sexual orientation, for example, or because they are viewed as illegal migrants, may also be reluctant to identify themselves as belonging to these groups for the purposes of data collection [20,21]. There may therefore be little available local evidence related to these groups and collecting such data may be very challenging. Those using local data need to explore the reasons for variations in its availability and consider such factors in the decision making process.

Availability may be limited in other ways. Firstly, evidence may be available from only one source, making it difficult to cross-check the information's reliability. Secondly, information may be available for a large area that includes the area of policy interest but in a form that does not allow this local area information to be separated from the wider dataset. Thirdly, policymakers may have access to good quality data from a neighbouring area and may then have to assess the extent to which these data can be generalised to the area of interest. Finally, local evidence may be available only for an indicator assessing a related health issue. For example, policymakers in Colombia required data on the number of hospitalisations for meningitis but this information was not available routinely. However, the number of deaths due to meningitis in Columbia was available from the WHOSIS information system http://apps.who.int/whosis/database/mort/table1.cfm webcite. In addition, data on meningitis mortality rates were available from a local source http://www.scielo.br/pdf/rsap/v8s1/v8s1a04.pdf webcite. From these two sets of data, it is possible to estimate the total number of meningitis cases in the country.

Quality and results

Different sources of local evidence may differ in quality. In addition, the quality of local evidence may differ from that of other forms of evidence used in decision making. For example, a study of routine malaria data in Mozambique compared paper-based district records of adult inpatient malaria cases and deaths with digital data captured at the provincial level. Large discrepancies between these sources of data were identified (a 62% difference for cases and a 48% difference for deaths). The authors suggested that these variations may be related to errors in the data entry process at the provincial level [22]. Such differences in data quality should be considered explicitly in the decision making process.

Variations in the results of local evidence on a particular health issue across sources of local evidence may occur for a number of reasons, including:

• Differences in the way in which the issue was defined and measured across the sources

• Differences between the individuals, groups or other entities about whom data were collected across the sources

• Differences in the comparators used

• Differences (where applicable) in the interventions delivered

• Differences in the ways in which data were collected and analysed across the sources

When considering such variations, users of these data should explore the following questions:

• Is the variation potentially important from a clinical or policy perspective?

• If the variation is important, is a reasonable explanation clear from the data sources, or can a reasonable explanation be hypothesised (e.g. differences in recruitment, measurement, analysis etc.)?

• Are there other sources of information against which the local evidence can be compared?

Users of data should document any decisions they take regarding the interpretation of the evidence and should note any uncertainties, as discussed below.

5. How should local evidence be incorporated with other information?

Policy decisions require a combination of global evidence (the best available evidence from around the world) - ideally from systematic reviews - and different types of local evidence, assumptions and judgements. When local evidence is key to a policy decision (i.e. it might influence a decision in one direction or another) it is important to:

• Describe the approach used to identify the local evidence. Ideally a systematic approach to accessing this evidence should be used

• Describe the approach used to assess the local evidence. As noted earlier, a systematic approach to assessing evidence is recommended. When shortcuts are necessary, or it is necessary to make assumptions or use informal observations, these should be made transparent

• Describe clearly what local evidence is used and from where the evidence is obtained. This should include detail related to the specific groups or communities from which the evidence is drawn. As far as possible, documents and other sources should be cited and made available to others involved in the decision making process

• Describe any important gaps or uncertainties in the evidence due to the lack of local information or its poor quality. A study of the use of data available from the national Australian Childhood Immunization Register, for example, found that there were challenges in using the Register to adequately measure immunisation rates and outcomes in specific populations, such as remote indigenous groups [13]. Similar uncertainties have been reported from LMICs [23,24]. There may also be uncertainties in evidence due to conflicting findings between different sets of local evidence. For example, hospital mortality rates, complication rates, or duration of stay in intensive care may all be used to assess the quality of surgical care. Studies have found a poor correlation between these different indicators [18,25,26]. Consequently, it may be difficult to decide which set of data best reflects the 'real' quality of surgical services in a hospital or region and therefore which dataset should be used to inform policymaking. The applicability of local evidence to particular population subgroups may also be uncertain. For example, local evidence on teenage pregnancy rates may be available for the general population but not available by population subgroups (e.g. by ethnicity or language)

• Finally, it is important to identify and discuss any differences between the findings obtained from global evidence and those obtained from local evidence. For example, global evidence suggests that lay health workers can be effective in improving the uptake of immunisation in children [27]. However, local evidence might suggest otherwise if there are strong local views that lay people are inadequately qualified to provide health advice. In this instance, the promotion of this cadre would be less effective locally. Such local evidence might lead to less confidence (i.e. greater uncertainty) about the applicability of global evidence on lay health workers for immunisation uptake, even though the global review would still be seen as providing the best available estimate of effectiveness. Caution also needs to be used in applying economic evidence from other settings to a particular jurisdiction as the relative costs of some inputs may vary greatly across settings. For example, human resource costs generally vary locally while pharmaceutical costs may be similar across settings.

A good understanding of the local context and conditions may be helpful in interpreting both local and global evidence [28]. Key elements of context that should be considered include: the physical context (such as health facilities, supply chains, banking systems, etc.), human resources, knowledge (including the skills to implement a policy or intervention), the socio-cultural context (including issues such as belief systems, values, corruption, etc.), and the political context. Tools such as political mapping may be useful in developing an understanding of political context [29,30].

Approaches such as rapid appraisal can be used to bring together the range of different data available at the local and global levels to address a specific policy question. For example, this approach has been used to draw together data related to the management of diabetes care in Georgia and in Kyrgyzstan [31,32]. Local evidence, together with an appraisal of its reliability, may also be incorporated into policy briefs and a range of other documents that are used to inform policy processes. We discuss the use of policy briefs in more detail elsewhere [33].

Conclusion

Local evidence may inform all stages of the policy process - from influencing the policy agenda through to shaping programme choices and monitoring programme sustainability (see Table 10 for examples of the types of local evidence that might be relevant to specific policy questions). Such evidence may be obtained from routine health information systems, from surveys or studies that can be disaggregated, or from studies in which data have been collected or analysed on a local level. Both the evidence needed and the evidence available will depend on the nature of the policy question under consideration and the context.

Table 10. Types of local evidence to address specific policy questions

In many settings, steps need to be taken to improve the quality and use of data about local conditions. These may include motivating data collectors by ensuring that such information is useful to them and fed back in a timely way. It may also be necessary to ensure that policymakers and those who support them are aware of the sources of data about local conditions. As with other forms of evidence, the quality of local evidence needs to be assessed. Policymakers should be cautious about using local evidence alone to assess the likely impacts of policy or programme options. Local evidence may be more directly relevant than studies conducted elsewhere. But it may also be less reliable due to the important limitations of studies that are undertaken locally.

Resources

Useful documents and further reading

- WHO. World Health Statistics. Indicator compendium (Interim version). Geneva: World Health Organisation. 2009 http://www.who.int/whosis/indicators/en/ webcite.

- The 'Creating Excellence' network in the United Kingdom has produced a short local evidence guide and a toolkit on gathering and analysing local level data. http://www.creatingexcellence.org.uk/regeneration-renewal-news262.html webcite

- Department for Education and Skills, United Kingdom. Using local evidence. A leaflet for service managers, planners and commissioners. http://www.dcsf.gov.uk/everychildmatters/_download/?id=5728 webcite

Links to websites

WHO Statistical Information System (WHOSIS): http://www.who.int/whosis/en webcite - This is an interactive database bringing together core health statistics for the 193 WHO Member States. It comprises more than 100 indicators, which can be accessed by way of a quick search, by major categories, or through user-defined tables.

African Index Medicus: http://indexmedicus.afro.who.int webcite - An international index to African health literature and information sources produced by the WHO in collaboration with the Association for Health Information and Libraries in Africa. It provides access to health information published in, or related to, Africa and can be searched at no cost.

The Cochrane Library: http://www3.interscience.wiley.com/cgi-bin/mrwhome/106568753/HOME webcite - The Cochrane Library contains high-quality, independent evidence to inform healthcare decision making. It includes reliable evidence from Cochrane and other systematic reviews and clinical trials. Cochrane reviews provide the combined results of the world's best medical research studies and are recognised as the gold standard in evidence-based healthcare.

PubMed: http://www.ncbi.nlm.nih.gov/pubmed webcite - The PubMed database contains more than 19 million citations for biomedical articles from a wide range of indexed journals and can be searched at no cost.

Health Metrics Network: http://www.who.int/healthmetrics/en webcite - A global partnership on health information system strengthening. The website provides a range of tools and information to support health information system strengthening.

Demographic and health survey data: http://www.measuredhs.com webcite - The demographic and health surveys programme has collected, analysed and disseminated data on population, health, HIV and nutrition through more than 200 surveys in over 75 countries. The website provides a range of freely available data from these surveys.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

SL prepared the first draft of this article. ADO, JNL, AF, SGM and SMB contributed to drafting and revising it.

Acknowledgements

Please see the Introduction to this series for acknowledgements of funders and contributors. In addition, we would like to acknowledge Xavier Bosch-Capblanch for helpful comments on an earlier version of this article.

This article has been published as part of Health Research Policy and Systems Volume 7 Supplement 1, 2009: SUPPORT Tools for evidence-informed health Policymaking (STP). The full contents of the supplement are available online at http://www.health-policy-systems.com/content/7/S1.

References

  1. Lavis JN, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). Introduction.

    Health Res Policy Syst 2009, 7(Suppl 1):I1. PubMed Abstract | BioMed Central Full Text OpenURL

  2. Oxman AD, Lavis JN, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 1. What is evidence-informed policymaking.

    Health Res Policy Syst 2009, 7(Suppl 1):S1. PubMed Abstract | BioMed Central Full Text OpenURL

  3. McLaren L, Ghali LM, Lorenzetti D, Rock M: Out of context? Translating evidence from the North Karelia project over place and time.

    Health Educ Res 2007, 22:414-24. PubMed Abstract | Publisher Full Text OpenURL

  4. Lavis JN, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 3. Setting priorities for supporting evidence-informed policymaking.

    Health Res Policy Syst 2009, 7(Suppl 1):S3. PubMed Abstract | BioMed Central Full Text OpenURL

  5. Crawford MJ, Rutter D, Manley C, Weaver T, Bhui K, Fulop N, Tyrer P: Systematic review of involving patients in the planning and development of health care.

    BMJ 2002, 325:1263. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  6. Meads GD, Griffiths FE, Goode SD, Iwami M: Lessons from local engagement in Latin American health systems.

    Health Expect 2007, 10:407-18. PubMed Abstract | Publisher Full Text OpenURL

  7. Brunetti M, Oxman A, Pregno S, Lord J, Shemilt I, Vale L, et al.: GRADE guidelines: 10. Special challenges - resource use.

    J Clin Epidemiol, in press. OpenURL

  8. Hoffmann C, Stoykova BA, Nixon J, Glanville JM, Misso K, Drummond MF: Do health-care decision makers find economic evaluations useful? The findings of focus group research in UK health authorities.

    Value Health 2002, 5:71-8. PubMed Abstract | Publisher Full Text OpenURL

  9. Oxman AD, Lavis JN, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 10. Taking equity into consideration when assessing the findings of a systematic review.

    Health Res Policy Syst 2009, 7(Suppl 1):S10. PubMed Abstract | BioMed Central Full Text OpenURL

  10. UN Habitat: [http:/ / www.ucl.ac.uk/ dpu-projects/ drivers_urb_change/ urb_society/ pdf_social_emancip/ HABITAT_BestPractice_Shack_Dwellers _Housing_Lobby_Group.pdf ] webcite

    Partnership between shack dwellers and a local NGO housing lobby group, Windhoek, Namibia. UN Habitat and the Together Foundation: Best Practices Database; 2002. OpenURL

  11. The South West Centre for Sustainable Communities: [http://www.creatingexcellence.org.uk/ceimages/CE_LocalEvidenceGuide.pdf] webcite

    Local evidence guide. A guide to finding local data and statistics in the South West. Creating: Excellence; 2008. OpenURL

  12. Mindell J, Klodawski E, Fitzpatrick J: Using routine data to measure ethnic differentials in access to coronary revascularization.

    J Public Health (Oxf) 2008, 30(1):45-53. PubMed Abstract | Publisher Full Text OpenURL

  13. Morrato EH, Elias M, Gericke CA: Using population-based routine data for evidence-based health policy decisions: lessons from three examples of setting and evaluating national health policy in Australia, the UK and the USA.

    J Public Health (Oxf). 2007, 29(4):463-71. PubMed Abstract | Publisher Full Text OpenURL

  14. Fretheim A, Oxman AD, Lavis JN, Lewin S: SUPPORT Tools for evidence-informed health Policymaking (STP). 18. Planning monitoring and evaluation of policies.

    Health Res Policy Syst 2009, 7(Suppl 1):S18. PubMed Abstract | BioMed Central Full Text OpenURL

  15. Lavis JN, Oxman AD, Grimshaw J, Johansen M, Boyko JA, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 7. Finding systematic reviews.

    Health Res Policy Syst 2009, 7(Suppl 1):S7. PubMed Abstract | BioMed Central Full Text OpenURL

  16. World Health Organization (WHO): World Health Statistics. Indicator compendium (Interim version). Geneva, World Health Organisation; 2009. OpenURL

  17. Ackoff RL: From data to wisdom.

    Journal of Applied Systems Analysis 1989, 15:3-9. OpenURL

  18. Powell AE, Davies HT, Thomson RG: Using routine comparative data to assess the quality of health care: understanding and avoiding common pitfalls.

    Qual Saf Health Care 2003, 12:122-8. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  19. Higgins JPT, Green S: [http://www.cochrane-handbook.org/] webcite

    Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1 [updated September 2008]. The Cochrane Collaboration; 2008. OpenURL

  20. Hajioff S, McKee M: The health of the Roma people: a review of the published literature.

    J Epidemiol Community Health 2000, 54:864-9. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  21. Meyer IH: Why lesbian, gay, bisexual, and transgender public health?

    Am J Public Health 2001, 91:856-9. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  22. Chilundo B, Sundby J, Aanestad M: Analysing the quality of routine malaria data in Mozambique.

    Malar J 2004, 3:3. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  23. Bosch-Capblanch X, Ronveaux O, Doyle V, Remedios V, Bchir A: Accuracy and quality of immunization information systems in forty-one low income countries.

    Trop Med Int Health 2009, 14:2-10. PubMed Abstract | Publisher Full Text OpenURL

  24. Lim SS, Stein DB, Charrow A, Murray CJ: Tracking progress towards universal childhood immunisation and the impact of global initiatives: a systematic analysis of three-dose diphtheria, tetanus, and pertussis immunisation coverage.

    Lancet 2008, 372:2031-46. PubMed Abstract | Publisher Full Text OpenURL

  25. Hartz AJ, Kuhn EM: Comparing hospitals that perform coronary artery bypass surgery: the effect of outcome measures and data sources.

    Am J Public Health 1994, 84:1609-14. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  26. Iezzoni LI: Using risk-adjusted outcomes to assess clinical practice: an overview of issues pertaining to risk adjustment.

    Ann Thorac Surg 1994, 58:1822-6. PubMed Abstract OpenURL

  27. Lewin SA, Dick J, Pond P, Zwarenstein M, Aja G, van Wyk B, Bosch-Capblanch X, Patrick M: Lay health workers in primary and community health care.

    Cochrane Database Syst Rev 2005, 1:CD004015. PubMed Abstract | Publisher Full Text OpenURL

  28. Lavis JN, Oxman AD, Souza NM, Lewin S, Gruen RL, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 9. Assessing the applicability of the findings of a systematic review.

    Health Res Policy Syst 2009, 7(Suppl 1):S9. PubMed Abstract | BioMed Central Full Text OpenURL

  29. Hawkes S, Miller S, Reichenbach L, Nayyar A, Buses K: Antenatal syphilis control: people, programmes, policies and politics.

    Bull World Health Organ 2004, 82:417-23. PubMed Abstract | PubMed Central Full Text OpenURL

  30. Reich M: Political mapping of health policy: a guide for managing the political dimension of health policy. Boston: Harvard School of Public Health; 1994. OpenURL

  31. Balabanova D, McKee M, Koroleva N, Chikovani I, Goguadze K, Kobaladze T, Adeyi O, Robles S: Navigating the health system: diabetes care in Georgia.

    Health Policy Plan 2009, 24:46-54. PubMed Abstract | Publisher Full Text OpenURL

  32. Hopkinson B, Balabanova D, McKee M, Kutzin J: The human perspective on health care reform: coping with diabetes in Kyrgyzstan.

    Int J Health Plann Manage 2004, 19:43-61. PubMed Abstract | Publisher Full Text OpenURL

  33. Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 13. Preparing and using policy briefs to support evidence-informed policymaking.

    Health Res Policy Syst 2009, 7(Suppl 1):S13. PubMed Abstract | BioMed Central Full Text OpenURL

  34. Lavis JN, Wilson M, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 4. Using research evidence to clarify a problem.

    Health Res Policy Syst 2009, 7(Suppl 1):S4. PubMed Abstract | BioMed Central Full Text OpenURL

  35. Mubyazi GM, Gonzalez-Block MA: Research influence on antimalarial drug policy change in Tanzania: case study of replacing chloroquine with sulfadoxine-pyrimethamine as the first-line drug.

    Malar J 2005, 4:51. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  36. Pan American Health Organization (PAHO): Informe Regional de SIREVA II: datos por país y por grupos de edad sobre las características de los aislamientos de Streptococcus pneumoniae, Haemophilus influenzae y Neisseria meningitidis en procesos invasores, 2000-2005. Washington, DC, Pan American Health Organization; 2007. OpenURL

  37. Consumers' Health Forum of Australia: [http://www.chf.org.au/pdfs/rep/rep-195-expectation-of-gp.pdf] webcite

    Consumers' Expectations of General Practice in Australia. Deakin. Consumer's Health Forum; 1999. OpenURL

  38. Cliff J, Lewin S, Woelk G, Fernandes B, Sharp B, Daniels K, Matinhure S, Oxman AD, Lis N: Policy development in malaria vector management in Mozambique, South Africa and Zimbabwe.

    Health Policy and Planning, in press. OpenURL

  39. Mnzava AEP, Ntuli MV, Sharp B: House replastering as a reason to shift from DDT spraying to synthetic pyrethroids.

    South African Medical Journal (Public Health) 1998, 88(8):1024-8. OpenURL

  40. Dick J, Henchie S: A cost analysis of the tuberculosis control programme in Elsies River, Cape Town.

    S Afr Med J 1998, 88:380-3. PubMed Abstract OpenURL

  41. Keating KM, Brewer NT, Gottlieb SL, Liddon N, Ludema C, Smith JS: Potential barriers to HPV vaccine provision among medical practices in an area with high rates of cervical cancer.

    J Adolesc Health 2008, 43:S61-7. PubMed Abstract | Publisher Full Text OpenURL

  42. Ndlovu N, Daswa R: [http://www.idasa.org.za/gbOutputFiles.asp?WriteContent=Y&RID=1543] webcite

    Monitoring AIDS Treatment Rollout in South Africa: Lessons from the Joint Civil Society Monitoring Forum (JCSMF). Budget Brief No. 161. 2006. OpenURL

  43. Giles-Corti B, Donovan RJ: The relative influence of individual, social and physical environment determinants of physical activity.

    Soc Sci Med 2002, 54:1793-812. PubMed Abstract | Publisher Full Text OpenURL

  44. Lavis JN, Boyko J: Evidence Brief: Improving Access to Primary Healthcare in Canada. Hamilton, Canada, McMaster Health Forum; 2009. OpenURL