Email updates

Keep up to date with the latest news and content from HARPS and BioMed Central.

This article is part of the supplement: SUPPORT Tools for evidence-informed health Policymaking (STP)

Open Access Guide

SUPPORT Tools for evidence-informed health Policymaking (STP) 9: Assessing the applicability of the findings of a systematic review

John N Lavis1*, Andrew D Oxman2, Nathan M Souza3, Simon Lewin4, Russell L Gruen5 and Atle Fretheim6

Author Affiliations

1 Centre for Health Economics and Policy Analysis, Department of Clinical Epidemiology and Biostatistics, and Department of Political Science, McMaster University, 1200 Main St. West, HSC-2D3, Hamilton, ON, Canada, L8N 3Z5

2 Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway

3 Health Research Methodology PhD Program and Department of Clinical Epidemiology and Biostatistics, 1200 Main St. West, HSC-2D1 Area, Hamilton, ON, Canada, L8N 3Z5

4 Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway; Health Systems Research Unit, Medical Research Council of South Africa

5 The Alfred Hospital, Monash University, Level 4, 89 Commercial Rd, Melbourne, VIC, Australia 3004

6 Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway; Section for International Health, Institute of General Practice and Community Medicine, Faculty of Medicine, University of Oslo, Norway

For all author emails, please log on.

Health Research Policy and Systems 2009, 7(Suppl 1):S9  doi:10.1186/1478-4505-7-S1-S9

The electronic version of this article is the complete one and can be found online at: http://www.health-policy-systems.com/content/7/S1/S9


Published:16 December 2009

© 2009 Lavis et al; licensee BioMed Central Ltd.

This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers.

Differences between health systems may often result in a policy or programme option that is used in one setting not being feasible or acceptable in another. Or these differences may result in an option not working in the same way in another setting, or even achieving different impacts in another setting. A key challenge that policymakers and those supporting them must face is therefore the need to understand whether research evidence about an option can be applied to their setting. Systematic reviews make this task easier by summarising the evidence from studies conducted in a variety of different settings. Many systematic reviews, however, do not provide adequate descriptions of the features of the actual settings in which the original studies were conducted. In this article, we suggest questions to guide those assessing the applicability of the findings of a systematic review to a specific setting. These are: 1. Were the studies included in a systematic review conducted in the same setting or were the findings consistent across settings or time periods? 2. Are there important differences in on-the-ground realities and constraints that might substantially alter the feasibility and acceptability of an option? 3. Are there important differences in health system arrangements that may mean an option could not work in the same way? 4. Are there important differences in the baseline conditions that might yield different absolute effects even if the relative effectiveness was the same? 5. What insights can be drawn about options, implementation, and monitoring and evaluation? Even if there are reasonable grounds for concluding that the impacts of an option might differ in a specific setting, insights can almost always be drawn from a systematic review about possible options, as well as approaches to the implementation of options and to monitoring and evaluation.

About STP

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. The series is intended to help such people ensure that their decisions are well-informed by the best available research evidence. The SUPPORT tools and the ways in which they can be used are described in more detail in the Introduction to this series [1]. A glossary for the entire series is attached to each article (see Additional File 1). Links to Spanish, Portuguese, French and Chinese translations of this series can be found on the SUPPORT website http://www.support-collaboration.org webcite. Feedback about how to improve the tools in this series is welcome and should be sent to: STP@nokc.no.

Additional file 1. Glossary

Format: DOC Size: 73KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Scenarios

Scenario 1: You are a senior civil servant and will be submitting a brief report to the Minister regarding the evidence to support an option that has been provisionally selected to address a high-priority problem. You are concerned about whether the findings of a relevant high-quality systematic review that was used to make the selection are likely to be applicable to your specific setting, and you want to ensure that this issue has been assessed by your staff.

Scenario 2: You work in the Ministry of Health and are preparing a brief report about an option that is being considered to address a high-priority problem. All that you have been told is that the report should summarise the findings from the most relevant high-quality systematic review and assess the applicability of the findings to your setting.

Scenario 3: You work in an independent unit that supports the Ministry of Health in its use of evidence in policymaking. You are preparing a detailed research report for the Ministry of Health about what is known and not known about an option to address a high-priority problem. You have been told that policymakers have found a particular systematic review to be persuasive but you want guidance on how to assess whether the findings of the review are applicable to your setting.

Background

For policymakers (Scenario 1), this article suggests a number of questions that they might ask their staff to consider when preparing a brief report about a systematic review that could form the basis for selecting an option and communicating the rationale for the selection. For those who support policymakers (Scenarios 2 and 3), this article suggests a number of questions to guide the assessment of the applicability of the findings of a systematic review to a specific setting. This article is the third of four articles in this series about finding and assessing systematic reviews to inform policymaking (see also Articles 7, 8 and 10 [2-4]). Figure 1 outlines the steps involved in finding and assessing systematic reviews to inform policymaking.

thumbnailFigure 1. Finding and assessing systematic reviews to inform policymaking.

Commonalities in human biology mean that a clinical procedure or drug will often work the same way in different patients. However, this is not always the case and questions have thus been developed to help healthcare providers to assess the applicability of research evidence to their patients [5]. Differences between health systems often mean that a policy or programme option being used in one setting may not be feasible or acceptable in another setting. These differences may also mean that an option may not work the same way in another setting or that it may achieve different impacts in another setting [6,7]. For example, the implementation of user fees failed to achieve consistent positive impacts in many sub-Saharan African countries compared to countries in other regions. In part, this was due to a number of contextual considerations, such as people's lack of familiarity with paying for public health services [8]. A key challenge that policymakers and those supporting them must face, therefore, is to determine whether research evidence about the impacts of an option are applicable to their setting.

Systematic reviews make this task easier by offering a single summary of studies from different settings. The word 'settings', in this instance, refers to political/country jurisdictions (e.g. Canada or Cameroon, or their constituent provinces). But settings can also include sectors (e.g. primary care or hospital care), and locales (e.g. urban or rural). Systematic reviews can also assist with the process of making judgements about the applicability of the evidence to specific settings by providing a framework and, when available, research evidence that can be used to identify those factors that are essential for an option to work - or that might modify its impacts. A systematic review of pharmaceutical policies (i.e. referencing pricing, other pricing, and purchasing polices for drugs), for example, provided a summary of the factors that could influence the impacts of reference pricing, as well as the rationale for each factor [9]. These factors included the equivalence of the drug, incentives, exemptions, drug availability, price levels, and electronic information systems.

Unfortunately, many systematic reviews do not do the following:

• Highlight the features of the settings in which studies were conducted, particularly those features that might modify the impacts of an option

• Provide a framework for identifying potential modifying factors, or

• Provide research evidence about modifying factors

In these cases, policy analytic articles or narrative reviews may provide more helpful frameworks that could be used to inform judgements about the applicability of the evidence in a systematic review.

A framework for understanding corruption in the health sector and its determinants, for example, highlighted how health system arrangements (e.g. governance arrangements that limit monopolies, require transparency, and support enforcement) and other factors, influenced 'opportunities' and 'pressure' to abuse, as well as the rationalisation for abuse, and how this in turn influenced the abuse of power for private gain [10]. But, ideally, a systematic review about the impacts of anti-corruption efforts would also have described the relevant health system arrangements in the settings where the studies were conducted. Those features that might influence opportunities and pressure to abuse and the rationalisation of such behaviour, would be of particular interest as they would allow the reader to link the determinants identified by the framework with the findings presented in the review.

Applicability considerations are equally, if not more important, for other types of systematic reviews. Reviews of administrative database studies and of community surveys can help to place problems in comparative perspective, for example, and reviews of observational studies can help to characterise an option's likely harms. Reviews of qualitative studies can assist in understanding the meanings that individuals or groups assign to particular problems, how and why particular options work, and stakeholder views about experiences with particular options. What follows in this article, however, is more focused on systematic reviews about the impacts of an option. That said, this may provide some help in assessing the local applicability of the findings of reviews of observational studies about harms, be supported by reviews about how and why particular options work, and give some insights into how to approach local applicability assessments of other types of reviews.

Questions to consider

The following five questions can guide how to assess whether the findings from a systematic review are applicable to a specific setting.

1. Were the studies included in a systematic review conducted in the same setting or were the findings consistent across settings or time periods?

2. Are there important differences in on-the-ground realities and constraints that might substantially alter the feasibility and acceptability of an option?

3. Are there important differences in health system arrangements that may mean an option could not work in the same way?

4. Are there important differences in the baseline conditions that might yield different absolute effects even if the relative effectiveness was the same?

5. What insights can be drawn about options, implementation, and monitoring and evaluation?

1. Were the studies included in a systematic review conducted in the same setting or were the findings consistent across settings or time periods?

If the studies included in a systematic review were conducted in the same setting where policymakers are based, or else in very similar settings, there may be little reason to be concerned about the applicability of the findings. Similarly, if the findings have been shown to be consistent across settings or time periods then similar impacts might be expected. On-the-ground realities and constraints, health system arrangements and baseline conditions, are likely to differ across settings and change over time, so consistent findings in these circumstances are likely to mean the findings are broadly applicable. (These three issues are the focus of the next three questions that follow in this section.)

The following information in systematic reviews can be used by policymakers to inform judgements related to such issues:

• Information about the settings of studies and specifications regarding the time periods over which the studies were conducted. This can typically be found in a section of the review entitled 'Characteristics of included studies' (or similar)

• Information about the consistency of findings can typically be found in the review abstract or in its 'Results' section

When information about settings and time periods is lacking in a systematic review, policymakers and those who support them could contact the authors of the review to see if they have this information and did identify key local applicability considerations. If this contact yields little of value, they could then retrieve the original studies to locate this information if the issue is of sufficiently high priority, and if resources and time allow. A potential benefit of the direct contact with review authors is that it may encourage them to give attention to information needed for local applicability assessments and considerations in future reviews.

Research comparing mortality rates in not-for-profit hospitals with mortality rates in for-profit hospitals provides an example of how such data can be used [11]. This research had been conducted over several decades in the United States during which the health system had changed dramatically. The research demonstrated remarkable consistency over time in the significant survival advantage of being treated in not-for-profit hospitals. Based on these findings, a policymaker from Canada might then conclude that a similarly consistent finding would be seen in a Canadian setting. And this conclusion might lead them to avoid the introduction of for-profit hospitals into the current system which consists only of not-for-profit hospitals (or at least to avoid using health benefits as a justification for doing so).

2. Are there important differences in on-the-ground realities and constraints that might substantially alter the feasibility and acceptability of an option?

If the studies included in a systematic review were conducted in settings with largely similar resource and capacity constraints to the setting where the findings may be applied, and largely similar perspectives and political influence amongst health system stakeholders, policymakers might reasonably expect that an option would be both feasible and acceptable in their own setting. However, policymakers will rarely be able to find information about resource and capacity constraints and stakeholder influence in a systematic review. Instead, they will find a description of the option that was studied. Typically they will be sufficiently familiar with the resources, capacity, and stakeholder influence in their own setting to enable them to judge the feasibility and acceptability of the option.

Policymakers in a setting with very significant resource and capacity constraints will have to think twice about the feasibility of an option [12]. Some settings, for example, may face a shortage of nurse practitioners and therefore any option requiring a significant role for this category of healthcare provider might not be feasible in the short-term [13]. Similarly, some settings have such limited financial resources that an option shown to have significant impacts, such as artemisinin-based combination therapies (ACT) to treat malaria, might not be considered feasible on a large scale without significant donor support [14]. Some health systems may be too overstretched to accommodate an increase in demand that may accompany the introduction of conditional cash transfers (i.e. the provision of money to households on the condition that they comply with certain health and healthcare-seeking behaviours) [14,15]. Or settings may lack the capacity within governments or among managers, healthcare providers and consumers (i.e. healthcare recipients and citizens) to support the widespread use of a particular option. Audit and feedback (i.e. the provision of healthcare providers with data about their performance), for example, might not be feasible in settings where routinely collected data are unreliable.

In a setting in which stakeholders are opposed to an option and have significant influence on practice and policy, policymakers may have to assess the likely acceptability of an option particularly carefully. Healthcare provider associations, such as nursing associations, for example, may resist the introduction or expansion of a lay health worker programme if they perceive that the income or status of nurses might be affected [14,16]. Civil society organisations, too, may actively oppose changes that would reduce prescription drug use among consumers, particularly for life-sustaining drugs, and drugs that are important in treating chronic conditions [14,17]. Such changes could include the introduction of caps (i.e. consumers are reimbursed up to a set maximum number of prescriptions), co-insurance (i.e. consumers pay a percentage of the price of the prescription drug), and co-payments (i.e. consumers pay a fixed amount per prescription drug).

Significantly, many on-the-ground realities and constraints can be addressed over time. Nurse practitioner training programmes, for example, can be scaled up and donors can subsidise the cost of an expensive drug like ACT. Similarly, governments can improve the quality of routinely collected data, and healthcare provider associations and civil society organisations can become engaged in a series of negotiations or dialogues.

3. Are there important differences in health system arrangements that may mean an option could not work in the same way?

If the studies included in a systematic review were conducted in settings with largely similar health system arrangements to the setting where the findings may be applied, particularly those that might substantially alter the potential impacts of an option, policymakers might reasonably expect similar relative effectiveness in their setting. Deciding whether health system arrangements might alter an option's impacts requires an understanding of how and why an option might work. Within a systematic review, policymakers may find both a framework and research evidence that will identify those factors essential for an option to work - or that might modify its impacts. Policymakers may also find a summary of those features of the settings in which studies were conducted that might modify the impacts of an option.

If a systematic review does not provide the information necessary to determine whether particular health system arrangements might result in an option not working in the same way, policymakers could look for:

• Policy analytic articles or narrative reviews incorporating helpful frameworks that could be used to identify factors that might modify the impacts of an option, and

• Detailed descriptions of the health system arrangements, specifically those that might substantially alter the potential impacts of an option, in the settings where the studies were conducted

The European Observatory on Health Systems and Policies publishes, and periodically updates, profiles of the health systems of a large number of middle- and high-income countries. These 'Health in Transition' (HiT) profiles can be found online http://www.euro.who.int/observatory/hits/20020525_1 webcite and downloaded free of charge. The Health Policy Monitor provides a searchable online database of key health system features in some of the same countries http://www.hpm.org/en/Search_for_Reforms/Search.html webcite. Many World Health Organization regional offices also provide profiles of the health systems of countries in their region (e.g. http://www.searo.who.int/EN/Section313/Section1515_6038.htm webcite).

Policymakers in a setting with very different health system arrangements, specifically arrangements that appear significant in determining whether an option will function in the same way, should be cautious about assuming that comparable relative effectiveness could be achieved. For example, in a review of reference drug pricing [9], six of the ten studies were conducted amongst older people/pensioners in British Columbia, Canada. Policymakers in other settings may well conclude that they will not be able to achieve comparable impacts to those seen in the Canadian example if they have any of the following issues within their own health system arrangements:

• Inadequate incentives for consumers, healthcare providers, pharmacists and pharmaceutical companies to comply with the reference drug price system, and

• An electronic processing system that lacks the capacity to realise the low administration costs associated with identifying, prescribing and dispensing the reference drugs and with handling exemptions

Similarly, other pricing policies examined in competitive pharmaceutical markets may yield a different relative effectiveness in markets with monopolies.

Unlike the possibility of associated change in on-the-ground realities that we discussed in Question 2 earlier, there is less chance that health system arrangements could be modified. Health system arrangements are difficult to change and typically the rationale underpinning a change would need to be more compelling than only the possibility that it would enhance the impact of a single option.

4. Are there important differences in the baseline conditions that might yield different absolute effects even if the relative effectiveness was the same?

If the studies included in a systematic review were conducted in settings with largely similar baseline conditions to those in which the findings may be applied, such as in terms of a programme's or policy's coverage of the population, policymakers might reasonably expect similar absolute effects in their setting (provided the answer they gave to Question 3 above led them to expect similar relative effectiveness). Policymakers will often be able to find information about baseline conditions within systematic reviews in a section titled 'Characteristics of included studies'. Alternatively, they may have to retrieve the original studies included in the review in the hope that baseline conditions were better described in them. Policymakers will typically be able to find local evidence about baseline conditions in their own setting. (Article 11 in this series addresses how to find and use local evidence [18].)

Policymakers in a setting with different baseline conditions may expect different absolute impacts. The absolute impact of audit and feedback, for example, is likely to be larger than in instances where the baseline compliance to recommended practice is low [19]. Similarly, the absolute impact of a pay-for-performance initiative may be larger in low- and middle-income countries (where small financial incentives may be larger relative to wages) than in high-income countries [20].

This question highlighting the link between baseline conditions and absolute effects is also highly relevant in clinical settings in which the relative effectiveness of a clinical intervention is often the same across patients but where patients' baseline risks may vary quite dramatically [21,22]. The question is also highly relevant in public health settings where immunisation programmes, for example, might be introduced in countries with very different baseline conditions. Article 16 of this series discusses the use of balance sheets to summarise important impacts and provides further detail about relative effectiveness and absolute impacts [23].

5. What insights can be drawn about options, implementation, and monitoring and evaluation?

Even if the findings from systematic reviews are not directly applicable to a given setting, important lessons can still be drawn. Policymakers may be provided with an idea for an option that they might otherwise not have considered. They may also gain insight into how options have been implemented in other settings. And they may be able to draw directly on the systematic review itself in developing a monitoring and evaluation plan. Policymakers may learn, for example, about a new approach to supporting team-based care, the importance of engaging both mid-level managers and front-line nurses in the implementation of an option, and what types of outputs and outcomes they should track as they monitor and evaluate the implementation of a selected option.

Table 1 and Table 2 provide examples of an assessment of the applicability of a systematic review.

Table 1. An assessment of the local applicability of a systematic review about home care (from the perspective of a Canadian policymaker)

Table 2. An assessment of the local applicability of a systematic review on lay health worker interventions (from the perspective of a South African policymaker)

Conclusion

Assessments of the applicability of the findings of a systematic review can take a lot of time to do well. Such assessments are critical, however, when an option is being proposed on the basis of a relevant high-quality systematic review. Policymakers and other stakeholders need to know whether they can expect similar findings in their own settings. Unlike an assessment of the quality of a review, which can often be delegated to researchers, a local applicability assessment must be done by individuals with a very good understanding of on-the-ground realities and constraints, health system arrangements, and the baseline conditions in the specific setting. The assessment of local applicability is a domain in which policymakers and those who support them need to be actively engaged.

Resources

Useful documents and further reading

- Dans AL, Dans LF, Guyatt GH: Applying results to individual patients. In Users' Guides to the Medical Literature. A Manual for Evidence-Based Clinical Practice. Edited by Guyatt GH, Rennie D, Meade MO, Cook DJ. New York, USA: McGraw Hill; 2008.

- Haynes RB: Can it work? Does it work? Is it worth it?: The testing of healthcare interventions is evolving, http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1116525/ webcite. BMJ 1999, 1999: 652-653.

Links to websites

- SUPPORT Collaboration: http://www.support-collaboration.org webcite - Example of a source of policymaker-friendly summaries of systematic reviews that provides an assessment of the applicability of the findings of each review (in this case to low- and middle-income countries), and that highlights the factors that policymakers need to bear in mind when assessing the applicability of the findings to their own setting.

- European Observatory on Health Systems and Policies: http://www.euro.who.int/observatory/hits/20020525_1 webcite - Example of a source of (Health in Transition) profiles of the health systems of a large number of middle- and high-income countries.

- Health Policy Monitor: http://www.hpm.org/en/Search_for_Reforms/Search.html webcite - Searchable online database of key health system features in a number of middle- and high-income countries.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

JNL prepared the first draft of this article. ADO, NMS, SL, RLG and AF contributed to drafting and revising it.

Acknowledgements

Please see the Introduction to this series for acknowledgements of funders and contributors. In addition, we would like to acknowledge Sara Bennett, Mike Kelly, and staff in the Ontario Ministry of Health and Long-Term Care (MoHLTC) Planning Unit for helpful comments on an earlier version of this Article.

This article has been published as part of Health Research Policy and Systems Volume 7 Supplement 1, 2009: SUPPORT Tools for evidence-informed health Policymaking (STP). The full contents of the supplement are available online at http://www.health-policy-systems.com/content/7/S1 webcite.

References

  1. Lavis JN, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). Introduction.

    Health Res Policy Syst 2009, 7(Suppl 1):I1. PubMed Abstract | BioMed Central Full Text OpenURL

  2. Lavis JN, Oxman AD, Grimshaw J, Johansen M, Boyko JA, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 7. Finding systematic reviews.

    Health Res Policy Syst 2009, 7(Suppl 1):S7. PubMed Abstract | BioMed Central Full Text OpenURL

  3. Lewin S, Oxman AD, Lavis JN, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 8. Deciding how much confidence to place in a systematic review.

    Health Res Policy Syst 2009, 7(Suppl 1):S8. PubMed Abstract | BioMed Central Full Text OpenURL

  4. Oxman AD, Lavis JN, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 10. Taking equity into consideration when assessing the findings of a systematic review.

    Health Res Policy Syst 2009, 7(Suppl 1):S10. PubMed Abstract | BioMed Central Full Text OpenURL

  5. Dans AL, Dans LF, Guyatt GH: Applying results to individual patients. In Users' Guides to the Medical Literature. A Manual for Evidence-Based Clinical Practice. Edited by Guyatt GH, Rennie D, Meade MO, Cook DJ. New York: McGraw Hill; 2008. OpenURL

  6. Gruen RL, Morris PS, McDonald EL, Bailie RS: Making systematic reviews more useful for policy-makers.

    Bulletin of the World Health Organization 2005, 83:480-1. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  7. Lavis JN, Posada FB, Haines A, Osei E: Use of research to inform public policymaking.

    Lancet 2004, 364:1615-21. PubMed Abstract | Publisher Full Text OpenURL

  8. Gilson L: The lessons of user fee experience in Africa.

    Health Policy Plan 1997, 12:273-85. PubMed Abstract | Publisher Full Text OpenURL

  9. Aaserud M, Dahlgren AT, Kosters JP, Oxman AD, Ramsay C, Sturm H: Pharmaceutical policies: Effects of reference pricing, other pricing, and purchasing policies.

    Cochrane Database Syst Rev 2006, (2):CD005979. PubMed Abstract | Publisher Full Text OpenURL

  10. Vian T: Review of corruption in the health sector: Theory, methods and interventions.

    Health Policy and Planning 2008, 23:83-94. PubMed Abstract | Publisher Full Text OpenURL

  11. Devereaux PJ, Choi PT, Lacchetti C, Weaver B, Schunemann HJ, Haines T, Lavis JN, Grant BJ, Haslam DR, Bhandari M: A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals.

    CMAJ 2002, 166:1399-406. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  12. Gericke CA, Kurowski C, Ranson MK, Mills A: Intervention complexity: A conceptual framework to inform priority-setting in health.

    Bulletin of the World Health Organization 2005, 83:285-93. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  13. Horrocks S, Anderson E, Salisbury C: Systematic review of whether nurse practitioners working in primary care can provide equivalent care to doctors.

    BMJ 2002, 324:819-23. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  14. World Health Organization (WHO): Guidelines for the Treatment of Malaria. Geneva, World Health Organization; 2006. OpenURL

  15. Lagarde M, Haines A, Palmer N: Conditional cash transfers for improving uptake of health interventions in low- and middle-income countries: a systematic review.

    JAMA 2007, 298:1900-10. PubMed Abstract | Publisher Full Text OpenURL

  16. Lewin SA, Dick J, Pond P, Zwarenstein M, Aja G, van Wyk B, Bosch-Capblanch X, Patrick M: Lay health workers in primary and community health care.

    Cochrane Database Syst Rev 2005, 1:CD004015. PubMed Abstract | Publisher Full Text OpenURL

  17. Austvoll-Dahlgren A, Aaserud M, Vist G, Ramsay C, Oxman AD, Sturm H, Kosters JP, Vernby A: Pharmaceutical policies: effects of cap and co-payment on rational drug use.

    Cochrane Database Syst Rev 2008, 1:CD007017. PubMed Abstract | Publisher Full Text OpenURL

  18. Lewin S, Oxman AD, Lavis JN, Fretheim A, García Martí S, Munabi-Babigumira S: SUPPORT Tools for evidence-informed health Policymaking (STP). 11. Finding and using research evidence about local conditions.

    Health Res Policy Syst 2009, 7(Suppl 1):S11. PubMed Abstract | BioMed Central Full Text OpenURL

  19. Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD: Audit and feedback: effects on professional practice and health care outcomes.

    Cochrane Database Syst Rev 2006, 2:CD000259. PubMed Abstract | Publisher Full Text OpenURL

  20. Oxman AD, Fretheim A: [http://www.kunnskapssenteret.no/binary?download=true&id=5800] webcite

    An overview of research on the effects of results-based financing. Report Nr 16-2008. Oslo, Nasjonalt kunnskapssenter for helsetjenesten; 2008. OpenURL

  21. Deeks JJ: Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes.

    Statistics in Medicine 2002, 21:1575-600. PubMed Abstract | Publisher Full Text OpenURL

  22. Engels EA, Schmid CH, Terrin N, Olkin I, Lau J: Heterogeneity and statistical significance in meta-analysis: An empirical study of 125 meta-analyses.

    Stat Med. 2000, 19(13):1707-28. PubMed Abstract | Publisher Full Text OpenURL

  23. Oxman AD, Lavis JN, Fretheim A, Lewin S: SUPPORT Tools for evidence-informed health Policymaking (STP). 16. Using research evidence in balancing the pros and cons of policies.

    Health Res Policy Syst 2009, 7(Suppl 1):S16. PubMed Abstract | BioMed Central Full Text OpenURL

  24. Shepperd S, Iliffe S: Hospital at home versus in-patient hospital care.

    Cochrane Database Syst Rev 2005, (3):CD000356. PubMed Abstract | Publisher Full Text OpenURL

  25. Shepperd S, Doll H, Angus RM, Clarke MJ, Iliffe S, Kalra L, Ricauda NA, Wilson AD: Admission avoidance hospital at home.

    Cochrane Database Syst Rev 2008, 4:CD007491. PubMed Abstract | Publisher Full Text OpenURL

  26. Flottorp S: [http://www.iecs.org.ar/support/administracion/files/20080909113811_13.pdf] webcite

    Do lay health workers in primary and community health care improve maternal and child health? A SUPPORT Summary of a systematic review . 2009. OpenURL

  27. Lewin S, Munabi-Babigumira S, Bosch-Capblanch X, Aja G, van Wyk B, Glenton C, Scheel IB, Zwarenstein M, Daniels K: [http://www.who.int/rpc/meetings/LHW_review2.pdf] webcite

    Lay health workers in primary and community health care: A systematic review of trials. A policy brief prepared for the International Dialogue on Evidence-Informed Action to Achieve Health Goals in Developing Countries (IDEAHealth). Geneva, World Health Organization; 2006. OpenURL