Email updates

Keep up to date with the latest news and content from HARPS and BioMed Central.

Open Access Highly Accessed Review

Improving the use of research evidence in guideline development: 1. Guidelines for guidelines

Holger J Schünemann1*, Atle Fretheim2 and Andrew D Oxman2

Author Affiliations

1 INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy

2 Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway

For all author emails, please log on.

Health Research Policy and Systems 2006, 4:13  doi:10.1186/1478-4505-4-13


The electronic version of this article is the complete one and can be found online at: http://www.health-policy-systems.com/content/4/1/13


Received:7 April 2006
Accepted:21 November 2006
Published:21 November 2006

© 2006 Schünemann et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the first of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.

Objectives

We reviewed the literature on guidelines for the development of guidelines.

Methods

We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.

Key questions and answers

We found no experimental research that compared different formats of guidelines for guidelines or studies that compared different components of guidelines for guidelines. However, there are many examples, surveys and other observational studies that compared the impact of different guideline development documents on guideline quality.

What have other organizations done to develop guidelines for guidelines from which WHO can learn?

• Establish a credible, independent committee that evaluates existing methods for developing guidelines or that updates existing ones.

• Obtain feedback and approval from various stakeholders during the development process of guidelines for guidelines.

• Develop a detailed source document (manual) that guideline developers can use as reference material.

What should be the key components of WHO guidelines for guidelines?

• Guidelines for guidelines should include information and instructions about the following components: 1) Priority setting; 2) Group composition and consultations; 3) Declaration and avoidance of conflicts of interest; 4) Group processes; 5) Identification of important outcomes; 6) Explicit definition of the questions and eligibility criteria ; 7) Type of study designs for different questions; 8) Identification of evidence; 9) Synthesis and presentation of evidence; 10) Specification and integration of values; 11) Making judgments about desirable and undesirable effects; 12) Taking account of equity; 13) Grading evidence and recommendations; 14) Taking account of costs; 15) Adaptation, applicability, transferability of guidelines; 16) Structure of reports; 17) Methods of peer review; 18) Planned methods of dissemination & implementation; 19) Evaluation of the guidelines.

What have other organizations done to implement guidelines for guidelines from which WHO can learn?

• Obtain buy-in from regions and country level representatives for guidelines for guidelines before dissemination of a revised version.

• Disseminate the guidelines for guidelines widely and make them available (e.g. on the Internet).

• Develop examples of guidelines that guideline developers can use as models when applying the guidelines for guidelines.

• Ensure training sessions for those responsible for developing guidelines.

• Continue to monitor the methodological literature on guideline development.

Background

The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the first of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.

The term guideline can be defined as "a rule or principle that provides guidance to appropriate behaviour" [1]. The Institutes of Medicine define clinical practice guidelines as "systematically developed statements to assist practitioner and patient decision about appropriate health care for specific clinical circumstances". The term "guidelines" in this document should be seen in the broad sense referring to any guideline or recommendation related to healthcare that is relevant to the mission of the WHO, including public health and health policy recommendations. A plethora of guidelines for clinical practice guidelines exist from various organizations, including national and governmental agencies and medical specialty societies. There are fewer guidelines for developing public health and health policy recommendations. We will use the term "guidelines for guidelines" as "guidelines for the development of guidelines and recommendations".

Guidelines for guidelines are important because of reports indicating that the lack of standardized guideline development leads to widely varying recommendations [2]. In this paper we addressed the following questions:

• What have other organizations done to develop guidelines for guidelines from which WHO can learn?

• What should be the key components of WHO guidelines for guidelines?

• What have other organizations done to implement guidelines for guidelines from which WHO can learn?

What WHO is doing now

An inter-cluster initiative (Guideline Development Group) led by the Evidence and Information for Policy (EIP) cluster produced the "Guidelines for WHO Guidelines" (GWG) as the recommended approach to development of WHO guidelines [3]. The process for developing the WHO document included drafting of the GWG by one group member before revision and approval by the committee. Following approval by the group, this document was reviewed and approved during a cabinet meeting before distribution as a technical cluster note to all WHO members.

The GWG (version March 10, 2003) included the following general proposals for process (see GWG "WHO documents that guide the development, dissemination and implementation of recommendations by WHO" section 5):

"b) choice of [guideline] topics; c) synthesis of the evidence; d) formulation of recommendations; e) dissemination of guidelines". The GWG makes special reference to the National Health and Medical Research Council (NHMRC) of Australia guidelines for guidelines. To accomplish proper guideline development the GWG recommends partnerships within and outside WHO according to a defined set of rules. Specific functions and composition of guideline groups are also described. The GWG also includes advice for the operationalisation of the process (section 6 GWG): a) Selection of partners; b) Organization of guideline groups; c) Process of developing guidelines; d) Guiding values. The committee also produced a self assessment checklist to ensure a consistent level of quality in the guidelines.

Although comprehensive in the coverage of topics, due to brevity most sections of the GWG could not provide the same level of detailed instructions for guideline groups to follow that other organizations provide. Moreover, it is not entirely clear to what extent WHO guideline developers adhere to the GWG, but it appears that few departments have used the GWG [4]. In part this may be a result of a lack of a more detailed handbook that WHO guidelines development committees could follow, although there are a number of other possible explanations.

What other organisations are doing?

The use and quality of guidelines for guidelines varies across organizations that develop guidelines. There is not an accepted international standard for guideline development. However, there are several specific and detailed examples of methods adopted by other organizations. Some of these are exemplary because they give detailed guidance and resulted from a thorough process. For example, the National Institute for Health and Clinical Excellence (NICE) and the Scottish Intercollegiate Guideline Network (SIGN), both large government agencies that develop guidelines, have produced comprehensive handbooks that provide guidance for its guideline developers [5,6]. A number of professional organizations have also developed detailed guidance documents that advise their guideline developers about methods including the development of templates [7-10].

We describe the single steps of what other organizations do in regards to guideline development and on what grounds they do it in other articles in this series (see [11] for a list of articles). The steps go from setting priorities for guideline topics to implementation of the guidelines. In addition, literature has emerged from independent groups, such as the Conference on Guideline Standardization [12], that address the critical appraisal of guidelines and suggest the need for guidance for each of these steps of guideline development [13]. For example, one tool (the AGREE instrument) has demonstrated its sensitivity to differentiate higher quality guidelines that followed technical documentation from those of lower quality [14].

Methods

The methods used to prepare this review are described in the introduction to this series [11]. Briefly, the key questions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). As a result of prior work in the area of guideline methodology we had knowledge of existing guidelines for guidelines by organisations such as NICE, SIGN, the US Preventive Services Task Force (USPSTF), the New Zealand Guideline Group and the Australian NHRMC as well as professional societies such as the American College of Chest Physicians (ACCP) and the American Thoracic Society (ATS). We attempted to search PubMed, but were unable to devise a search strategy that was both sensitive and reasonably specific.

Given time constraints we avoided duplication with work of others and focused on sources that had systematically compiled relevant literature. We searched databases maintained by the Agency for Healthcare Research and Quality (AHRQ [15]), the Guidelines International Network (GIN [16]), information obtained from prominent organizations and our own files. The AHRQ database (guideline development methodology and guideline structure) is a comprehensive database that included 1205 references to both journal but also non-journal sources. The GIN database included 104 references. While there was overlap we reviewed these citations in detail and evaluated each of these references for relevance. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.

Findings

We found no experimental research that compared different formats of guidelines for guidelines or studies that compared different components of guidelines for guidelines. However, there are many examples, surveys and other observational studies of the impact of guideline development documents on guideline quality.

What have other organizations done to develop guidelines for guidelines that WHO can learn from?

Many large organizations that claim to develop evidence based guidelines have produced accessible, transparent and detailed guidelines for guidelines. To make a guideline for guideline credible and acceptable, individuals with expertise in methodology, process and implementation of guidelines were involved in developing a guideline for guideline document. For example, NICE involved not only various internal groups (the national collaborating centres that develop guidelines, NICE patient involvement units, etc.), but also external advisors, including individual academics and governmental institutions (e.g. SIGN) [5]. SIGN and the RAND corporation published a detailed description of the processes involved in producing guidelines for guidelines involving various stakeholders [6,17].

Other guideline developers carefully select the panels that produce guidelines for guidelines ensuring that methodologists and clinicians as well as representatives of the organization are involved. Most specialty societies have included experts and authorities in the relevant fields. While this bears the risk of involving individuals with less methodological and, therefore, relevant training for guideline development, it ensures that individuals who are knowledgeable about the relevant clinical aspects, including ongoing research, are represented and may supports buy-in by users. Aspects focusing on group processes and selection including patient representation are described in other articles in this series [18,19]. Most organizations obtain approval of the final document by a board or other governing body. While no experimental research indicates that providing a source document (e.g. a handbook) for guideline developers improves the quality of guidelines, observational studies suggest that organizations publishing their guidelines for guidelines in the form of reference material produce more methodologically sound guidelines [20].

What should be the key components of WHO guidelines for guidelines?

We have identified 19 components that are already or should be included in the GWG and that should be described in detail in a handbook or manual for WHO guideline developers. Other reviews in this series will describe these components in greater detail. We list in parenthesis the review that describes the component in more detail and the section of the GWG that has mentioned the component. The sections in the GWG cited below often consist of a single sentence. The components are:

1) Priority setting ([21] and GWG 5b "Choice of topics for development of WHO guidelines")

2) Group composition (and consultations) ([22] and GWG 6A "Organization of guideline groups")

3) Declaration and avoidance of conflicts of interest ([23] and GWG 6B Note 1, annex A)

4) Group processes ([24] and GWG 6C3 "Process of developing guidelines")

5) Identification of important outcomes including cost ([25], not addressed in GWG)

6) Explicit definition of the question and eligibility criteria ([26,27], not addressed in GWG)

7) Type of study designs for different types of questions ([27], not addressed in GWG)

8) Identification of evidence ([27], GWG 6C2 "Undertake a systematic review")

9) Synthesis and presentation of evidence ([28] and GWG 5C "Synthesizing the evidence")

10) Specification and integration of values ([29] and GWG 6D "Guiding values")

11) Making judgments about desirable and undesirable effects ([29] and [30] and GWG 5d "Making recommendations")

12) Taking account of equity ([31], not addressed in GWG)

13) Grading evidence and recommendations ([30] and GWG Annex B)

14) Taking account of costs ([32] and GWG 5d "Making recommendations")

15) Applicability, transferability and adaptation of guidelines ([33] and GWG 5a "A 3 stage process")

16) Structure of reports ([34] and GWG 6C)

17) Methods of peer review ([20] and [32]not addressed in GWG)

18) Planned methods of dissemination & implementation ([35] and GWG 5e "Dissemination of guidelines")

19) Evaluation of the impact of the guideline ([36] and GWG 6C6)

What have other organizations done to implement guidelines for guidelines from which WHO can learn?

Other prominent guideline developers, such as NICE and SIGN, have ensured that those stakeholders who will become involved in guideline development also take part in the development of the guidelines for guideline. Similarly, obtaining buy-in from regions and country level representatives for GWG before agreeing on and disseminating a revised version is likely to be important. Once WHO reaches agreement on a revised version of the GWG, it should be widely disseminated and made easily available (e.g. on the Internet).

Examples and worksheets should be provided to facilitate implementation of the GWG. In addition, WHO should ensure training sessions for those responsible for developing guidelines. In their survey of 18 prominent international clinical guideline developers, Burgers and colleagues found that almost all guideline programs offer (in some organizations mandatory) training sessions to guideline developers [37]. SIGN, for example, offers a specific software program to guideline panel members and helps them with identifying specific learning needs [6]. SIGN also electronically records the amount of training of individuals who contribute to the guidelines.

The GWG should not be a static document. NICE, for example, has outlined the process for updating its guidelines for guidelines. This specifies that the formal process for updating its manual will begin three years after publication of the original manual. Interim updates may be completed to accommodate small changes outside of the regular renewal process. NICE specifies four criteria that must all be fulfilled to qualify for a minor update: a fundamental stage in the guideline for guideline process is neither added nor removed, a fundamental methods technique or step is neither added nor removed, one or more stakeholders will not obviously be disadvantaged, and the efficiency, clarity or fairness of the process or methodology will be improved. To develop revisions of the GWG, WHO should monitor the methodological literature on guideline development and review updates of other organizations' guidelines for guidelines.

Discussion

The studies we identified and practical experience suggest that guidelines for guidelines facilitate the development of guidelines. Our review is limited in that it is not a systematic review and is based on our own judgments. However, our review has identified practical advice and components that a handbook or manual that accompany the GWG should include.

Although WHO's leadership has endorsed the GWG, a detailed handbook does not exist. Moreover, implementation of the GWG in WHO guideline programs appears to be very limited [4]. Potential reasons for this shortcoming are discussed elsewhere [4]. Key explanations include a lack of resources, technical (methodological) capacity, knowledge about the GWG, and a tradition of using non-systematic, expert opinion-based approaches [4]; as well as a lack of training and a lack of a more detailed manual.

Other organizations have invested substantial resources into guideline development, including resources to develop and implement guidelines for guidelines. Because WHO has limited resources and because well described processes used by other organizations already exist, WHO can build on existing high quality guidelines for guidelines. As we discuss in another article in this series, WHO should also consider adapting guidelines developed by other organizations, if high quality guidelines already exist [33]. In addition, WHO should consider establishing collaborations with other guideline developers to avoid unnecessary duplication of efforts and use of resources.

Further work

A systematic review of guidelines for guidelines is unlikely to yield empirical information beyond what we have found in this review, but could provide useful information about what other organisations are doing with respect to key steps in the guideline development process. We do not consider such a review to be a priority for WHO. Similar information has been obtained through surveys [20, 38, 39]. More information about specific questions regarding, for example, processes that are used for updating, implementing and evaluating guidelines for guidelines, is more likely to come from further surveys than from a systematic review.

WHO should develop a handbook or manual that provides detailed information and examples for its guideline developers. This handbook should build on existing work, but will require time and resources. Nonetheless, this is likely necessary to improve the quality of WHO guideline development.

Efforts are needed that ensure guideline developers begin speaking the same "guideline" language and improve the standardisation of the guideline development processes used by WHO. This standardisation would help facilitate the production of guidelines that can easily be adapted to different contexts, and thus reduce global resources spent on guideline development, particularly for low and middle-income countries [33]. WHO should participate in international efforts aimed at improving guidelines for guidelines, and should aim to take a leading role in these efforts in the future.

Competing interests

ADO and AF work for the Norwegian Knowledge Centre forthe Health Services, an agency funded by the Norwegian government that produces systematic reviews and health technology assessments. All three authors are contributors to the Cochrane Collaboration. ADO and HJS are members of the GRADE Working Group. HJS is documents editor and chair of the documents development and implementation committee for the American Thoracic Society and senior editor of the American College of Chest Physicians' Antithrombotic and Thrombolytic Therapy Guidelines.

Authors' contributions

HJS prepared the first draft of this article. AF and ADO contributed to drafting and revising it. All authors have read and approved of the final version of this manuscript.

Acknowledgements

We gratefully acknowledge WHO and the Norwegian Knowledge Centre forthe Health Services for financial support, the other authors of these reviews for their contributions, the other members of the ACHR SURE, and all of those who have provided us with feedback on earlier versions of this paper. The authors thank Mr. John Fox for administrative assistance.

References

  1. www.wordreference.com: Word reference dictionary. [http://www.wordreference.com/definition] webcite

  2. Beck C, Cody M, Souder E, Zhang M, Small GW: Dementia diagnostic guidelines: methodologies, results, and implementation costs.

    J Am Geriatr Soc 2000, 48(10):1195-1203. PubMed Abstract OpenURL

  3. Global Programme on Evidence for Health Policy. Guidelines for WHO Guidelines. EIP/GPE/EQC/2003.1 Geneva , World Health Organization; 2003.

  4. Panisset U: A Review of WHO recommendations published in 2005.

    Health Res Policy Syst 2006. OpenURL

  5. National Instutites for Health and Clinical Excellence [http://www.nice.org.uk/pdf/GDM_Allchapters_0305.pdf] webcite

  6. Scottish Intercollegiate Guideline Network. SIGN 50: A guideline developers' handbook. Section 6. [http://www.sign.ac.uk/guidelines/fulltext/50/section6.html] webcite

    (accessed on March 17, 2006).

  7. Beyea SC: Clinical practice guidelines--an organizational effort.

    Aorn J 2000, 71(4):852, 855-8. OpenURL

  8. Jones RH, Ritchie JL, Fleming BB, Hammermeister KE, Leape LL: 28th Bethesda Conference. Task Force 1: Clinical practice guideline development, dissemination and computerization.

    J Am Coll Cardiol 1997, 29(6):1133-1141. PubMed Abstract | Publisher Full Text OpenURL

  9. ATS: Attributes of ATS documents that guide clinical practice. Recommendations of the ATS Clinical Practice Committee.

    American Journal of Respiratory & Critical Care Medicine 1997, 156((6)):2015-2025. OpenURL

  10. Schünemann HJ, Munger H, Brower S, O'Donnell M, Crowther M, Cook D, Guyatt G: Methodology for guideline development for the Seventh American College of Chest Physicians Conference on Antithrombotic and Thrombolytic Therapy: the Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy.

    Chest 2004, 126:174S-178S. PubMed Abstract | Publisher Full Text OpenURL

  11. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: Introduction.

    Health Res Policy Syst 2006. OpenURL

  12. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, Deshpande AM: Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization.

    Ann Intern Med 2003, 139(6):493-498. PubMed Abstract | Publisher Full Text OpenURL

  13. Guideline development in Europe. An international comparison

    Int J Technol Assess Health Care 2000, 16(4):1039-1049. PubMed Abstract | Publisher Full Text OpenURL

  14. Collaboration AGREE: Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project.

    Qual Saf Health Care 2003, 12(1):18-23. PubMed Abstract | Publisher Full Text OpenURL

  15. Agency for Healthcare Research and Quality [http://www.guidelines.gov] webcite

  16. Guidelines International Network [http://www.g-i-n.net] webcite

  17. Woolf SH: Practice guidelines: a new reality in medicine. I. Recent developments.

    Arch Intern Med 1990, 150(9):1811-1818. PubMed Abstract | Publisher Full Text OpenURL

  18. Fretheim A Schünemann HJ, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 3. Group composition.

    Health Res Policy Syst 2006. OpenURL

  19. Fretheim A Schünemann HJ, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 5. Group process.

    Health Res Policy Syst 2006. OpenURL

  20. Burgers JS, Cluzeau FA, Hanna SE, Hunt C, Grol R: Characteristics of high-quality guidelines: evaluation of 86 clinical guidelines developed in ten European countries and Canada.

    Int J Technol Assess Health Care 2003, 19(1):148-157. PubMed Abstract | Publisher Full Text OpenURL

  21. Oxman AD SHJ Fretheim A.: Improving the Use of Research Evidence in Guideline Development: Priority setting.

    Health Res Policy Syst 2006. OpenURL

  22. Fretheim A SHJ Oxman AD: Improving the Use of Research Evidence in Guideline Development: Group composition.

    Health Res Policy Syst 2006. OpenURL

  23. Boyd E Bero L: Improving the Use of Research Evidence in Guideline Development: 4. Managing conflicts of interest.

    Health Res Policy Syst 2006. OpenURL

  24. Fretheim A SHJ Oxman AD: Improving the Use of Research Evidence in Guideline Development: Group process.

    Health Res Policy Syst 2006. OpenURL

  25. Schünemann HJ Fretheim A, Oxman, AD: Improving the Use of Research Evidence in Guideline Development: 6. Determining which outcomes are important.

    Health Res Policy Syst 2006. OpenURL

  26. Oxman AD Schünemann HJ, Fretheim A.: Improving the Use of Research Evidence in Guideline Development: 2. Priority setting.

    Health Res Policy Syst 2006. OpenURL

  27. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 7. Deciding what evidence to include.

    Health Res Policy Syst 2006. OpenURL

  28. Oxman AD Schünemann HJ, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 8. Synthesis and presentation of evidence.

    Health Res Policy Syst 2006. OpenURL

  29. Schünemann HJ Fretheim A, Oxman, AD: Improving the Use of Research Evidence in Guideline Development: 10. Integrating values and consumer involvement.

    Health Res Policy Syst 2006. OpenURL

  30. Schünemann HJ Fretheim A, Oxman, AD: Improving the Use of Research Evidence in Guideline Development: 9. Grading evidence and recommendations.

    Health Res Policy Syst 2006. OpenURL

  31. Oxman AD Schünemann HJ, Fretheim A.: Improving the Use of Research Evidence in Guideline Development: 12. Incorporating considerations of equity.

    Health Res Policy Syst 2006. OpenURL

  32. Edejer TTT: Improving the Use of Research Evidence in Guideline Development: Incorporating considerations of cost-effectiveness, affordability and resource implications.

    Health Res Policy Syst 2006. OpenURL

  33. Schünemann HJ Oxman AD, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 13. Adaptation, applicability and transferability.

    Health Res Policy Syst 2006. OpenURL

  34. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 14. Reporting guidelines.

    Health Res Policy Syst 2006. OpenURL

  35. Fretheim A SHJ Oxman AD: Improving the Use of Research Evidence in Guideline Development: Disseminating and implementing guidelines.

    Health Res Policy Syst 2006. OpenURL

  36. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 16. Evaluation.

    Health Res Policy Syst 2006. OpenURL

  37. Burgers JS, Grol R, Klazinga NS, Makela M, Zaat J: Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs.

    Int J Qual Health Care 2003, 15(1):31-45. PubMed Abstract | Publisher Full Text OpenURL

  38. Oxman AD Lavis J, Fretheim A: The use of research evidence in WHO recommendations. (submitted)

  39. Moynihan R Oxman AD, Lavis JN, Paulsen E: Evidence-Informed Health Policy: Using Research to Make Health Systems Healthier. A review of organizations that support the use of research evidence in developing guidelines, technology assessments, and health policy, for the WHO Advisory Committee on Health Research. Oslo , Norwegian Knowledge Centre for the Health Services; 2006.