Email updates

Keep up to date with the latest news and content from HARPS and BioMed Central.

Open Access Highly Accessed Review

The utilisation of health research in policy-making: concepts, examples and methods of assessment

Stephen R Hanney1*, Miguel A Gonzalez-Block2, Martin J Buxton1 and Maurice Kogan3

Author Affiliations

1 Health Economics Research Group, Brunel University, Uxbridge, Middlesex UB8 3PH, UK

2 Alliance for Health Policy and Systems Research, World Health Organization, CH 1221, Geneva 27, Switzerland

3 Centre for the Evaluation of Public Policy, Brunel University, Uxbridge, Middlesex UB8 3PH, UK

For all author emails, please log on.

Health Research Policy and Systems 2003, 1:2  doi:10.1186/1478-4505-1-2


The electronic version of this article is the complete one and can be found online at: http://www.health-policy-systems.com/content/1/1/2


Received:20 December 2002
Accepted:13 January 2003
Published:13 January 2003

© 2003 Hanney et al; licensee BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.

Abstract

The importance of health research utilisation in policy-making, and of understanding the mechanisms involved, is increasingly recognised. Recent reports calling for more resources to improve health in developing countries, and global pressures for accountability, draw greater attention to research-informed policy-making. Key utilisation issues have been described for at least twenty years, but the growing focus on health research systems creates additional dimensions.

The utilisation of health research in policy-making should contribute to policies that may eventually lead to desired outcomes, including health gains. In this article, exploration of these issues is combined with a review of various forms of policy-making. When this is linked to analysis of different types of health research, it assists in building a comprehensive account of the diverse meanings of research utilisation.

Previous studies report methods and conceptual frameworks that have been applied, if with varying degrees of success, to record utilisation in policy-making. These studies reveal various examples of research impact within a general picture of underutilisation.

Factors potentially enhancing utilisation can be identified by exploration of: priority setting; activities of the health research system at the interface between research and policy-making; and the role of the recipients, or 'receptors', of health research. An interfaces and receptors model provides a framework for analysis.

Recommendations about possible methods for assessing health research utilisation follow identification of the purposes of such assessments. Our conclusion is that research utilisation can be better understood, and enhanced, by developing assessment methods informed by conceptual analysis and review of previous studies.

Review

Introduction and Background

The Director General of the World Health Organization (WHO) has decided that the World Health Report 2004, Health Research: Knowledge for Better Health, should involve a careful reflection of how advances in health research lead to improved health and health equity. The WHO has launched a broad Health Research Systems Analysis (HRSA) Initiative that will inform the 2004 report. One component of this initiative is a project focusing on the assessment of health research utilisation. The utilisation project itself consists of various elements. This paper was commissioned by the Research Policy and Co-operation Department of WHO, Geneva, to review the issues related to the utilisation of health research in policy-making, and, based on that review, make recommendations about appropriate methods for assessment of such utilisation.

WHO is giving increased emphasis to the role of health systems [1] and attention is focusing on the importance of policy-making in achieving effective health systems [2,3]. The World Bank made estimates of the costs of attaining the health-related Millennium Development Goals of between $20 and $25 billion a year. However, the report notes that: 'these unit cost estimates only apply when the policy and institutional environment is conducive to additional health spending being effective' [2]. The importance of health policy-making, in turn, being research-informed is recognised by a growing number of bodies [3-5].

The existence of relevant research, though necessary, is not sufficient. Evidence-based policy is difficult to achieve and it is widely agreed that health policies do not reflect research evidence to the extent that in theory they could [5-11]. Examination of the policy-making process confirms it to be extremely complex, with many genuine obstacles to evidence-based policy-making at the same time as there are factors that could increase research utilisation. A full review of the many possible meanings of research impact reveals that there may be more utilisation in policy-making than is sometimes recognised. Such a review also enhances understanding of the issues, including the differential scope for utilisation associated with different types of research and policy environments. Developing a conceptual framework of the processes of utilisation should assist with the formulation of assessment tools that reveal the full picture of the way research is used in policy-making. Furthermore, it should allow the growing demands for accountability for research expenditure [12-18] to be addressed appropriately, which could also be of benefit to the research community.

There is a rich background of material for each of these areas, including key contributions from Weiss identifying the multiple meanings that can be attached to research utilisation in policy-making [19]. Their importance lies in the fact that some of these meanings, or models, point to less obvious patterns of use than those suggested by instrumental research exploitation which involves research findings being directly used in policy formulation.

About twenty years ago there was recognition of the need for analysis to combine a range of factors such as the nature of different types of health research knowledge and the diverse institutional arrangements for policy-making. In their assessment of the attempt in the 1970s to increase utilisation of research funded by the UK's Department of Health, Kogan and Henkel found, 'the interconnections between epistemologies and institutional relationships were a recurring theme' [7]. The importance of interactions across the interfaces between researchers and policy-makers was identified. The role of policy-makers as the receivers, or receptors, of research and the need for careful priority setting were highlighted. Various elements of this analysis were recently reported also to be relevant for health research in Mexico [10].

The context of the current move to attempt to increase research utilisation is important. There is now a broad coalition pressing for improvements. Various organisations came together in 2000 to support the formation of the Alliance for Health Policy and Systems Research with its 300 partner institutions. It aims to promote capacity building for, and the dissemination and impact of, research both on and for policy [20]. At the level of specific programmes within international bodies, there is a growing stress on the role of policy-making: 'Research on implementation, on policy-making or programme development is as important as basic clinical research for improving child health' [21]. Recent weeks have seen publication of the first systematic review to address research utilisation in policy-making [22]. Furthermore, the developing interest in research informed policy-making coincides with the extensive efforts being made to increase the implementation of health research findings more generally. Indeed, the emphasis on evidence-based medicine is itself generating extra pressure from practitioners that policy-makers, too, should have a duty to consider research evidence [23]. The role of research utilisation in policy-making is seen as a key element in the growing interest within WHO on research utilisation and its assessment [3,24].

A further important part of the context is that developments in the UK in the 1970s, and in other European countries [25], could be seen as early attempts to develop a system to augment the traditional individualistic determination of medical research priorities in universities and hospitals. A similar emphasis on issues such as priority setting is seen in recommendations made for middle and low income countries by the Commission on Health Research for Development [26]. The concept of Health Research Systems (HRSs) is now of growing significance [27]. One of the main elements that distinguishes a HRS is the attempt to develop mechanisms and networks to facilitate the greater use of health research.

Building on the above analysis, it is our contention that many factors need to be brought together if assessment of research impact on policy-making is to contribute to an understanding of the issues and an enhancement of utilisation. The prime focus should be the policy-maker. This paper first examines the concept of policy-making, and the underlying assumption that it is better if it is research-informed. Then we consider the range of types of health research and the levels of policy-making at which they could be applied. These strands are brought together to provide an analysis of the wide range of ways in which research can have an impact on health policy-making.

The focus then moves to examining contributions from previous studies of knowledge utilisation in health policy-making, including those using standardised measures. Various dimensions of our conceptual analysis form the next sections. We start with the interfaces, both at the priority setting stage and when findings are communicated between researchers and policy-makers. The role of policy-makers as receivers, or receptors, of research is examined along with the accompanying institutional arrangements. Incentives are also important. The material is brought together in a wide-ranging interfaces and receptor model of research utilisation in policy making. Finally, the various possible purposes of assessment of research utilisation are considered before suggestions are made about suitable methods for assessing the impact of research on policy-making. Such assessments would be best undertaken as part of a wider evaluation of the utilisation of health research by industry, medical practitioners and the public.

The nature of policy-making and its role in knowledge utilisation

Policy-making can be viewed as involving the 'authoritative allocation of values' [28], and when interpreted broadly can include people making the policy as government ministers and officials, as local health service managers, or as representatives of a professional body. Policy-making involves those in positions of authority making choices that have a special status within the group to which they will apply. The results take many forms ranging from national health policies made by the government to clinical guidelines determined by professional bodies. This broad usage of the term policy-making has advantages when conducting knowledge utilisation or payback assessments, and has contributed to a conceptual framework for a series of such studies [14,17,29]. In this article, however, the analysis mainly concentrates on public policy-making rather than that undertaken by professional bodies.

This framework consists of two elements. These are a multidimensional categorisation of benefits from health research, going through from the primary and secondary outputs to the final outcomes, and a model of how to assess them. A revised version of the model is shown as Figure 1 and consists of a series of stages. This sequence can be useful when examining how a health research project could be utilised, in policy-making and practice, in ways that result in final outcomes such as health gains and economic benefits. Public engagement with research can play a key role in research utilisation. The model incorporates the concept of the stock, or pool, of knowledge and the idea that there are various interfaces between research and the wider political, professional and social environments. These points, together with various feedback loops and forward leaps, mean that although the stages are presented in a linear form, the model recognises that the actual steps involved in utilisation and achieving final outcomes are often multidirectional and convoluted. That said, the model helps both to organise assessments and to indicate where the various elements of the multidimensional categorisation of benefits might occur.

thumbnailFigure 1. The Place of Policy-Making in the Stages of Assessment of Research Utilisation and Final Outcomes.

Key:

Direct lines within the flow or feedback

Indirect lines of communication

Primary Outputs – Publications, trained researchers

Secondary Outputs – Policies from national, local and professional bodies

Final Outcomes – Health and equity gains, cost-effectiveness and economic benefits

Source: Adapted from S Hanney, S Kuruvilla: HRSPA Project 4: Utilisation of research to inform policy, practice and public understanding and improve health and health equity. WHO/Wellcome Trust Technical Workshop. London, January 2002; and S Hanney et al 2000, Evaluation, 6, published by Sage [29].

The framework is also important for the structure of this article because it demonstrates that assessment of research impact on policy-making is best undertaken as part of a wider analysis of the utilisation of research. Throughout the paper it will become increasingly clear that policy-making is itself influenced by industry, by health professionals who might be expected to apply research findings in their practice, and by the public who might engage with research, either as patients or more generally in society. The interaction of all these groups with research findings is an important consideration and the interfaces operate at many levels.

In terms of the utilisation of the knowledge, research-informed policies can be referred to as secondary outputs from research [14]. This distinguishes them from the primary, or direct, outputs of research processes such as journal articles, other publications, and trained researchers. Neither, however, are the policies the desired final outcomes; they represent a step in the process. It is sometimes possible to identify how research findings have informed policy-making even when it might be extremely difficult to trace influences at other stages in the utilisation processes. Furthermore, the approach enables the processes of research utilisation to be identified in ways that would be impossible if the analysis attempted to jump immediately to the final outcomes. In particular, detailed analysis at this stage can address the counterfactual, ie consider what might have happened without the relevant research: would the policy have been changed anyway?

Not all examples of health knowledge utilisation go through a policy-making stage, and in some cases the policy comes after partial translation of the findings into practice. For example, clinical guidelines are usually developed after leading clinicians in the field have already adopted an evidence-based practice and then seek to encourage its wider diffusion throughout the profession. Nevertheless, often a policy-making stage in knowledge utilisation is important if the final outcomes of health, health equity, and social and economic gain are to be achieved. The potential importance of a policy stage in the process of turning evidence into application is increasingly being accepted, even for clinical practice [30].

A positive case can be set out for the contribution research can make to policy-making. The basic assumption of knowledge utilisation related to policy-making is that policies which are research informed will be better than otherwise would have been the case. It is assumed that research exposes policy-making to a wider range of validated concepts and experiences than those that can be drawn from the normal time-limited and politically constrained processes of policy deliberation. It thus allows a broader choice of policy options to emerge. Research often enables policies to be generated upon technically well-informed bases. It gives warnings of reasons why some policies succeed and others fail. It can make connections between otherwise separate factors such as the nature of the substantive field and organisational patterns set up to manage them, or the power of environments over health outcomes. It legitimises some policies and throws legitimate doubts on others. Analysis of policy-making, and of research utilisation, often identifies at least three broad areas of activity: policy agenda setting, policy formulation, and policy implementation [11,31]. Potentially, research could play a part in all three areas. Evaluation is also often seen as an important activity, and one that adopts a research approach. Indeed, in this paper evaluation is primarily being viewed as a form of research, the utilisation of which will be examined in the other phases of policy-making.

Davies and Nutley, in a recent analysis of the role of evidence, or 'What Works', in a series of public services, suggest 'the research community in healthcare is truly global, and the drive to evidence-based policy and practice is pandemic' [32]. Of the public services, health care is seen as the one where, despite the difficulties, the utilisation of knowledge is most advanced. Although this analysis does examine a range of policy-making models, critics claim that the theoretical basis for evidence-based policy-making is not strong because it rests too much on a rational view of policy-making [33,34]. The debate needs to be informed by various models of policy-making such as those set out below.

Many categorisations of policy-making exist. The categorisation of policy-making presented here is not intended to describe the models comprehensively. Instead, it is based on previous analyses of public policy-making that were specifically made in the process of analysing research utilisation. The categorisation incorporates work undertaken by Kogan and Tuijnman [35], for the Organisation for Economic Co-operation and Development, and by many others [9,36-38]. The various models described are not mutually exclusive, but are included because each makes a specific contribution that is built on in later analysis:

Rational models

Rational models of policy-making assume policy-makers identify problems, then gather and review all the data about alternative possible solutions, and their consequences, and select the solution that best matches their goals. Sometimes this approach is known as ends-means rationality; it is thus different from some of the models below which might, nevertheless, also seem rational to the policy-makers involved. The various models of policy-making should be seen as a spectrum. Thus Simon [39] is sometimes seen as writing from the rationalist tradition, but he was critical of the more basic rational models and his concept of 'bounded rationality' involves concentrating the review of data on a more limited range of possible solutions.

Incrementalist models

It has long been recognised that policy-making is a complex process. It can involve scientific knowledge and a range of other factors including interests, values, established positions within institutions, and personal ambitions. Furthermore, evidence from research has to compete with what Lindblom and Cohen [40] call 'ordinary knowledge' which owes its origins to, 'common sense, casual empiricism, or thoughtful speculation and analysis'. In models such as 'disjointed incrementalism' [41] policy-making does not involve a clear movement towards predetermined goals but rather is more a series of small steps in a process of 'muddling through' [42] or 'decision accretion' [43]. Incrementalists allow for a greater role for interests in policy-making debates and emphasise the many sources of information that impinge on policy-makers.

Networks

A networks approach also highlights the role of different interests and how the relationships between such groups and policy-makers can result in an incremental policy process. The term 'policy network' is defined as a generic label for the different types of state/interest group relationships, for example 'policy communities' in which the long term relationships between government officials and representatives of leading interest groups are particularly powerful [44-46]. Other definitions of the term networks involve a wider membership and are more likely to include researchers. It is claimed that researcher involvement in 'social networks' [47] is important for research utilisation. Others suggest that leading experts who share a similar approach on an issue can be seen as an 'epistemic community' [48] and can influence policy. Analysis of health systems often suggests the influence of the medical profession over policy-making is particularly strong [49]. Its domination of the policy networks led to the use of the term 'a professionalised policy network' [50,51]. Its influence is likely to be a factor in setting agendas and determining the type of knowledge to which most notice is taken in the policy debate.

The 'garbage can' model

The 'garbage can' model of policy-making [52] looks at these issues in an idiosyncratic way. It suggests that sometimes solutions that might have been disgarded nevertheless remain in the policy-making system, and occasionally there are problems to which they become attached. Models such as this highlight the way in which policy-making can be seen as a most untidy process, rather than neatly going through a series of phases [53].

These various models of policy-making, even occasionally the final one, are likely to be found relevant to different circumstances and parts of health systems. They will have different implications for the utilisation of research and although they do not stack up as connected paradigms, or have much predictive power, they help put shape onto otherwise inchoate patterns. We shall explore how far they specifically map on to models of research utilisation after considering the range of overlapping categorisations of health research.

Categories of health research and possible levels of utilisation

The categorisation of health research discussed here has a potential importance for the analysis of utilisation. Often a broad distinction is made between basic, clinical and applied research. By its nature basic, or blue-skies, research is not often likely to be utilised until further, less basic, research has been undertaken and perhaps some synthesis with other findings has occurred. Research that follows priorities determined by the researchers themselves, according to the 'internalist' norms of science [54], is more often, though not always, going to be basic. Applied research is more likely than basic research to be following an agenda driven by forces other than the scientific imperative. Just because the research topic has been set by non-researchers does not, of course, ensure its impact. Nevertheless, where such drivers and sponsors are also the most likely potential users of the research, this provides some of the circumstances that might encourage utilisation [7,53,55].

There is generally greater resistance within health services to the use of social science, despite it often being applied and user-driven, than there is to the adoption of the findings from natural sciences [7,10] such as those used in clinical science. Possible explanations include the fact that the more highly technical and specific the research, the more there might be circumstances in which it can be utilised directly by policy makers without ideological or political considerations intervening too much. Moreover, the receptors of research are likely to place more confidence in the strictly controlled natural sciences than in the more eclectic social sciences. Much of policy-making can be seen as a craft, which draws substantially on ordinary knowledge and in which the contextual component will often be more significant than the type of evidence offered by social science research [56].

Another partially overlapping distinction is between national and international research. International research findings might be more likely to be utilised where there is greater technical content in the research and also potential for application to an issue of patient care. The report of the Commission on Health Research for Development also identified the particular contributions that national and global health research could make [26]. It developed the concept of Essential National Health Research (ENHR). This entails a strategy in which each country plans its health research according to country-specific health problems and the contribution it can make to regional and global health research. Mechanisms for the synthesis or systematic review of research might become even more important in relation to international research.

Adding to the epistemological debate about the most appropriate forms of production of knowledge intended for utilisation, Trist [57] argued that domain-based research represented a third category alongside basic and applied research. Domain-based, or policy-oriented, research is essentially interdisciplinary and the crossing of new boundaries and the creation of new syntheses may advance both knowledge and human betterment. It also entails wider reference groups, beyond the scientific or clinical communities. Along similar lines, Gibbons et al [58] claim to identify a shift from the traditional discipline-centred mode of knowledge production that they characterise as Mode 1, towards a broader conception of knowledge production described as Mode 2. In this, knowledge is generated in a context of application and addresses problems identified through continual negotiation between actors from a variety of settings. The results are communicated to those who have participated in their production. Although the degree of change described by Gibbons et al could be exaggerated [59], this general approach, as with that of Trist, is compatible with attempts to increase utilisation by focusing research production on the interests of at least some potential users.

A slightly different dimension, but one also associated with utilisation, is that of the features of specific research studies. When a particular piece of research is seen to be of high quality this might help reinforce a policy-maker's inclination to use it [14,60], as might a favourable view about the quality of the specific researcher [10]. The argument, as developed by Weiss and Bucuvalas, is that where the policies are potentially controversial the decision-makers will not want the credibility of their case undermined by critics pointing to flaws in the research behind the policy [60]. When Ministers in the UK supported actions to address gender inequalities in the medical profession, they did so with full confidence in the quality of the research that demonstrated the problem [61]. Examinations of the use of economic evaluations and Health Technology Assessments (HTAs) in policy-making have considered the importance of the quality, reliability, timeliness and comprehensiveness of research in influencing the level of utilisation [62-67]. For example, the latter two factors were highlighted as important determinants of the usefulness of the information in the context of drug formulary decisions in the USA [68].

Different types of research are likely to be most relevant for various levels and situations of policy-making, and for different aspects of those policies. There is no agreed typology of policy categories suitable for utilisation assessment [11]. Above, we suggest that the interpretation of policy being adopted here covers national policies, local health service policies and policies made by professional bodies. Along not dissimilar lines Black [34] argues that an earlier threefold categorisation [69] could be appropriate when examining health research and policy-making. The three categories are: governance policies which relate to organisational and financial structures; service policies which cover resource allocation issues and pattern of services; and practice policies which relate to the use of resources by practitioners in delivering patient care. A similar division appears in the threefold categorisation proposed by Lomas: 'legislative, administrative and clinical' [70]. Legislative policies relate to the overall framework for organising health services; administrative to the running of the service and allocation of resources within the overall framework; and clinical to the policies about what therapies are applied. These categorisations are best seen as a spectrum, but it is generally agreed that research has least impact on the first of these categories and most on the third where often the relevant knowledge comes from clinical research. This is despite the frequent delays in turning research evidence into improved patient care [71,72].

Some of the issues in this section are illustrated in relation to Health Technology Assessments (HTAs). Various features of HTAs might be associated with the sometimes quite high levels of translation into policy-making and through into the final outcomes [14,73-75]. Many HTAs are undertaken, commissioned, or produced by technology 'sponsors' specifically for agencies set up to advise governmental bodies setting policies for delivering patient care in national health systems. Frequently they address a very specific question that has been identified and prioritised by the health care system: presumably, a question to which the system wants an answer, and by implication is willing and able to accept alternative outcomes if they can be supported by evidence.

Whilst these HTAs are 'technical' in the sense that they typically relate to quantitative measures of effectiveness and cost-effectiveness of specific interventions, they do have important distributional and equity implications. Policies deriving from them may induce strong public and patient reactions (as is evident in media coverage of proposed guidance from bodies such as The National Institute for Clinical Excellence in the UK). This emphasises the need in such systems to differentiate between the research activity of health technology assessment and the decision-making (or guidance forming) process of appraisal of that evidence and its implications [76].

But even for HTA the evidence of widespread, direct impact on policy (with policy seen as entirely convergent with the research evidence) is at best patchy. A study in the Netherlands by van den Heuvel et al [77] concludes that policy decisions concerning the introduction of (new) technologies in health care are not based on the results of medical technology assessments. Rather, 'political arguments and interest groups decide the outcomes'. In a recent literature review, Barbieri and Drummond [78] found few examples of HTA impact in European health care systems. At the local level, those involved in making policies about the introduction of new medical technologies are likely to view the contribution that effectiveness research can make in different ways, depending on their professional backgrounds [79]. This discussion underlines the fact that even in the circumstances most favourable for 'rational' policy-making there are limitations upon it. This indicates the need to consider the full range of models of research utilisation.

Models of research utilisation

Having reviewed various models of policy-making in the second section, and examined different types of research in the third section, it will now be useful to consider models of research utilisation in policy-making. Then we can see how far the various strands from these sections can be drawn together and developed in our later construction of a conceptual framework. We start by looking at previous models of research utilisation, and then suggest ways in which they could be elaborated.

Following the work of Weiss [19,80], and others [7,14,37,38,55,81,82], various models of research utilisation in policy-making have been identified, and they are thought to be applicable beyond the social sciences:

The classic/purist/knowledge-driven model

This suggests a linear sequence in which research generates knowledge that impels action.

The problem-solving/engineering/policy-driven model

This also follows a linear sequence, but begins with the identification of a problem by a customer who requests the researcher to identify and assess alternative solutions. This was explicitly the model behind the changes attempted by the UK Department of Health in the 1970s [7].

The interactive/social interaction model

The process here is a set of interactions between researchers and users rather than a linear move from research to decisions. It ensures they are exposed to each other's worlds and needs.

The enlightenment/percolation/limestone model

According to this, research is more likely to be used through the gradual 'sedimentation' of insight, theories, concepts and perspectives. This model has the advantage of extending the range of ways in which research is seen to be utilised.

The political model

In this, research findings become ammunition in an adversarial system of policy making.

The tactical model

Here research is used when there is pressure for action to be taken on an issue, and policy-makers respond by announcing that they have commissioned a research study on the matter. Whilst this can sometimes be seen as a cynical delaying tactic, there are other occasions on which the commissioning of research provides the political system with a valuable breathing space, thus reducing the chances of irrational policy-making.

There is no precise overlap between the principal characteristics of policy-making models discussed earlier and the utilisation processes listed above such as would allow them to be presented in neat pairs of singletons. The first two categories of utilisation both fit with rational models of policy-making, but it is the problem-solving model that shares the same starting point: identification of a problem by a policy-maker. The more incremental models of policy-making have the longer time frame implied by interactive and enlightenment models of utilisation, but sometimes these forms of utilisation lead to paradigm shifts which are much more radical than is inherent in incrementalism.

Policy networks are seen as providing a useful framework for studying research utilisation [36]. Where researchers become part of a policy network, or find their ideas taken up by elements within it, this could be a strong version of the interactive model and be an important route for such findings to enter the policy arena. Network approaches can highlight the role of stakeholders in research utilisation [9,14]. (The network concepts could also, however, help to explain the difficulties some research faces in gaining acceptance, or even a hearing. Policy-making systems can be relatively impermeable to research findings that are contrary to the consensus developed as a result of the strong, long-term, links between departmental officials and leaders of the main interest groups).

It is of value to explore the variety of policy-making/utilisation connections because they underline the argument that it is not realistic to expect policy-making always to follow the ends-means rational model that might entail the clearest use of research. Weiss also suggests that there are three main forms in which research might appear and be utilised in policy-making: as data and findings; as ideas and criticism in the enlightenment mode; or as briefs and arguments for action [83]. Along similar lines the utilisation of research in policy-making is sometimes considered to be instrumental, conceptual or symbolic [11]. As we have seen, instrumental use involves research findings being directly used in policy formulation, conceptual use refers to a type of enlightenment mode of utilisation, and symbolic to the use of the research to support a position already taken, which may be to continue with existing policies.

Taking another of Weiss's arguments, that utilisation of research can be usefully be defined as a process of interaction between research inputs and decision outputs [43], we next elaborate the range of possible uses of research. Given the diversity of forms of knowledge and policy decision, their interaction has to be understood in the context of both the diverse values shaped by philosophies of knowledge and the practical aspects of policy-making. With regard to the former, policy-makers may privilege empirical findings against more abstract and general models of reality. In terms of policy-making it is useful to distinguish three dimensions: the nature of decisions, that is, the extent to which they are explicit and specific versus implicit and diffuse; the extent of choice available in a given situation; and the political or technical character of actors participating in decisions. These are shown on Figure 2 and developed below but it is useful to consider the categories as a spectrum.

thumbnailFigure 2. Decision Context, Research Inputs and Forms of Research Utilisation in Policy-Making

Sometimes policy-makers make rational and weighted decisions along a well charted course of action, yet more often apply knowledge through largely routine or unconscious processes in response to ad hoc situations; here the context is implicit rather than explicit. A situation of choice will exist when several alternatives are perceived as viable, contrasting with a situation where a decision has been taken and the role for research is rather to support this choice. Support in turn covers two types of situation. Specific findings can be used to legitimate decisions when these have been formed, have hardened or when they are being implemented. In relation to the concept of models referred to above, support is more a matter of explicit policies being seen to be made by institutions that are research-based and therefore the policies gaining greater credibility. Political decisions are normally justified in terms of social values and understandings shaped in the political arena, but there can be a role for scientific inputs in the policy formulation. Technical decisions are those that are expected to be justified in terms of scientific or specialised methods.

The combination of diverse forms of scientific inputs and decision outputs shapes the processes of utilisation and creates specific expectations and opportunities. The components of Figure 2 are expounded in more detail as follows:

Conceptual modelling

Knowledge to inform complex situations is frequently demanded in the form of concepts to model or shape the general nature of the policy problems and possible solutions. Planning health sector reforms or identifying health policy in areas normally outside its purview, such as poverty or economic development, are likely to demand such knowledge, as they provide new disciplinary or social perspectives on a given problem and activate new associations and meanings for policy issues [84]. They can be a first step to other forms of research utilisation.

Data-based policy

This form of utilisation aims to influence courses of action on the basis of the strength of empirical findings. Scientists may take the lead through a 'knowledge-driven' approach, or policy-makers can demand such knowledge to solve specific problems ('policy-driven' model). In either case scientific rigour, robustness and objectivity would be principles trusted by both researchers and policy-makers.

Constrained modelling

Constrained political conditions give rise to utilisation that, from the perspective of researchers, uses only a restricted range of available knowledge. Likewise, policy-makers will not commission or will discourage research that, in its broad outlines, poses more political risks than benefits [85].

Strategic research

Policy is most often formulated in a context where lay (as opposed to technocratic) actors vie for power and resources. The choice of policy may be open, but only through politically controlled windows of opportunity. Under these circumstances the ultimate validity of research will be assessed together with other and often competing evidence. The aim of researchers is usually to influence policy choice or to make explicit the costs of not adopting a recommended course of action [86].

Symbolic payback

Science has become a potent cultural symbol that permeates modern life and confers privileges on its users. Likewise, there is a political pay-off in supporting research and building research capacity in strategic areas. Research has become 'an essential mode of communication and persuasion in the public arena' [43]. In complex organisations, research could be a common language used to talk across the boundaries of interests and content areas given its capacity to effectively link disparate realities and because of 'the patina of rigor that science confers to discourse'. This might suggest that policies from bodies known to be research-informed might be more likely to be supported.

Symbolic argumentation

Policy making may be based mainly on reasons of interest, ideology or intellect. Under these circumstances, however, research can still be used as ammunition to support the decisions made and being implemented. Science content is here used as a collection of arguments, rather than as data or evidence to be weighed. Arguments may be fashioned as by-products of formal research publications, particularly by policy analysis units, consulting firms and the media [87].

Paradigms

Given the large measure of unconscious elements in every-day decision making, accepted ways of interpreting reality and facing problems are the most important influence. An aggregate of normative expectations may amount to an overriding view of what is desirable health policy, such as those advanced in Welfare State thinking. Such policy paradigms may be triggered off or supported by single or grouped assumptions derived from research, which also may achieve paradigmal status. Individual policies are likely to reflect the dominant paradigms of their time.

Policy-Makers' practice wisdom

How far individual policy-makers will automatically attempt to use research findings on a regular basis will depend on multiple influences, such as training, continuous education, exposure to the media and to the demand of clients.

Although these categories are not water-tight, they help indicate the breadth of types of research utilisation and, therefore, areas on which any assessment methods should focus.

Various elements from this and preceding sections will be particularly useful as we develop our own conceptual framework for analysing and assessing research utilisation in the context of the increasing attention on HRSs. For example, the importance, but also the limitations, of the problem-solving model must be considered when examining the role of research priority setting. Furthermore, in light of the practical limitations on rational models, the importance of interactive perspectives will be highlighted as a way of encouraging policy-makers to be responsive to relevant research. Overall, both these points, and others, suggest that it is appropriate to focus on the actions that could be taken to encourage permeability at the interfaces between policy-makers and researchers. Such actions should help ensure both that researchers are aware of policy-makers' needs, and that the policy-making system is willing and able to absorb relevant research findings.

It is also possible to link some of the ideas in this section to the three phases of policy-making referred to earlier: agenda setting; policy formulation; and policy implementation. For agenda setting the research could impact in one of several ways. Research could demonstrate the existence or extent of a problem, through either specific findings or a process of enlightenment. Alternatively, it could be that, as in the knowledge-driven model, the mere generation of the findings leads to pressure to act upon the new knowledge. The use of research in policy formulation could be in either the instrumental or conceptual/ enlightenment mode. A further possibility is that it could be used as briefs to inform arguments as set out in the political model of research use. The implementation of health policies is widely acknowledged often to be difficult [37,53]. At the implementation stage, research could play some part in demonstrating the best way to implement policy and could inform decisions. It could also be of symbolic use in helping to build support for implementation through assistance with communicating or justifying the policy and being used to generate support for it in terms of financial resources, political commitment, and public opinion.

Before developing the material into a conceptual framework and methods for assessment of utilisation, it will be useful to review the focus and methods adopted in previous studies of policy-making. This review will provide examples of where wide interpretations of utilisation have been incorporated into studies of the impact of research.

Contributions from selected previous studies of health policy-making and knowledge utilisation

Studies of the nature and impacts of policy-making can each exploit a wide range of analytic methods. The area covered by them is extensive and here we review one study: that of policy change in relation to health financing reforms in South Africa and Zambia [31]. It illustrates several of the approaches that can be applied when analysing policy-making in general, and is included here as a backdrop to the accounts, immediately following this one, specifically on research utilisation in policy-making. The case study in each country was organised according to a conceptual framework consisting of a process of policy change moving from, Gilson et al state, 'agenda setting around a reform of focus, to reform design, and then through implementation to the achievement of immediate and longer term changes' [31]. The policy analysis approach of Walt [37,88] was also drawn upon so that the factors influencing each stage of the reform process were categorised and analysed according to four broad factors: context, actors, process and content.

In the two countries data were collected through: documentary analysis; key informant interviews with policy-makers and analysts; media analysis; and review of secondary sources. The data analysis techniques included: development of a timeline for each reform; stakeholder analysis; policy mapping techniques; impact analysis through use of secondary data; and a review process. The two case studies incorporated examination of the impact on the policy development made by research analysts, both inside and outside government, and found it to be 'strongly dependent upon the presence of a policy champion' [31].

This section now draws selectively upon a review of a wide range of previous studies specifically about research utilisation in health policy. From this several key themes are discussed:

• focus of the study;

• how far the study was based around a conceptual framework and how far comparisons or conclusions were drawn;

• methods and standardised measures used in assessing impacts and outcomes;

• levels of utilisation and other benefits shown.

Focus of the study

Studies that start with the research project or programme, and examine the impact that it has had [14,17,75,89,90], have the advantage of a reasonably tightly defined focus. They can be of use for research funders, but often run into the problem that any impact research makes is usually of a contributive nature and it is difficult to identify the impact of one project or programme from that of others with which the findings get mixed [16]. Other studies consider the portfolio of researchers' work over a particular period [91], or the contribution of specific health research centres [29,92-95]. Studies such as these lend themselves to network analysis.

Health policy-makers have been the focus of studies. Some examine the policy-makers' use of research in general [96]. Others can involve health policy-makers, for example in mental health, being shown research papers describing evaluations of programmes and then asked how useful they would find such research [60]. Drummond et al asked local policy-makers about their attitude to using economic evaluations in general, and whether they had used specific evaluations [65]. A major way in which impact on policy has been assessed is through studies that start with a policy area, or theme, and then seek to identify how far the policy-making or implementation has been informed by research [11,77,96-102]. Focusing on the policy area has the advantage of facilitating some assessment of how far lack of relevant research is the problem, as opposed to the underutilisation of existing research.

The two broad approaches, starting with research or starting with policy, have elements of overlap. A study of the role of research in the regulation of private health care providers in an Indian state focused on the activities of a key research centre [103]. In some studies of the impact of research on particular health care programmes, interviews with researchers produce examples of how their own research has been utilised [10,104]

Some studies have cases from a range of countries and have been organised by, or involved, international bodies including the Council on Health Research for Development (COHRED) [5], the European Union (EU) [101], and the Cochrane Collaboration [102]. Other studies explicitly take examples from two or more sub-national units [11]. Many studies cover a series of examples from the same country or sub-national unit.

Conceptual framework and comparisons

Some of the most illuminating studies are organised according to a conceptual framework. Thus Landry et al [91] attempted to operationalise Knott and Wildavsky's Ladder of Knowledge Utilisation which suggests there are six stages of utilisation [105]. These are: transmission, cognition, reference, effort, influence, and application. Generally, studies that adopt a conceptual framework involve a series of cases. Buxton and Hanney [14,17,29,106] organised several series of studies around the conceptual framework described in Figure 1. Walt's framework for analysing health policy-making [88], described earlier, was used to organise impact studies in Mexico [10]. The EU-funded set of studies in developing and developed countries [101] was based around a conceptual framework, as described by Sauerborn et al [9], that concentrates on analysis of the role of the various interests or stakeholders.

A key feature of all these studies is that the conceptual framework facilitates comparable analysis of the results. A considerable amount of this analysis is drawn upon elsewhere in this paper, but one example is given here. Lessons were learnt from the EU-funded studies regarding: factors enhancing or hindering research use; possible indicators to measure research use; and possible strategies to increase such use. However, it has become clear that making generalisations is difficult because of country specific factors such as differing contexts (including the political system and culture), differing stakeholder constellations, and differing availability and quality of research. A conclusion, therefore, is that careful consideration of these factors, rather than recipe-like approaches, will be needed for successful enhancement of the use of research. (A. Gerhardus-personal communication). Similarly, Berridge and Stanton [107], after reviewing various case studies that were not undertaken as part of a set, identify a series of factors considered important in the research/policy relationship. They go on, however, to note that 'they are necessarily historically determined, and culture and context specific, rather than part of a reproducible general formula for action' [107].

There are clear tensions in our analysis between recognition of the genuine limitations in the ability to make generalisations and a desire to learn as much as possible from comparisons. It is useful, therefore, to note the findings from a study that addresses the issue of how far analysis from developed countries might be appropriate elsewhere. Trostle et al found there were various issues where, in comparison with developed countries, there could be different emphases in Mexico and 'in other developing country contexts' [10]. Nevertheless, most of the factors that Husén and Kogan [108] had identified as encouraging utilisation of educational research in industrialised nations were, they state, 'also found to be important in our study. These included decision-makers' willingness to consider research results as input for decision-making, and political stability...and the existence of research networks or commissions which provide a favourable arena for interaction between research and decision-making' [10].

Methods and standardised measures used in assessing impacts and outcomes

The two methods used most frequently, and usually used together, come from the qualitative tradition: documentary analysis and in-depth interviews [10,11,14,63,75,77,97,98,104]. The need for having the flexibility interviewing can provide is well illustrated by studies in which the original interview schedule had to be amended to take account of the different perspectives held by policy-makers of the role of research [79]. Interviewing is useful for understanding many aspects of research utilisation, including tracing networks between researchers and users [90].

Some studies use insider knowledge [94,102] and there has been some adoption of questionnaires to researchers about the utilisation of their work [11,106]. Particularly where the policy-making is at a local level, questionnaires have been used and administered either by phone [68] or by post [65]. In the latter case, Drummond et al also attempted to assess the problem of inaccurate responses. They included two fictitious studies in the list sent to policy-makers, and almost 20% claimed to have seen these studies; some of those admitted to having been influenced by them [65]. Bibliometric analysis is sometimes incorporated into broader studies [106] and, in an analysis of the papers that were cited in clinical guidelines, Grant et al specifically adopted a bibliometric approach [109]. Historical approaches have also been adopted [99,107] and allow a more contextualised analysis. The methods used are diverse, and only partially depend on the focus and purpose of the studies. The list of studies described here partially overlaps with, but is not identical to, those 24 included in the recent systematic review [22]. The 24 include a greater proportion that use questionnaires administered either by phone or post, but still face-to-face interviews form the majority.

There have been a few recent attempts to scale or score the degree of impact. Four such studies are described below, starting with two where the focus of each case study was a specific piece of research. An assessment of the impact of HTAs in Quebec used a case study approach [75]. Initially seven levels of critical incidents were identified. The impact of each HTA was scored on the basis of documentary analysis, and the information completed and validated through contact with key witnesses. In this way, Jacob and McGregor explain, 'by taking into consideration the level and number of critical incidents, an overall estimate of impact on policy was awarded to each HTA. This was reported in a scale extending from 0 (no impact) to +++(major impact). The weight awarded to critical incidents was adjusted according the nature of the decision at issue' [75].

Despite the largely, though not exclusively, qualitative methods used by Buxton and Hanney, in one study they attempted to score the impact made by projects on the basis of material from questionnaires [106]. This was partially validated by re-scoring those projects for which more detailed information became available from case studies. The correlation was quite good at an overall level, though there were differences in both directions at the individual project level. This scoring was entirely a methodological exercise and the results for specific projects were not identified.

Lavis et al coded their interview material according to whether the research had been used at agenda setting and/or policy formulation stages, and whether the research had impacted on all the policy or only partially [11].

The above three examples involved members of the assessment team undertaking the scoring and coding. The EU-funded project described by Gerhardus et al developed a model for mapping research to policy flows based on the qualitative case studies. From this model a set of numerical indicators was devised which entailed scoring by both the assessment team and stakeholders [101]. The intention, in what is probably the most methodologically ambitious research utilisation study included in this review, was that the indicators would be used in each of the eight countries to facilitate comparisons between research utilisation before and after interventions aimed at increasing such utilisation. In each country a policy was identified along with the content, conclusions and recommendations of the relevant research. Next, a series of questions was put to the stakeholders and points allocated according to their recall of the content etc. Further points were allocated depending on the references to the research made in speeches, statements, guidelines and similar sources. Finally, the stakeholders were asked to rate on a five-point scale a range of factors, including research, that influenced their decision making [101].

Preliminary results from the study suggest that conceptually the set of indicators has proven to be helpful, but problems with computation of the indicators arose due to the generally small sample of stakeholders interviewed. There were also problems related to the data collection for applying the knowledge-related indicators. This part of the stakeholder interviewing frequently created a possibly stressful, or even embarrassing, exam-like situation and in some cases revealed problems due to the considerable recall period involved when probing about older pieces of research. The body of indicators developed in the project was quite large, and it is considered appropriate to reduce it to a simplified set of core indicators (U. Sunderbrink – personal communication).

Level of utilisation and final outcomes

The examples suggest a greater level of utilisation and final outcomes in terms of health, health equity and social and economic gain than is often assumed, whilst still showing much underutilisation. There is considerable variation, both within and between studies. The study of the role of research in child health policy and programmes in Pakistan [104] found some examples of immediate clear-cut linkage between research and decisions, but in general the view was that research was little utilised. A mixed picture was reported in the Mexican studies: biomedical or clinical research was thought to be 'a critical resource for decision-making in each of the four programmes', but the importance of other types of research varied [10]. Of the eight policies examined in two Canadian provinces, four seemed to have been influenced by research, for example, in terms of agenda setting, research identified the need for increased pneumococcal immunisation in Saskatchewan [11]. Research utilisation is also demonstrated in some of the primarily insider-accounts, including that by Phoolcharoen [94] describing the role of the Health Systems Research Institute in Thailand in enabling research to impact on the reform of the health system.

Considerable utilisation is reported in some of the studies that focus on specific pieces of research. For two of these sets of studies, a wide interpretation of utilisation in policy-making was used [14,17,89,106], and one focused explicitly on evaluative research [89]. In some, but not all, such cases there was purposive selection. The study of the HTAs in Quebec showed over 85% had had an impact on policy [75]. The latter study is also one of the comparatively few to trace through from impact on policy to actual outcomes or benefits. It was suggested there had been cost savings of between $16 and $27 million annually. The Buxton and Hanney studies also attempted to trace through to the outcomes although this proved difficult. In one case, the evidence suggested that the research had strongly influenced the policy on heart transplantations. Buxton was able to estimate the increased number of QALYs (Quality-Adjusted Life Years) that resulted from the programme being properly funded and organised, as opposed to the counterfactual which might have been a less substantial and piece-meal development of heart transplantation in the UK [110].

While not specifically measuring levels of benefits, some of the studies have clearly shown an improvement in health equity as a result of policy changes: 'research has also played an important role in the expansion of Medicaid to poor pregnant women, young children, the elderly, and disabled' [111]. Other studies have not only demonstrated a major impact on policy but also been able to describe how research led to a paradigm shift [95].

The review of previous studies is now build upon in the following section as we develop our conceptual framework for assessing the level research utilisation in policy-making.

The interfaces between the health research system and policy-makers: priority setting and research commissioning

Increasing attention is focusing on the concept of interfaces between researchers and the users of research [6,7,14,29,93,112]. This incorporates the idea that there are likely to be different values and interests between the two communities [113], with their different time-frames [6,7], and that research is less likely to be utilised in a significant way unless networks and mechanisms are established at the interfaces. One version of the interfaces concept is presented in Figure 1. The 'permeability of the interfaces' [14] becomes important given the potential problems in the transmission of views and findings between researchers and policy-makers. Issues around interfaces need to be considered at various stages including priority setting, commissioning of research and communication of the findings.

The power relationship between publicly funded researchers and policy-makers may be described in terms of an exchange relationship [114]. The policy-maker receives new knowledge, and the testing of existing knowledge, in return for providing resources and public legitimacy. If the exchange becomes imbalanced, a reduction in the value of its outcomes becomes likely. Some of the analysis below attempts to identify both ways of enhancing the exchange, and the items upon which any assessment of utilisation should focus.

As shown in Figure 1, however, the picture is broader than this because many of the research findings flow into the pool of knowledge. Furthermore, some research that is potentially of use to policy-makers will not have been funded by them. This includes research from the international stock, which highlights the role of research as a global public good [3]. If a national system is to draw on this to maximum benefit, various interface mechanisms might be needed. This section, however, describes a mechanism specifically related to user-driven research, and the next section covers the broader interfaces.

It is not necessary here to describe all the expert approaches to research priority setting – see The 10/90 Report on Health Research, 2001–2002 [4] for a recent review. Given all the current activities, however, it is important to consider problems identified in previous attempts to enhance utilisation through priority setting [7]. Resistance to priority setting comes from those who adopt the 'internalist' view of research. They share Polanyi's opinion that the best science comes from the freedom of the researcher to pursue the priorities that emerge from the scientific imperatives [115]. Most now accept the contention, as voiced by Kogan and Henkel, that if health research is 'internalist and freely sponsored, the problem for government will be that of securing adequate brokerage with it...because it has not taken part in the setting of problems' [7].

In addition to the technical questions to do with how best to identify the most important priorities in terms of health needs, the utilisation aspects of the debate perhaps revolve around two key questions:

• are priorities being set that will produce research that policy-makers and others will want to use?

• are priorities being set that will engage the interests and commitment of the research community?

Research that Policy-makers will be more likely to use

Policy-makers have not always found it easy to identify their needs or to aggregate the demands from various sources [7]. Again, the limitations on the ends-means model of rationality must be recognised and it should not be assumed that sophisticated priority setting mechanisms will automatically produce research regarded as relevant by policy-makers. This is why it is so important that the methods described do incorporate stakeholder involvement and an iterative approach [4], and that, particularly when overseas agencies/researchers are involved, efforts are made to link the research to the priorities of the national policy-makers [21]. This should boost local ownership of the research. From the perspective of the policy-maker it is important that the research not only seems relevant, but also timely. Involvement in such priority setting is itself sometimes seen as a way of informing policy [7]. Any assessment of utilisation should include identification of policy-makers' attitudes towards the priorities set.

The ability of policy-makers to set priorities, and the likelihood of them using the eventual research findings, will probably be increased if they are able to develop long-term links with researchers. This is especially the case for researchers in centres where they can build up their own shared reservoir of knowledge on the key issues and discuss this with policy-makers [14,29,103]. In these circumstances, researchers help develop the policy-makers' views about what are the important issues that should be addressed by research. Crucially, this allows researchers to play an interactive role in shaping policy-makers perceived needs.

Priorities to engage the commitment of researchers

There is a danger that the more the agenda is set unilaterally by non-researchers, the less the research community will be committed to working on it. At the commissioning interface between priority setting and the funding of specific pieces of research, there is some scope for subtle defection from the agreed priorities [116]. It is possible that the move towards Mode 2 research [58] means that an increasing number of researchers are moving away from belief in the superiority of the internalist Mode 1 approach. Where the policy-makers are working with the researchers as suggested by Trist [57] and Gibbons et al [58] this could result in research that has more chance of being utilised, but much of Gibbons et al's analysis is not related to formal priority setting exercises. Iterative research commissioning processes [34,117] and priority setting [4] might be ways of addressing both problems identified in this section.

Finally, despite the importance of priority setting, there is no monopoly of wisdom and those who wield the enormous power of government do well to foster their own critics and counter-analysis [7]. Independent research can provide critical commentaries and alternative perspectives that are important for healthy policy-making in the long term.

The interfaces between the health research system and policy-makers: transfer of research to policy-makers

Much previous work stresses the importance of interactions between policy-makers and researchers in increasing the likelihood of attention being given to the knowledge produced. This continues the above discussion and fits especially well with Weiss's interactive model [19], and with the view that policy-makers are unlikely to take much notice of research if the first they know about it is when it arrives on their desk [89]. It is claimed that previous interaction increases the possibilities of the findings moving up the Ladder of Research Utilisation [105], and that the building of bridges between researchers and policy-makers is important and could be achieved by 'decision-linked research' [6,118].

The studies described earlier provide many examples to support this analysis, including discussion of 'linkage strategies' [104], and 'interactions' [11]. A cholera researcher is quoted in the Mexican study as saying: ' "if there isn't a good relationship between a researcher and a decision-maker...it is difficult for research results to be taken into account" ' [10]. Buxton's insider account of his own work evaluating the emerging UK heart transplantation programme illustrates the benefits that can come from close liaison with the potential users [110,119]. As a result of frequent liaison the Department of Health knew the likely results of the final report. Then, on the day it was received, a major decision was made to fund a full heart transplantation programme in the UK, the benefits of which were described earlier. This demonstrates that although building interactive relationships is often a long-term endeavour, it can result in rapid policy-making.

Some of the studies provide examples of how good interaction was achieved through informal communications as a result of deliberate actions by researchers or even through chance relationships [10]. Researchers themselves sometimes provide policy briefing for policy-makers, which is seen as a useful but underdeveloped approach [53]. The existence of researchers, or research responsive members, in policy networks can also be important. These can be international [120]. These types of observations are broadly supported by some of the three most commonly mentioned facilitators of the use of research in the 24 studies included in the systematic review. The three are: personal contact between researchers and policy-makers (13/24); the timeliness and relevance of research (13/24); and the inclusion of a summary with clear recommendations (11/24) [22].

The various actions of individuals can be important, but it is desirable to consider the role of the HRS in encouraging or facilitating interactions, networks and mechanisms at a system-wide level. Priority setting approaches are one such mechanism. The development of long-term research centres focusing on particular topics [10,14,29] is one of the potentially strongest ways a HRS can take action to increase the possibilities of research being used to inform policy. Here the concept, noted above, of 'epistemic communities' [48] is useful and has explicitly been applied to assessments of the benefits from health research centres [29]. Furthermore, accounts from various countries or provinces describe the importance being attached to the creation of an institute for health research. Examples include: Mexico [6]; Thailand [94]; Canada [121] and Manitoba, Canada [93]. The desirability of such institutes engaging with stakeholders is being addressed by the Alliance for Health Policy and Systems Research [122]. Once established, such links can build on mutual respect and help develop an understanding of the differing perspectives.

HRSs could also ensure long-standing committees or fora are formally established to allow scientists and policy-makers to discuss issues. These could operate at both interfaces – feeding into the priority setting, and ensuring key policy-makers are aware of relevant research. Such approaches have been used in various countries including the UK [7] and Burkina Faso [9]. Other brokerage mechanisms that could also be provided by the HRS include arranging seminars for policy-makers, and funding individuals to act as research brokers [7,80,123], or translators [12,82]. Such individuals, who may be in key knowledge management roles within the HRS, take the findings from researchers and bring them to the attention of policy-makers and others. It is useful to think of diffusion of the findings at several levels. In addition to directing findings at policy-makers within the health system, efforts at wider diffusion might also help build support for adoption of the findings.

Whatever the direction of the dissemination, however, mechanisms are needed that review and synthesise research and attempt to identify the research that should be promoted from that that should not. HRSs have a clear responsibility in this area in terms of funding such reviews and their dissemination; the latter through a range of mechanisms including the internet. It can go further than this, however, and the attempt to provide some structure, or 'knowledge warehouse' [29], to the pool or stock of knowledge should be seen as a key knowledge management function of the HRS. The international Cochrane Collaboration plays an important part in this, and was inspired by the UK Cochrane Centre that was a mechanism funded as part of the information system of the UK's HRS [124,125]. The need to use and develop databases of evidence, and reviews of research, has been explored in relation to preparing evidence to inform policies on the reduction of health inequalities [126].

Many, but not necessarily all, of the mechanisms for transmission of the relevant national and international research are the responsibility of the HRS to provide. Some of the above considerations are important in the interface between national health systems and international research and international bodies promoting health. In drawing conclusions from the COHRED studies, Chunharas comments: 'National research co-ordinating bodies, such as the ENHR mechanisms promoted by COHRED, can also play a mediating role to better foster research to policy linkages. International agencies too have an important contribution to make as intermediaries in linking knowledge and action' [110]. The integration of research into the health care programmes of international organisations can be an effective mechanism for research-informed policies to be brought about [21].

The role of policy-makers as receptors of research

There is increased recognition of the significance of policy-makers in their role as recipients, or receptors, of research [7,9,11,104,112,127,128]. Despite the low response rate to their questionnaires, the findings from Landry et al's study illustrate this point. They claim: 'factors such as dissemination and linkage mechanisms that are generally considered to be powerful explanatory factors and to be the most efficient targets for policy interventions are less important than factors such as the receptive capacity of users when one climbs from the stage of transmission to the higher stages in the ladder of knowledge utilisation. Future research must recognize that the same factors do not explain success at all stages of knowledge utilisation' [91].

Beyer and Trice [129] also set out a series of steps policy-makers go through when using research and this has been applied to health research [11,62]. Epistemological, social and institutional issues are all relevant to the role of the research receptor [7,128]. The types of research relevant to policy-making vary greatly. The key questions could be seen as a spectrum:

• is there research available that is either relevant to policy issues, or could help bring new issues onto the agenda?

• is such research being effectively brought to the attention of policy-makers in diverse positions within the health system?

• is the policy-making system capable of absorbing the research findings?

• are there situations where the policy-makers are willing, and able, to use it?

The HRS can assist here in the various ways described, but the wider policy system has a responsibility to create the right institutional mechanisms and staff capacity. Broadly, the responsibility of the HRS is greatest in the first part of this spectrum. It is recognised that it is much more difficult to make recommendations about how to increase the use of research in the development and implementation of policy, than it is to suggest how to improve communications [10]. There is, however, no neat division of responsibility. The main thrust of our analysis is that the issues need to be addressed on a system-wide basis, and that there is a series of measures the HRS can take to maximise the possibilities of research utilisation. These include encouraging policy-makers to see the benefits in general, and in specific situations, of using research to help build a policy environment which will result in improvements in the health system.

Institutional arrangements do matter [6,7]. A policy machine must face the problems involved in using research, some of which it will not have commissioned itself. It needs a capacity to decode the results of research or to discern a policy problem that might yield to disciplined enquiry. To some extent these needs might be met by the use of scientific or policy advisers from outside the policy-making body, but they may not have full access to the generative stage of policy development. Hence the need for internal brokerage. These might be officials with either a scientific or a professional or a policy-making background. The evidence is that, whatever their provenance, they may be able to assume the skills and value-set of boundary-crossers and research enablers [7]. Some have become famous for their ability to empathise with the needs, problems and potentials of researchers whilst enabling policy-makers to secure otherwise inaccessible skills and knowledge [130].

The response of policy-makers to research varies not just with the type of issues and research being dealt with, but also with the differing attitudes they adopt towards the whole policy-making process [113]. As individuals, some policy-makers are much more receptive to research than others. The issues are wider than individual preferences, however, and also depend on: how far the research accords with the political and social zeitgeist of the time [128]; the national political and administrative culture [10]; and the institutional arrangements for policy-making. The historical study from Uruguay demonstrates the detrimental effect military dictatorship can have on research utilisation [99]. By contrast, the study from South Africa illustrates how, despite the problems, the new political environment can help foster the better use of research in the policies related to some programmes [98]. There will be clear opportunities for research findings to have greater impact when they are in tune with the wider developments of the time, but there are also dangers that such research could sometimes be accepted and acted upon without sufficient analysis to test its validity.

There are variations in patterns of bureaucratic recruitment and other characteristics that can influence research utilisation. In the countries where the research and policy connections are strongest, the relationship has been enhanced by the fact that some of the senior administrators have had research experience or interests as part of their prior education [131,128]. This should make mutual institutionalisation of the relationship easier to secure. The willingness of officials to undertake policy analysis is seen as important [7]. In some systems specific policy analysis units [132], or think tanks of researchers [9], are established in health policy-making bodies. An important determinant of their success will be their position within the policy-making organisation.

Too often, however, officials in policy-making bodies are resistant to research because they display strong distrust of information generated outside the organisation or system [133]. Furthermore, the career patterns of policy-makers are often not compatible with strong research utilisation if the latter depends on developing long term relations to boost receptivity. Given the length of many research projects, the original sponsor of research is often not in place when the findings are reported. Patton, the arch proponent of making evaluations more likely to be used through being utilisation-focused, notes that the major problem with his approach is the frequent turn-over of the primary intended users [134]. Various studies support a greater emphasis on training of policy-makers, at least those in bureaucratic positions [9,10]. If such training fosters a more positive attitude towards the use of research findings, where relevant, in the policy-making system as a whole, this could mitigate some of the problems.

There will be situations, particularly where the policies are likely to be made at local level, where there is much less likelihood that the researchers will have the opportunity to develop an interactive relationship with potential policy-makers. Several consequences flow from this. As noted previously, the characteristics of specific pieces of research can become important determinants of its uptake. There is an onus on the HRS to ensure it identifies and publicises those characteristics of research that are likely to increase its appeal to policy-makers. It should encourage such research to be undertaken.

In some countries there are specific mechanisms that lead to the incorporation into policy instruments of research such as Health Technology Assessments (HTAs) and clinical trials. This is one of the reasons noted previously for the greater likelihood of HTAs making an impact. A collaborative working group examining these issues in Europe concluded that, whilst they were able to identify occasional examples of systematic integration of HTA in decision-making structures, there was no direct link between the amount of money spent on HTA and its impact on the decision-making process [67]. Indeed, they suggest that small programmes can be involved in the core of the policy-making structure whilst larger HTA programmes have difficulty in demonstrating impact.

It seems clear that HTAs have had most impact in those situations where there are specific mechanisms in place that require research evidence to support well-defined policy decisions on provision, coverage or reimbursement (and these impact on practice where there are further mechanisms to ensure local adherence to national policy). The European countries where there is some evidence of such integration include: Germany, Spain, Switzerland, Sweden and, despite our earlier example, the Netherlands [67]. Conversely, HTA has had much less impact where these specific mechanisms are not in place and policy-makers are exposed to HTA only in a diffuse or indirect manner.

Governments that set up what could be considered rational policy-making arrangements in which primacy is given to the role of research evidence might find the results face considerable criticisms in the media. Even with a population fully engaged in the cost effectiveness/rationing debate, there would still be scope for disappointed interests to campaign against decisions. This illustrates the desirability of an integrated approach to utilisation and an awareness of all the pressures on policy-makers.

In the context of the above discussions more attention should be given to the role of incentives, both for researchers to produce utilisable research [6] and for policy-makers, at the system or individual level, to pay attention to it. In an exercise of empirically based modelling, Bardach [56] assumed classical economic rationality on the part of individual policy-makers. He showed how research reaches those for whom its utility exceeds the disutility of obtaining it and noted that co-operative relationships grow up with research consumers when producers try to reduce the cost to them of obtaining information.

Engaging in 'useful' research produces some clear benefits for researchers. It may be a source of satisfaction that one's work is being taken notice of and contributing to the formation of policy or the improvement of practice. At present, however, it is widely thought that the traditional academic criteria still dominate the crucial assessments of research performance upon which career advancement and peer recognition depend [7,29,98,135,136]. The assessment of utilisation, therefore, could become a key issue if rewards are to focus on relevance as well as research excellence [6,137].

The interfaces and receptor model

Any assessment of the utilisation of health research in policy-making has to integrate two factors: an awareness of the wider influences on policy-makers and a detailed analysis of the specific ways the HRS could contribute to improving the health system through providing the research to inform policies. An appropriate model for assessing research utilisation in policy-making is also likely to be one that combines both an emphasis on the importance of actions at the interfaces and an analysis of the role of receptors. As we have seen there are many models already in existence. We are proposing an interfaces and receptor model because it allows a range of key issues to be integrated into the analysis. These include:

A focus on the need for multi-layered analysis

Multi-directional interactions with practitioners and the public are important for policy-makes and augment the crucial interface, for research utilisation in policy-making, between the HRS and the policy system. As noted above, this interface itself has various dimensions including: priority setting; research commissioning; and the transfer of research findings to policy-makers.

An appreciation that both researchers and policy-makers have their own values and interests

Therefore, for example, priority setting has to be sophisticated to maximise the likelihood that the research community will be engaged on a research agenda producing knowledge that the policy-makers will use. Similarly, just because research centres undertake large scale dissemination does not necessarily guarantee their research will be utilised [11]. Hence the importance of analysis that goes beyond examining dissemination and considers the nature of productive interactions and the characteristics of research to which receptors are responsive.

An emphasis on the role of the receptor

This is necessary because ultimately it is up to policy-makers to make the decisions; this can be a convoluted process with many stages at which research could potentially have a role. Again as described above, there are various features of the organisation and training within the receptor (or policy-making) body that can enhance the utilisation of research. Even though responsibility lies with the receptors, the HRS should take every action possible to facilitate the use of the research. These are important considerations for any assessment of the success of the HRS in relation to utilisation. First, because they highlight the wider political context which is beyond the control of the HRS. Second, because they still leave room for assessments of the activities of the HRS, within its given context, to increase the permeability at the interfaces [14] and thus promote the uptake of the research findings by the receptors.

An approach that facilitates analysis of the key paradox highlighted by the systematic review

Innvær et al concluded that, 'two-way personal communication, the most common suggestion, may improve the appropriate use of research evidence, but it might also promote selective (inappropriate) use of research evidence' [22]. This potential problem can be addressed in several ways through the interfaces and receptor model. First, links between researchers and policy-makers should ideally develop on a long-term basis so that together at the priority setting interface they produce a research agenda that reflects some synthesis between the needs of policy-makers and the perspectives of independent research analysis. Second, the interfaces and receptor model emphasises the importance of the role of organisational and training issues such as the need for capacity to undertake systematic reviews and policy analysis within any system. While such capacity is seen as a way of enhancing the ability of the receptors to absorb research, it should also allow proper analysis of all evidence to be undertaken.

There could, therefore, be value in having assessments of utilisation that integrate the modelling of research utilisation with the epistemological, social and institutional analysis [7] inherent in concepts such as interfaces and receptor functions. This might contribute to future research policies and strategies in such a way as to promote greater utilisation.

Purposes of assessing the utilisation of research in health policy-making

Before showing how all the previous analysis could be built upon in the generation of appropriate tools for the assessment of the nature and extent of knowledge utilisation in health policy-making, it is desirable to consider the purposes of such assessments. The purpose of the assessment is likely to differ depending on the level at which it is conducted.

Buxton and Hanney [14,15] identified three main reasons for undertaking their case study, and more general, assessments of the benefits from research:

• justifying spending resources on health research;

• assisting with the prioritisation of future expenditure;

• indicating ways to improve the conduct and management of research so as to increase the likelihood or magnitude of subsequent beneficial consequences.

These considerations are particularly relevant when the assessment is related to the justification of, and accountability for, funding at a national level, even if the case studies are conducted at project or research unit level. For a body such as the WHO, there could well be an important role in conducting such assessments with the aim of providing evidence of the possibility of the effective use of research resources. This could support advocacy for greater resources to be made available for health research. Such advocacy has recently been powerfully made as part of the report from WHO's Commission on Macroeconmics and Health [138]. This report is seen as convincing [139], and thus perhaps is helping to generate a more promising climate in which research utilisation could be assessed. Cross-national studies of research utilisation around common themes might be the best way to conduct assessments that could illustrate effective ways in which health research can be used. Understanding could be gained from the comparisons between and within countries. The potential link with advocacy would be strengthened if the policies on which the studies were based were specifically in those areas where the Commission is calling for increased research funding. These areas include: reproductive health, maternal and child health, tropical diseases, and health systems research.

We noted previously the increasing WHO focus on the importance of research informing key policy areas [3,21]; this perspective is shared by WHO regions, for example, in relation to policies for improving health equity [140]. In this context it is important to recognise the claim, made in section 8, that assessment can influence the activities given priority by researchers. This is likely to be particularly relevant when the focus of the evaluation is the performance of specific research units, teams, or even individuals, especially when funding is at stake. Given this, it is argued that moves towards giving more importance to the assessment of utilisation of health research should help encourage researchers to devote effort to activities likely to stimulate impact, and reward those who are already doing so [7,29,136]. The greater the significance of the assessment, however, the more dispute there will be over the methods to use.

In particular, the role of numerical indicators needs to be considered in relation to the purposes of the assessment. It is argued that if the indicators used in performance evaluation lack 'decision relevance' they are ignored [141]. The introduction of performance indicators into a process such as research may, however, have a dysfunctional impact unless great care is taken to establish the purposes and likely consequences of assessments [13,18]. For example, an assessment system that resulted in more dissemination in general, as opposed to more targeted dissemination of relevant knowledge, would be repeating the dangers of increasing the overload on policy-makers [105]. Where indicators are involved, they can be used as either 'dials' to measure inputs and outputs accurately, or as 'tin openers' to identify issues needing further examination or to aid judgement [142]. Although the use of numerical indicators as dials has been advocated by some, in an area such as the assessment of research and its utilisation in policy-making, where measurements are so difficult to make, caution is usually recommended [13,18,142]. It would seem only sensible to use indicators as tin openers to aid judgements when the purpose of the assessment involves funding decisions.

Even when funding is not an issue, if any comparisons, especially international, are to be made, there would be dangers in using simple indicators outside of a wider qualitative assessment. They would become de-contextualised. The long-standing fears about such an assessment process include the danger of manipulation through collusion and the difficulty of making comparisons across programmes with a different composition of user groups [13]. Depending on the purpose of the assessment there is, nevertheless, scope for innovative thinking in terms of methods.

Methods for assessment of research utilisation in policy-making

Appropriate methods for assessment therefore have to be developed to reflect:

• the purposes for which the assessment is to be conducted, for example, to increase accountability, or to support advocacy for health research;

• the analysis about the various types of research, the range of utilisation possibilities, and the wider conceptual frameworks, for example, the interfaces and receptors model; and

• the different roles that can be played by retrospective assessments and ones that focus on the current position.

Various lists have been produced of the type of information that could be gathered to produce numerical indicators to inform either self-evaluation/peer review of research teams [136], or to inform regular monitoring of the benefits from work originating from a particular health research funder [12,18]. Items from these lists relevant for policy-making include a numerical record of: presentations to policy-makers; production of fact sheets; membership of advisory committees; and membership of committees issuing a policy document or a treatment guideline. These are not really measures of actual impact and although one such measure, references in policy publications, was also proposed, the list would probably need to be supplemented; in the case of regular monitoring, for example, by a set of structured case studies. When an evaluation within a country is to be used for making funding decisions, it would be unwise to use the numerical indicators as dials because of the contextual issues and possible biases described above. Instead they should be used to inform judgements.

Nor would it be sensible to use such indicators in any cross-national comparative study unless they were informing wider qualitative studies. Furthermore, to understand the peculiar difficulties of using raw questionnaire data in relation to assessing research utilisation in policy-making on specific issues, it is helpful to return to the definition of policy-making given earlier. This emphasised that those who make policies are in a particular, authoritative, position. This presents a rather different set of circumstances from those encountered when assessing utilisation of specific findings by practitioners and members of the public. In such cases a sample might be thought to be representative of a wider group, and individual characteristics and circumstances might even out within the sample. Moreover, it could be claimed that the opinions of each practitioner or member of the public are equally valid as regards the influence of research upon their own behaviour. In a study of policy-making on a specific issue, by contrast, the interviewees or questionnaire respondents will be likely to include some representatives from relevant interest groups, commentators, and researchers as well as policy-makers with varying degrees of involvement in different aspects of the making of that policy. In relation to understanding the processes involved in the policy-making, therefore, the respondents might have conflicting views that do present truthful representations of what people saw and heard. Nevertheless, depending on the respondent's degree of involvement with the specific events under consideration, these views are likely to be of varying validity in relation to providing an account of the key actions.

Such complexities no doubt help explain why qualitative interviewing and documentary analysis were used most frequently the research utilisation studies described in the review of previous work. Questionnaires could provide some information from a wider range of informants than it might be possible to interview. They could also be used to help identify aspects on which to focus detailed parts of the interviewing. In-depth interviews, however, are widely seen as the most appropriate method when there is a need to unravel situations with diverse layers and subtle nuances. According to Rossi et al, 'whereas written surveys and questions might be useful for some limited purposes, that approach lacks the flexibility to tailor the line of discussion to the expertise of the individual, probe and explore issues in depth, and engage the informant in careful reflection' [143]. The growth of health policy and systems research suggests there is an increasing number of researchers who could undertake such interviewing [122].

Our review of previous studies demonstrated the great difficulties of making generalisations about specific factors associated with high levels of utilisation. To address this in any cross-national initiative it would be useful to adopt several strategies. First, as far as possible, structure all the assessment studies around a conceptual framework such as the interfaces and receptor framework presented earlier in the report. The framework is probably sufficiently broad to allow it to be applied to many situations. It would, nevertheless, help inform any interview schedule so as to ensure the questions were focusing on both how research findings were communicated across the interfaces, and the degree of policy-maker receptivity to them. This would be done not to provide a check-list of items that it is expected would all have to be present if research utilisation is to be achieved. Rather, it would be so that the interview covered a range of items, some of which might emerge as the reasons linked to utilisation, or lack of it, in each particular study.

The second strategy would be to base the studies on common policy themes as far as possible. Possible specific topics within the areas identified in the previous section include multi-drug therapy for leprosy and equitable access to health services. For each of the common themes, a key body of international research would be identified and some of the analysis would relate to that, and some to the impact of the full body of research available to policy-makers in the specific country. Some of the potential purposes the WHO might have in conducting such a cross-national assessment were set out above. An approach that uses common elements in several detailed studies, but which also expects each study to produce its own narrative or story of what caused research utilisation in the particular situation and context, has similarities to broader approaches to the study of innovation and organisational change [144].

Analysis of documents and semi-structured interviews would appear to be appropriate methods to use in a retrospective study of research impact on policy-making related to specific issues, especially where the policy is made at national or sub-national government level. Indeed, the recently conducted systematic review recommended that future research in this field, 'should combine interviews with document analysis' [22]. Questionnaires could also have a role, particularly in securing a wide range of opinions about the current situation regarding knowledge sources for research utilisation in policy-making and the relevant HRS mechanisms. A combination of these approaches would provide triangulation of methods and data-sources. The account below focuses particularly on four main elements of the recommended methods for the retrospective part of the policy-making element of the Structured Cross-national Thematic Studies that could be undertaken in the WHO research utilisation project [24]:

• documentary analysis;

• interviews;

• application of scales reporting the level of research utilisation in policy-making;

• overall analysis.

Documentary analysis

Documentary analysis would be undertaken in each study. Initially it would be used in an attempt to identify the degree of consistency between the policy in the country and the body of international research that is being centrally collated by the WHO utilisation project team. Further documentary analysis would also cover issues such as how far policy-makers drew on research findings in speeches during the policy formulation and implementation stages, and accounts in reports from research funding bodies of their efforts at developing mechanisms to enhance research utilisation. The documentary data-sources would include: research publications and reports; legislation; administrative/executive regulations or orders; reimbursement arrangements; guidelines/advice; meeting reports and minutes (if available); policy statements, speeches, and articles; and reports from research funding bodies. A draft protocol for the first element of the documentary analysis has been prepared (see 1: Elements of a protocol for documentary analysis).

Additional File 1. Elements of a Protocol for Documentary Analysis

Format: DOC Size: 22KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Interviews

A stakeholder analysis could identify whom to interview first, and then snowball techniques, together with review of the questionnaires, would ensure other key people were approached for an interview. In devising the semi-structured interview schedule to be used for all interviews, in all the countries participating in a cross-national study, it would be most important to allow interviewer flexibility. This would be necessary to deal with local circumstances and with situations, as described above [79], where the interviewee has a much more limited conception of research informed policy than the interviewer. Despite these caveats it would also be desirable to develop a semi-structured interview schedule that covered as many as possible of the points discussed in the previous analysis. A draft semi-structured interview schedule has been developed (see 2: Draft interview schedule for assessing research utilisation in policy-making), but it would have to be administered with considerable flexibility.

Additional File 2. Draft Interview Schedule for Assessing Research Utilisation in Policy-Making

Format: DOC Size: 27KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

The interviews would allow:

• comparability across themes and countries yet sensitivity to specific contexts;

• detailed investigation of the level of research impact, in relation to the particular issue, on the three stages of the policy-making process: agenda setting; policy formulation; and implementation;

• rolling triangulation ie using later interviews to test information gathered during earlier ones;

• investigation of key HRS and other mechanism that operated at the interfaces to enhance the responsiveness of the receptors, including: priority setting and research commissioning mechanisms; the creation of research centres and facilitating links with policy-makers; encouraging and funding research brokerage/translator/promoter activities; encouraging and funding reviews and syntheses of relevant research findings and the production of policy briefs; and facilitating interaction between researchers and policy-makers at long-standing committees or one-off seminars etc;

• investigation of a wide range of other relevant issues: the role of key institutions and their mechanisms, such as policy analysis, for absorbing research and their exposure to forces in addition to research findings; the responsiveness of policy-makers to different types and sources of research knowledge; the features of specific research findings that made policy-makers more responsive to their findings; the aspects of policy-making where research was seen as most valuable; the role of networks, international bodies, practitioner and advocacy groups, NGOs, the media and the public in bringing research findings into the policy debate; and developments in the wider political system;

• collection of data for the wider assessment in the overall utilisation project about how far any research-informed policy formulation and implementation was contributing to an increase in any of the final outcomes such as health and health equity gains.

Application of scales describing the level of research utilisation in policy-making

Whilst there are reservations about the extent to which numerical indicators should be used for cross-national comparisons, it is possible to see how the type of exercises undertaken by Buxton and Hanney [106], Jacob and McCregor [75] and Lavis et al [11] could be built upon. It might be possible to develop indicators in the form of descriptive scales of the degree of utilisation. These would be used to give an account of the impact of research on the policy-making in the specific context of each of the countries participating in the WHO research utilisation project. In the three studies cited above, the scoring or coding for each example was undertaken by the same team. Even clearer agreement about interpretation of scales would be necessary in an international exercise. Before starting any initiative, it would be desirable for the scales to be agreed between the assessment teams in the participating countries.

The previous analysis indicates that it would probably be appropriate to consider developing four scales to apply to research utilisation in each policy area. The first scale would focus on a slightly narrower range of research that, as noted above, would be the international research. This would examine the consistency between the research and the policy. Previous studies illustrate, however, that consistency with research findings does not necessarily demonstrate that the particular findings influenced the policy [14]. Where the policy consists of a clinical guideline developed by a professional group there could be circumstances in which the first scale on the degree of consistency, based on documentary analysis, might be the only scale appropriate to apply. In such circumstances the analysis should probably concentrate on the quality of the evidence used in the guidelines [145].

The remaining three scales would each relate to assessing the actual role played by research in each of the three phases of policy-making described previously. The relationship between policy-making and research is often messy and varied. Therefore, it is inevitable that some research might play a part in only one of these three phases, but other research might play several roles. For example, epidemiological research might cause an issue to be placed on the policy agenda, other research that developed a specific way of improving treatment could be used in policy formulation, but might also have helped force the issue onto the policy agenda by showing improvements were possible. The details of each scale have been prepared (see 3: Draft scales of the level of research utilisation in health policy-making). The key issues covered in them are described here:

Additional File 3. Draft Scales of the Level of Research Utilisation in Health Policy-Making

Format: DOC Size: 26KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

(i) Consistency of policy with research findings

This scale would relate to how far the content of the policy on issue X was in agreement with the findings from a defined body of international research (irrespective of the actual degree of influence of research on the policy formulation). It would initially be applied during the documentary analysis.

(ii) Degree of influence of research on policy agenda setting

This scale would relate to the extent to which research (including local research) had been responsible for getting the issue onto the policy-makers' agenda. It would cover research that: either showed the existence/extent of a problem; or was so dramatic/decisive that it instigated action to be taken to turn it into policy; or contained findings/theoretical frameworks that gradually changed the perception of policy-makers and others as to the importance of the issue in a process of enlightenment. It would rely on interviews, questionnaires and documentary analysis.

(iii) Degree of influence of research on policy formulation

This scale would relate to the actual influence the research had in the policy formulation process. It would aim not only to confirm any instrumental use of the research (ie direct use of the findings or research theories in formulating the content of the policy) but also to capture examples of the much wider range of possible impacts on policy, including the gradual sedimentation of insights, theories, concepts and perspectives in the enlightenment mode. This scale would consider the utilisation of research both in the actual development of the policy content, and in policy discussions and debates. The scale would be based primarily on the data from the interviews, but also use survey and documentary data.

(iv) Degree of influence of research on policy implementation

The key issues for this scale would be the use of research in assisting implementation, either through findings which are used to inform decisions about how best to implement the policy, or by providing justification of the policy and being used to generate support for it in terms of financial resources, political commitment, and public opinion. The scale would be based on data primarily from interviews and documentary analysis.

Overall analysis

The interviews, questionnaires, and documentary analysis should also provide material to help identify the relative importance, in relation to the level of utilisation recorded, of each of the HRS mechanisms listed in the bullet point above. The types and sources of research used, and reasons for their use, should also be recorded and attempts made to correlate them with the previous priority setting approaches. It would be appropriate to enhance the internal validity of the judgements about the list and the scales by discussing the emerging findings with the respondents. The account of each study would also involve description both of the value given to research in the country and of the broader cultural and socio-political environment, to the extent that they seem relevant to the degree of research utilisation achieved.

The findings from the assessments in each participating country could be collated. For each research theme the analysis would compare two sets of data: the scales for level of utilisation in each country, and the contextualised lists of the HRS activities and other mechanisms and networks thought to be important. Organising the studies around common themes might assist assessment of how far the use of the international stock of knowledge was dependent on local research.

As noted previously, although the account here has focused on research impact on policy-making, the evaluations would be stronger as part of a wider analysis covering research utilisation and interactions with practitioners, industry and the public. The fuller analysis would be both most useful in itself, and provide greater understanding of the environment in which the policy-making occurred. By building on the framework described in Figure 1, it should provide a holistic approach [112] to these issues. Thus, the WHO research utilisation project was conceived as an integrated whole in which retrospective assessment of research utilisation in policy-making would examine one step in a process that should eventually lead to health and health equity gains [24].

Given appropriate and targeted topic and country selection, this approach is likely to meet the purpose of using structured methods to provide examples of effective research utilisation. It should contribute towards enhanced understanding of the issues and could provide the basis of an assessment tool which, if used widely in countries, could give a boost to the importance attached to the utilisation of health research.

Conclusions

Increasing global attention is focusing on ways to improve health systems and the contribution that research-informed policies can make to this. It has long been recognised that a range of factors is involved in the interactions between health research and policy-makers. The emerging focus on Health Research Systems (HRS) has identified additional mechanisms through which greater utilisation of research could be achieved. Assessment of the role of health research in policy-making is best undertaken as part of a wider study that also includes utilisation of health research by industry, medical practitioners, and the public.

The utilisation of health research in policy-making should eventually lead to desired outcomes, including health gains. Research can make a contribution in at least three phases of the policy-making process: agenda setting; policy formulation; and implementation. Descriptions of these processes, however, can over-estimate the degree of rationality in policy-making. Therefore, the analysis should be informed by a review of the full range of policy-making models. Various categories of research are likely to be used differently in health policy-making. Applied research might be more readily useable by a policy system than basic research, but health policy-makers tend to relate more willingly to natural sciences than social sciences. There also appears to be a greater chance of research being used in clinical policies about delivering care to patients, than in national policies on the structures of the health service.

Models of research utilisation in policy-making start with a link to rational or instrumental views of policy-making, and include descriptions of how commissioned research can help to find solutions to problems. Other models relate to an incrementalist view in which policy-making involves a series of small steps over a long period; research findings might gradually cause a shift in perceptions about an issue in a process of 'enlightenment'. Interactive models of research utilisation stress the way in which policy-makers and researchers might develop links over a long period. Research can also be used symbolically to support decisions already taken.

Many previous studies of research utilisation can provide lessons for future assessments. Two broad approaches can be identified. Some studies start with pieces, or programmes, of research and examine their impact. Others consider policy on a particular topic and assess the role of research in the policy-making. To facilitate comparison, studies of research utilisation are best organised around a conceptual framework. Despite that, the influence of contextual factors in different settings makes it difficult to generalise. The two methods used most frequently, and usually together, come from the qualitative tradition: documentary analysis and in-depth interviews. Questionnaires, bibliometric analysis, insider knowledge and historical approaches have all been applied. A few recent studies have attempted to score or scale the level of utilisation. The examples suggest there is a greater level of utilisation and final outcomes in terms of health, health equity, and social and economic gain than is often assumed, whilst still showing much underutilisation. There is considerable variation in the degree of utilisation, both within and between studies.

Increasing attention is focusing on the concept of interfaces between researchers and the users of research. This incorporates the idea that there are likely to be different values and interests between the two communities. At the prioritisation interface there are two key questions: whether priorities are being set that will produce research that policy-makers and others will want to use, and whether priorities are being set that will engage the interests and commitment of the research community.

Interactions across the interface between policy-makers and researchers are important in transferring research to policy-makers. This fits especially well with the interactive model of utilisation. Actions by individual researchers can be useful in generating interaction, but it is desirable to consider the role of the HRS in encouraging or facilitating interactions, networks and mechanisms at a system-wide level. The HRS could provide funding and organisational support for various items including: long-term research centres; research brokerage/translator mechanisms; the creation of official committees of policy-makers and researchers; and mechanisms for review and synthesis of research findings.

There is increased recognition of the significance of policy-makers in their role as the receptors of research. In relation to the perspective of policy-makers there is a spectrum of key questions. These range from whether relevant research is available and effectively being brought to their attention, to whether they are able to absorb it and willing to use it. The HRS has a responsibility, especially in the early parts of the spectrum, but the wider health system also has a responsibility to create appropriate institutional mechanisms and ensure there are staff willing and able to incorporate relevant research. More attention should be given to the role of incentives. The assessment of utilisation becomes a key issue if rewards are to focus on relevance as well as research excellence.

An appropriate model for assessing research utilisation in policy-making combines analysis of two issues: the role of receptors and the importance of actions at the interfaces. An emphasis on the role of the receptor is necessary because ultimately it is up to the policy-maker to make the decisions. Any assessment of the success of the HRS in relation to utilisation must accept that the wider political context is beyond the control of the HRS, but consider the activities of the HRS, within its given context, to enhance the utilisation of research by increasing the permeability of the interfaces.

The reasons for assessing the utilisation of research in policy-making include: advocacy, accountability, and increased understanding. For the World Health Organization there could be a role in conducting such assessments with the aim of providing evidence of the effective use of research resources. This could support advocacy for greater resources to be made available for health research. It is important that the purposes of any assessment are taken into account in planning the methods to be used.

Previous studies demonstrated the difficulties of making generalisations about specific factors associated with high levels of utilisation. To address this in any cross-national WHO initiative involving a series of studies in a range of countries, it would be desirable to structure all the studies around a conceptual framework (such as the interfaces and receptor framework considered here) and base the studies in each country on common themes. These could include policies for the adoption of multi-drug therapy for treating leprosy, and for the equitable access to health services.

Analysis of documents and semi-structured interviews would be appropriate methods in each study assessing the role of research in policy-making on a specific policy theme. Surveys could also have a role. These approaches would provide triangulation of methods and data-sources and should also provide material to help identify the relative importance, in relation to the level of utilisation recorded, of the HRS mechanisms described in the previous analysis. The types and sources of research used, and reasons for their use, should also be recorded and attempts made to correlate them with the previous priority setting approaches. It is expected that each study will produce its own narrative or story of what caused utilisation in the particular context, but the data gathered could also be applied to descriptive scales of the level research utilisation. The four scales could cover the consistency of policy with research findings, and the degree of influence of research on agenda setting, policy formulation, and implementation.

The findings from the assessments in each participating country should be collated. For each policy theme or topic the analysis would compare two sets of data: the scales for level of research utilisation in each country, and the contextualised lists of the HRS activities and other mechanisms and networks thought to be important. Although the account here has focused on research impact on policy-making, the evaluations would be stronger as part of a wider analysis covering research utilisation and interactions with practitioners, industry and the public.

Given appropriate and targeted topic and country selection, this approach is likely to meet the purpose of using structured methods to provide examples of effective research utilisation. The approach should contribute towards enhanced understanding of the issues and could provide the basis of an assessment tool which, if used widely in countries, could lead to greater utilisation of health research.

Competing interests

SH, MB, and MK were funded for this study by the Research Policy and Co-operation Department of the World Health Organization. MG-B is manager of the Alliance for Health Policy and Systems Research.

Authors' Contributions

All authors were involved in devising the structure of the article and contributed text. MG-B supplied Figure 2 and accompanying text. SH drafted and developed the article.

Acknowledgements

This article is based on a report commissioned by the Research Policy and Co-operation (RPC) Department of the World Health Organization, Geneva, whose funding is gratefully acknowledged. The report is intended as a background paper to contribute to the health research utilisation project that is part of the broader Health Research Systems Analysis (HRSA) Initiative being undertaken by RPC/WHO. This initiative will inform the World Health Report 2004, Health Research: Knowledge for Better Health. The authors thank the members of the team from WHO working on these issues, especially Shyama Kuruvilla and Tikki Pang, for their generous encouragement and constructive comments. Most helpful comments were also kindly provided by Andrew Pleasant, Mary Henkel and reviewers used in the journal's open review process: Jonathan Grant and Justin Keen. We are also grateful for the helpful comments made on an earlier version of the report by participants at the User-driven Health Research workshop organised by the Alliance for Health Policy and Systems Research/Tufts University School of Medicine at Talloires, France, in September, 2002. In particular, we are grateful for the review from John Lavis, a member of the utilisation project's panel of experts.

Steve Hanney and Martin Buxton are also funded by the UK Department of Health's Policy Research Programme. Miguel Gonzalez-Block gratefully acknowledges funding for the Alliance for Health Policy and Systems Research from the governments of Norway and Sweden, the World Bank and International Development Research Council of Canada.

The opinions expressed are those of the authors alone and responsibility for them does not lie with the funding bodies.

References

  1. Murray C, Frenk J: A framework for assessing the performance of health systems.

    Bull World Health Organ 2000, 78:717-731. PubMed Abstract OpenURL

  2. World Bank: The Costs of Attaining the Millennium Development Goals. [http://www.worldbank.org/about/whatwedo/mdgs.htm] webcite

    accessed:06/04/02

  3. Ad Hoc Committee on Health Research Relating to Future Intervention Options: Investing in Health Research and Development.

    Geneva, World Health Organization 1996. OpenURL

  4. Global Forum for Health Research: The 10/90 Report on Health Research 2001–2002.

    Geneva, World Health Organization 2002. OpenURL

  5. COHRED Working Group on Research to Action and Policy: Lessons in Research to Action and Policy – Case Studies from Seven Countries.

    Geneva, The Council on Health Research for Development 2000. OpenURL

  6. Frenk J: Balancing relevance and excellence: organizational responses to link research with decision-making.

    Soc Sci Med 1992, 35:1397-1404. PubMed Abstract OpenURL

  7. Kogan M, Henkel M: Government and Research.

    London, Heinemann 1983. OpenURL

  8. Davis P, Howden-Chapman P: Translating research findings into health policy.

    Soc Sci Med 1996, 43:865-872. PubMed Abstract | Publisher Full Text OpenURL

  9. Sauerborn R, Nitayarumphong S, Gerhardus A: Strategies to enhance the use of health systems research for health sector reform.

    Trop Med Int Health 1999, 4:827-835. PubMed Abstract | Publisher Full Text OpenURL

  10. Trostle J, Bronfman M, Langer A: How do researchers influence decision-makers? Case studies of Mexican policies.

    Health Policy Plan 1999, 14:103-114. PubMed Abstract | Publisher Full Text OpenURL

  11. Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, Abelson J: Examining the role of health services research in public policymaking.

    Milbank Q 2002, 80:125-154. PubMed Abstract | Publisher Full Text OpenURL

  12. Richardson A, Jackson C, Sykes W: Taking research seriously.

    London, Department of Health 1990. OpenURL

  13. Cave M, Hanney S: Performance indicators for higher education and research. In Output and Performance Measurement in Government. Edited by Cave M, Kogan M, Smith R. London, Jessica Kingsley Publishers; 1990:59-85. OpenURL

  14. Buxton M, Hanney S: How can payback from health research be assessed?

    J Health Serv Res Policy 1996, 1:35-43. PubMed Abstract OpenURL

  15. Buxton M, Hanney S: Evaluating the NHS Research and Development Programme: will the programme give value for money?

    J R Soc Med 1998, 91(suppl 35):2-6. PubMed Abstract OpenURL

  16. Cozzens S: The knowledge pool: measurement challenges in evaluating fundamental research programs.

    Eval Program Plann 1997, 20:77-89. OpenURL

  17. Buxton M, Hanney S, Packwood T, Roberts S, Youll P: Assessing benefits from Department of Health and National Health Service research and development.

    Public Money and Management 2000, 20:29-34. OpenURL

  18. Croxson B, Hanney S, Buxton M: Routine monitoring of performance: what makes health research and development different?

    J Health Serv Res Policy 2001, 6:226-232. PubMed Abstract | Publisher Full Text OpenURL

  19. Weiss C: The many meanings of research utilization.

    Public Adm Rev 1979, 39:426-431. OpenURL

  20. Gonzalez-Block M: Networks focusing on policies and cross-cutting issues affecting health: Alliance for Health Policy and Systems Research.

    In: The 10/90 Report on Health Research 2001–2002 (Global Forum for Health Research) Geneva, World Health Organization 2002, 193-200. OpenURL

  21. Child and Adolescent Health and Development/World Health Organization: Child Health Research: A Foundation for Improving Child Health.

    Geneva 2002. OpenURL

  22. Innvær S, Vist G, Trommald M, Oxman A: Health policy-makers' perceptions of their use of evidence: a systematic review.

    J Health Serv Res Policy 2002, 7:239-244. PubMed Abstract | Publisher Full Text OpenURL

  23. Ham C, Hunter DJ, Robinson R: Evidence based policymaking.

    BMJ 1995, 310:71-72. PubMed Abstract | Publisher Full Text OpenURL

  24. Hanney S, Gonzalez-Block M, Kogan M, Buxton M: The Utilisation of Health Research in Policy-making: Concepts, Examples and Methods of Assessment – A Report to the Research and Co-operation department, World Health Organization, Geneva.

    HERG Research Report No 28. Uxbridge, Health Economics research Group, Brunel University 2002. OpenURL

  25. Smith R: A national health research policy.

    BMJ 1988, 297:805-806. PubMed Abstract OpenURL

  26. Commission on Health Research for Development: Health Research: Essential Link to Equity in Development.

    New York, Oxford University Press 1990. OpenURL

  27. World Health Organization: National Health Research Systems – Report of an International Workshop.

    Geneva 2002. OpenURL

  28. Easton D: The Political System.

    New York, Alfred A Knopf 1953. OpenURL

  29. Hanney S, Packwood T, Buxton M: Evaluating the benefits from health research and development centres: a categorization, a model, and examples of application.

    Evaluation: The International Journal of Theory, Research and Practice 2000, 6:137-60. OpenURL

  30. Haynes B, Haines A: Barriers and bridges to evidence-based clinical practice. In Getting Research Findings into Practice. Second edition. Edited by Haines A, Donald A. London, BMJ Books; 2002:115-122. OpenURL

  31. Gilson L, Bowa C, Brijlal V, Doherty J, Antezana I, Daura M, Mabandhla M, Masiye F, Mulenga S, Mwikisa C, et al.: The Dynamics of Policy Change: Lessons From Health Financing Reform in South Africa and Zambia.

    Bethesda, Partnerships for Health Reform Project, Abt Associates Inc 2000. OpenURL

  32. Davies H, Nutley S: Healthcare: evidence to the fore. In What Works? – Evidence-based Policy and Practice in Public Services. Edited by Davies HTO, Nutley SM, Smith PC. Bristol, The Policy Press; 2000:43-67. OpenURL

  33. Klein R: From evidence-based medicine to evidence-based policy.

    J Health Serv Res Policy 2000, 5:65-66. PubMed Abstract OpenURL

  34. Black N: Evidence based policy: proceed with care.

    BMJ 2001, 323:275-279. PubMed Abstract | Publisher Full Text OpenURL

  35. Kogan M, Tuijnman A: Educational Research and Development: Trends, Issues and Challenges.

    Paris, OECD 1995. OpenURL

  36. Nutley S, Webb J: Evidence and the policy process. In What Works? – Evidence-based Policy and Practice in Public Services. Edited by Davies HTO, Nutley SM, Smith PC. Bristol, The Policy Press; 2000:13-41. OpenURL

  37. Walt G: Health Policy: An Introduction to Process and Power.

    London, Zed Books 1994. OpenURL

  38. Booth T: Developing Policy Research.

    Aldershot, Gower 1988. OpenURL

  39. Simon H: Models of Man: Social and Rational.

    London, John Wiley 1957. OpenURL

  40. Lindblom C, Cohen D: Usable Knowledge: Social Science and Social Problem Solving.

    New Haven, Yale University Press 1979. OpenURL

  41. Braybrooke D, Lindblom C: A Strategy of Decision.

    New York, Free Press 1963. OpenURL

  42. Lindblom C: The science of muddling through.

    Public Adm Rev 1959, 19:79-88. OpenURL

  43. Weiss C: Knowledge creep and decision accretion.

    Knowledge: Creation, Diffusion, Utilization 1980, 1:381-404. OpenURL

  44. Rhodes RAW, Marsh D: New directions in the study of policy networks.

    European Journal of Political Research 1992, 21:181-205. OpenURL

  45. Kogan M, Hanney S: Reforming Higher Education.

    London, Jessica Kingsley Publishers 2000. OpenURL

  46. von Walden Laing D: HIV/AIDS in Sweden and the United Kingdom: Policy Networks 1982–1992.

    Stockholm, Department of Political Science, University of Stockholm 2001. OpenURL

  47. Yin R, Gwaltney M: Knowledge utilization as a networking process.

    Knowledge: Creation Diffusion Utilization 1981, 2:555-580. OpenURL

  48. Haas PM: Introduction: epistemic communities and international policy coordination.

    Int Organ 1992, 46:1-35. OpenURL

  49. Alford RA: Health Care Politics.

    Chicago, University of Chicago Press 1975. OpenURL

  50. Rhodes RAW: Beyond Westminster and Whitehall.

    London, Unwin-Hyman 1988. OpenURL

  51. Wistow G: The health service policy community: professionals pre-eminent or under challenge? In Policy Networks in British Government. Edited by Marsh D, Rhodes RAW. Oxford, Clarendon Press; 1992:51-74. OpenURL

  52. Cohen M, March J, Olsen J: A garbage can model of organizational choice.

    Adm Sci Q 1972, 17:1-25. OpenURL

  53. Janovsky K, Cassels A: Health policy and systems research: issues, methods and priorities. Annex 9. In Investing in Health Research and Development. Edited by Ad Hoc Committee on Health Research Relating to Future Intervention Options. Geneva, World Health Organization; 1996:271-278. OpenURL

  54. Merton RK: The Sociology of Science.

    Chicago, Chicago University Press 1973. OpenURL

  55. Thomas P: The Aims and Outcomes of Social Policy Research.

    London, Croom Helm 1985. OpenURL

  56. Bardach E: The dissemination of policy research to policymakers.

    Knowledge: Creation, Diffusion, Utilization 1984, 6:125-145. OpenURL

  57. Trist E: Types of output mix of research organisations and their complementarity. In Social Science and Government: Policies and Problems. Edited by Cherns AB, Sinclair R, Jenkins WI. London, Tavistock Publications; 1972:101-137. OpenURL

  58. Gibbons M, Limoges C, Nowotny H, Schwartzman S, Scott P, Trow M: The New Production of Knowledge.

    London, Sage Publications 1994. OpenURL

  59. Godin B: Writing performative history: the new New Atlantis.

    Soc Stud Sci 1998, 28:465-83. OpenURL

  60. Weiss C, Bucuvalas MJ: Truth tests and utility tests: decision-makers' frames of reference for social science research.

    Am Sociol Rev 1980, 45:302-313. OpenURL

  61. Hanney S: Doctors and their careers. In Assessing Payback from Department of Health Research and Development: Preliminary report. HERG Research Report No. 19. Volume 2. Edited by Buxton M, Elliott R, Hanney S, Henkel M, Keen J, Sculpher M, Youll P. Uxbridge, Health Economics Research Group, Brunel University; 1994::1-14. OpenURL

  62. Coyle D: Increasing the Impact of Economic Evaluations on Health-Care Decision– Making.

    Discussion Paper 108. York, Centre for Health Economics, University of York 1993. OpenURL

  63. Ross J: The use of economic evaluation in health care: Australian decision makers' perceptions.

    Health Policy 1995, 31:103-110. PubMed Abstract | Publisher Full Text OpenURL

  64. Sloan FA, Grabowski HG: Introduction and overview.

    Soc Sci Med 1997, 45:505-510. PubMed Abstract | Publisher Full Text OpenURL

  65. Drummond M, Cooke J, Walley T: Economic evaluation under managed competition: evidence from the U.K.

    Soc Sci Med 1997, 45:583-595. PubMed Abstract | Publisher Full Text OpenURL

  66. Drummond M, Weatherly H: Implementing the findings of health technology assessments.

    Int J Technol Assess Health Care 2000, 16:1-12. PubMed Abstract OpenURL

  67. Henshall C, Koch P, von Below GC, Boer A, Conde-Olasagasti JL, Dillon A, Gibis B, Grilli R, Hardy C, Liaropoulos L, Martín-Moreno JM, Roine R, Scherstén T, et al.: Health Technology Assessment in Policy and Practice. Working Group 6 Report.

    Int J Technol Assess Health Care 2002, 18:447-455. PubMed Abstract OpenURL

  68. Lyles A, Luce BR, Rentz AM: Managed care pharmacy, socio-economic assessments and drug adoption decisions.

    Soc Sci Med 1997, 45:511-521. PubMed Abstract | Publisher Full Text OpenURL

  69. Webb A, Wistow G: Planning, Need and Scarcity.

    London, Allen and Unwin 1986. OpenURL

  70. Lomas J: Finding audiences, changing beliefs: the structure of research use in Canadian health policy.

    J Health Polit Policy Law 1990, 15:525-542. PubMed Abstract OpenURL

  71. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC: A comparison of results of meta-analyses of RCTs and recommendations of clinical experts.

    JAMA 1992, 268:240-248. PubMed Abstract OpenURL

  72. Haines A, Jones R: Implementing findings of research.

    BMJ 1994, 308:1488-1492. PubMed Abstract OpenURL

  73. Battista RN: Health care technology assessment: linking science and policy-making.

    CMAJ 1992, 146:461-462. PubMed Abstract OpenURL

  74. Tugwell P, Sitthi-Amorn C, O'Connor A, Hatcher-Roberts J, Bergevin Y, Wolfson M: Technology assessment: old, new and needs-based.

    Int J Technol Assess Health Care 1995, 11:650-661. PubMed Abstract OpenURL

  75. Jacob R, McGregor M: Assessing the impact of health technology assessment.

    Int J Technol Assess Health Care 1997, 13:68-80. PubMed Abstract OpenURL

  76. Woods K: Health Technology Assessment for the NHS in England and Wales.

    Int J Technol Assess Health Care 2002, 18:161-165. PubMed Abstract OpenURL

  77. van den Heuvel WJA, Wieringh R, van den Heuvel LPM: Utilisation of medical technology assessment in health policy.

    Health Policy 1997, 42:211-221. PubMed Abstract | Publisher Full Text OpenURL

  78. Barbieri M, Drummond M: The Use of HTA in Decision Making: Report to ECHTA/ECAHI Working Group.

    York, University of York 2002. OpenURL

  79. Rosen R: Applying research to health care policy and practice: medical and managerial views on effectiveness and the role of research.

    J Health Serv Res Policy 2000, 5:103-108. PubMed Abstract OpenURL

  80. Weiss C: Introduction. In Using Social Research in Public Policy Making. Edited by Weiss C. Lexington, Lexington Books; 1977:1-22. OpenURL

  81. Yin R, Moore G: Lessons on the utilization of research from nine case experiences in the natural hazards field.

    Knowledge in Society 1988, 1:25-44. OpenURL

  82. Walt G: How far does research influence policy.

    Eur J Public Health 1994, 4:233-235. OpenURL

  83. Weiss C: Policy research: data, ideas or arguments? In Social Sciences and Modern States. Edited by Wagner P, Weiss C, Wittrock B, Wolman H. Cambridge, Cambridge University Press; 1991:307-332. OpenURL

  84. Saxe L: Policymakers' use of social science research.

    Knowledge: Creation, Diffusion, Utilization 1986, 8:59-78. OpenURL

  85. Orosz E: The Impact of social science research on health policy.

    Soc Sci Med 1994, 39:1287-1293. PubMed Abstract OpenURL

  86. Frenk J: Comprehensive policy analysis for health system reform.

    Health Policy 1995, 32:257-277. PubMed Abstract OpenURL

  87. Miller D: Risk, science and policy: definitional struggles, information management, the media and BSE.

    Soc Sci Med 1999, 49:1239-1255. PubMed Abstract | Publisher Full Text OpenURL

  88. Walt G, Gilson L: Reforming the health sector in developing countries: the central role of policy analysis.

    Health Policy Plan 1994, 9:353-370. PubMed Abstract OpenURL

  89. Patton MQ, Grimes P, Guthrie K, Brennan N, French K, Blyth D: In search of impact: an analysis of the utilisation of federal health evaluation research. In Using Social Research in Public Policy Making. Edited by Weiss C. Lexington, Lexington Books; 1977:141-163. OpenURL

  90. Molas-Gallart J, Tang P, Morrow S: Assessing the non-academic impact of grant-funded socio-economic research: results from a pilot study.

    Research Evaluation 2000, 9:171-182. OpenURL

  91. Landry R, Amara N, Lamari M: Climbing the Ladder of Research Utilization.

    Science Communication 2001, 22:396-422. OpenURL

  92. Spaapen J, Sylvain C: Societal Quality of Research – Toward a Method for the Assessment of the Potential Value of Research for Society. SPSG Concept Paper No.13.

    London, Science Policy Support Group 1994. OpenURL

  93. Roos NP, Shapiro E: From research to policy: what have we learned?

    Med Care 1999, 37(suppl):JS291-JS305. PubMed Abstract OpenURL

  94. Phoolcharoen W: Health Systems Reform in Thailand: The Role of the Health Systems Research Institute.

    Geneva, Alliance-HPSR Working Paper No. 14 2002. OpenURL

  95. Henkel M: PSSRU research on Care Management: Kent Community Care Project. In Assessing Payback from Department of Health Research and Development: Preliminary report. HERG Research Report No. 19. Volume 2. Edited by Buxton M, Elliott R, Hanney S, Henkel M, Keen J, Sculpher M, Youll P. Uxbridge, Health Economics Research Group, Brunel University; 1994::15-37. OpenURL

  96. Anderson M, Cosby J, Swan B, Moore H, Broekhoven M: The use of research in local health service agencies.

    Soc Sci Med 1999, 49:1007-1019. PubMed Abstract | Publisher Full Text OpenURL

  97. Florin D: Scientific uncertainty and the role of expert advice: the case of health checks for coronary heart disease prevention by general practitioners in the UK.

    Soc Sci Med 1999, 49:1269-1283. PubMed Abstract | Publisher Full Text OpenURL

  98. Moodley J, Jacobs M: Research to action and policy: combating Vitamin A deficiencies in South Africa. In Lessons in Research to Action and Policy – Case studies from seven countries. Edited by Geneva, The Council on Health Research for Development. COHRED Working Group on Research to Action and Policy; 2000:54-66. OpenURL

  99. Salvatella R, Muzio F, Sánchez D: Chagar Disease and Foot and Mouth Disease Eradication in Uruguay. In Lessons in Research to Action and Policy: Case Studies from Seven Countries. Edited by Geneva, The Council on Health Research for Development. COHRED Working Group on Research to Action and Policy; 2000:67-76. OpenURL

  100. Ginzberg E: Health Services Research: Key to Health Policy.

    Cambridge, Harvard University Press 1991. OpenURL

  101. Gerhardus A, Dujardin M, Kiet PHT, Siddiqi S, Sauerborn R: A Methodology to Assess the Use of Research for Health Policy Development. Discussion Paper 04.

    Heidelberg, University of Heidelberg Medical School 2000. OpenURL

  102. Milbank Memorial Fund/The Cochrane Collaboration: Informing Judgement: Case Studies of Health Policy and Research In Six Countries.

    New York 2001. OpenURL

  103. Yesudian CAK: Policy Research in India: A Case Study for Regulating Private Health Providers.

    Geneva, Alliance-HPSR Working Paper No.18 2002. OpenURL

  104. Hilderbrand M, Simon J, Hyder A: The role of research in child health policy and programs in Pakistan. In Lessons in Research to Action and Policy – Case Studies From Seven Countries. Edited by COHRED Working Group on Research to Action and Policy. Geneva, The Council on Health Research for Development; 2000:77-85. OpenURL

  105. Knott J, Wildavsky A: If dissemination is the answer, what is the problem?

    Knowledge: Creation, Diffusion, Utilization 1980, 1:537-78. OpenURL

  106. Hanney S, Davies A, Buxton M: Assessing benefits from health research projects: can we use questionnaires instead of case studies?

    Research Evaluation 1999, 8:189-199. OpenURL

  107. Berridge V, Stanton J: Science and policy: historical insights.

    Soc Sci Med 1999, 49:1133-1138. PubMed Abstract | Publisher Full Text OpenURL

  108. Husén T, Kogan M: Educational Research and Policy: How do They Relate?

    Oxford, Pergamon Press 1984. OpenURL

  109. Grant J, Cottrell R, Cluzeau F, Fawcett G: Evaluating "payback" on biomedical research from papers cited in clinical guidelines: applied bibliometric study.

    BMJ 2000, 320:1107-1111. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  110. Buxton M: The costs and benefits of heart transplant programmes at Harefield and Papworth hospitals. In Assessing Payback from Department of Health Research and Development: Preliminary Report. HERG Research Report No. 19. Volume 2. Edited by Buxton M, Elliott R, Hanney S, Henkel M, Keen J, Sculpher M, Youll P. Uxbridge, Health Economics Research Group, Brunel University; 1994::75-89. OpenURL

  111. Davis K, Rowland D: Financing health care for the poor. In Health Services Research: Key to Health Policy. Edited by Ginzberg E. Cambridge, Harvard University Press; 1991:95-125. OpenURL

  112. Chunharas S: Research to action and policy: the need for a new concept. In Lessons in Research to Action and Policy – Case Studies From Seven Countries. Edited by COHRED Working Group on Research to Action and Policy. Geneva, The Council on Health Research for Development; 2000:1-8. OpenURL

  113. Caplan N: The use of social research knowledge at the national level. In Social Research in Public Policymaking. Edited by Weiss C. Lexington, Lexington Books; 1977:183-197. OpenURL

  114. Blau PM: Exchange and Power in Social Life.

    Chichester, Wiley 1964. OpenURL

  115. Polanyi M: The Republic of Science – its political and economic theory.

    Minerva 1962, 1:54-73. OpenURL

  116. Hanney S, Henkel M, von Walden Laing D: Making and implementing foresight policy to engage the academic community: health and life scientists' involvement in, and response to, development of the UK's technology foresight programme.

    Research Policy 2001, 30:1203-1219. OpenURL

  117. Lilford R, Jecock R, Shaw H, Chard J, Morrison B: Commissioning health services research: an iterative method.

    J Health Serv Res Policy 1999, 4:164-167. PubMed Abstract OpenURL

  118. World Health Organization: Improving Health Care Through Decision-linked Research: Application in Health Systems and Manpower Development. Part II: Options for Implementation.

    Geneva 1986. OpenURL

  119. Buxton MJ: Heart transplantation in the UK: the decision-making context of an economic evaluation. In Expensive Health Technologies: Regulatory and Administrative Mechanisms in Europe, Commission of the European Communities, Health Services Research Series No. 5. Edited by Stocking B. Oxford, Oxford University Press; 1988:36-48. OpenURL

  120. Garner P, Kale R, Dickson R, Dans T, Salinas R: Getting research findings into practice: implementing research findings in developing countries.

    BMJ 1998, 317:531-535. PubMed Abstract | Publisher Full Text OpenURL

  121. Lomas J: Using 'linkage and exchange' to move research into policy at a Canadian Foundation.

    Health Aff 2000, 19:236-240. PubMed Abstract OpenURL

  122. Gonzalez-Block M, Mills A: Assessing capacity for health policy systems and research in developing countries.

    Health Research Policy and Systems 2003., 1 OpenURL

  123. Sundquist JL: Research brokerage: the weak link. In Knowledge and Policy: The Uncertain Connection. Edited by Lynn LE. Washington DC, National Academies of Science; 1978:126-144. OpenURL

  124. Peckham M: Preface. In Scientific Basis of Health Services. Edited by Peckham M, Smith R. London, BMJ Books; 1996:ix-xi. OpenURL

  125. Black N: National strategy for research and development: lessons from England.

    Annu Rev Public Health 1997, 18:485-505. PubMed Abstract | Publisher Full Text OpenURL

  126. Macintyre S, Chalmers I, Horton R, Smith R: Using evidence to inform health policy: case study.

    BMJ 2001, 322:222-225. PubMed Abstract | Publisher Full Text OpenURL

  127. Lomas J: Improving Research Dissemination and Uptake in the Health Sector: Beyond the Sound of One Hand Clapping.

    Hamilton, Centre for Health Economics and Policy Analysis, McMaster University 1997. OpenURL

  128. Kogan M, Henkel M: Future directions for higher education policy. Getting inside: policy reception of research. In The Institutional Basis of Higher Education Research (Edited by Teichler U) Dordrecht, Kluwer. ; 2000:25-43. OpenURL

  129. Beyer J, Trice H: The utilization process: a conceptual framework and synthesis of empirical findings.

    Adm Sci Q 1982, 27:591-622. OpenURL

  130. Björklund E: Swedish research on higher education in perspective. Eskil Björkland in conversation. In University and Society. Essays on the Social Roles of Research and Higher Education. Edited by Trow MA, Nybom T. London, Jessica Kingsley Publishers; 1991:173-192. OpenURL

  131. Premfors R: Scientific bureaucracy. Research implementation and Swedish civil servants. In University and Society. Essays on the Social Role of Research and Higher Education. Edited by Trow MA, Nybom T. London, Jessica Kingsley Publishers; 1991:88-98. OpenURL

  132. Thompson PR, Yessian MR: Policy analysis in the Office of the Inspector General, U.S. Department of Health and Human Services. In Organizations for Policy Analysis: Helping Government Think. Edited by Weiss C. Newbury Park, Sage Publications; 1992. OpenURL

  133. Leicester G: The seven enemies of evidence-based policy.

    Public Money and Management 1999, 19:5-7. OpenURL

  134. Patton MQ: Discovering process use.

    Evaluation: The International Journal of Theory, Research and Practice 1998, 4:225-233. OpenURL

  135. Henkel M: Academic Identities and Policy Change in Higher Education.

    London, Jessica Kingsley Publishers 2000. OpenURL

  136. Smith R: Measuring the social impact of research.

    BMJ 2001, 323:528. PubMed Abstract | Publisher Full Text OpenURL

  137. Henkel M: Excellence versus relevance: the evaluation of research.

    International Journal of Institutional Management in Higher Education 1986., 10 OpenURL

  138. Commission on Macroeconomics and Health: Macroeconomics and Health: Investing in Health for Economic Development.

    Geneva, World Health Organization 2001. OpenURL

  139. Smith R: A Time for Global Health.

    BMJ 2002, 325:54-55. PubMed Abstract | Publisher Full Text OpenURL

  140. Pan American Health Organization: Strategies for Utilization of Scientific Information in Decision-Making for Health Equity.

    Washington, Division of Health and Human Development, PAHO 2001. OpenURL

  141. Mayston D: Non-profit performance indicators in the public sector.

    Financial Accountability and Management 1985, 1:51-74. OpenURL

  142. Carter N, Klein R, Day P: How Organisations Measure Success – The Use of Performance Indicators in Government.

    London, Routledge 1992. OpenURL

  143. Rossi PM, Freeman HE, Lipsey MW: Evaluation – A Systematic Approach. Sixth Edition.

    Thousand Oaks, Sage Publications 1999. OpenURL

  144. Poole MS, Van de Ven AH, Dooley K, Holmes ME: Organizational Change and Innovation Processes: Theory and Methods for Research.

    New York, Oxford University Press 2000. OpenURL

  145. Grimshaw J, Eccles M: Clinical practice guidelines. In Evidence-based Practice in Primary Care. Edited by Silagy C, Haines A. London, BMJ Books; 1998. OpenURL