Research

The feasibility of determining the impact of primary health care research projects using the Payback Framework

Elizabeth C Kalucy^1* , Eleanor Jackson-Bowers^1* , Ellen McIntyre^1* and Richard Reed^2*

¹  Primary Health Care Research and Information Service, Department of General Practice, Flinders University, Adelaide, South Australia, Australia

²  Discipline of General Practice, Flinders University, Adelaide, South Australia, Australia

author email corresponding author email* Contributed equally

Health Research Policy and Systems 2009, 7:11doi:10.1186/1478-4505-7-11

Published:	8 May 2009

Abstract

Background

Primary health care research is under pressure to be accountable to funders in terms of benefits for practice and policy. However, methods to assess the impact of primary health care research must be appropriate to use with the diverse topics, settings and approaches of this sector. This project explored the feasibility of using the Buxton and Hanney Payback Framework to determine the impact of a stratified random sample (n = 4) of competitively funded, primary health care research projects.

Methods

The project conducted telephone interviews based on the Payback Framework with leaders of the research teams and nominated users of their research, used bibliometric methods for assessing impact through publication outputs and obtained documentary evidence of impact where possible. The purpose was to determine the effectiveness of the data collection methods and the applicability of the Payback Framework, and any other issues which arose around the assessment of impact of primary health care research.

Results and discussion

The thirteen interviews were resource intensive to organise conduct and analyse but provided better information about impact than bibliometric analysis or documentary analysis. Bibliometric analysis of the papers published from the four projects was hampered by the inclusion of only one of the journals in major citation indexes. Document analysis provided more evidence of dissemination than of impact.

The payback framework and logic model were a sound basis for assessing impact. Chief investigators and nominated users of research provided substantial information relevant to the impact categories closest to their spheres of influence and awareness, but less about the impact their research had on the wider health sector, population health or economic benefits. An additional category of impact emerged from the interviews, that of strengthening research networks which could enhance the impact of later work. The framework provided rich information about the pathways to impact, better understanding of which may enhance impact.

Conclusion

It is feasible to use the Buxton and Hanney Payback framework and logic model to determine the proximal impacts of primary health care research. Though resource intensive, telephone interviews of chief investigators and nominated users provided rich information.