Evaluation of the international forum on evidence informed health policymaking: Addis Ababa, Ethiopia – 27 to 31 August 2012

Abstract

Background

Meetings and conferences are often used as a tool to disseminate information, network with colleagues, and/or set direction for a field of study, but there is little evidence to support whether such events achieve their objectives. This study evaluates the International Forum on Evidence Informed Health Policymaking (EIHP), a three-day meeting held in Addis Ababa, Ethiopia, in 2012, to determine the success of the meeting based on pre-determined objectives.

Methods

The evaluation strategy was developed based on a previously published conference evaluation framework and operationalized as an end-of-conference participant survey that incorporated both process (programme/organization) and outcome measures (potential changes in behaviour).

Results

Sixty seven of approximately 121 attendees filled out a questionnaire (a 55% response rate) and, overall, participants rated the programme components and plenary sessions very highly. The top three benefits reported by participants were: i) sharing experiences and lessons learned (75%); ii) new opportunities for future collaboration (69%); and iii) new knowledge (67%). Conversely, only 25% or less of meeting participants reported an intent to utilize any of the potential benefits highlighted in the questionnaire, with the notable exception of pursuing new opportunities for future collaboration.

Conclusions

The evaluation findings suggest that the International Forum achieved its objectives of sharing experiences with EIHP and providing opportunities for networking among EIHP initiatives, although there are limited prospects for direct improvements to efforts to support EIHP.

Report

Introduction

Meetings and conferences are often used as a tool to disseminate information, network with colleagues, and/or set direction for a field of study; however, there is little evidence to support whether these meetings are effective in relation to their objectives[1, 2]. While previous studies have examined the effectiveness of small educational meetings (or workshops)[3], international meetings that bring in a diverse group of stakeholders and have similarly diverse objectives have not been evaluated to the same extent. One way to improve our understanding of the impact of these meetings is to increase the value of end-of-conference evaluations through: i) building the capacity of meeting organizers to develop their own evaluation strategies and ii) increasing the dissemination of meeting evaluations through publication in peer-reviewed journals and other avenues beyond just meeting participants.

In 2012, we published a scoping review to map out what types of objectives and evaluative practices were being utilized at large multi-day meetings[4]. The review culminated in the development of a conference evaluation framework, adapted in Figure 1, to aid the organizers of such meetings in evaluating the impact of their conference on participants and their associated sectors. We utilized this conference evaluation framework to support the development of the evaluation strategy for the International Forum on Evidence Informed Health Policymaking (hereafter referred to as the International Forum).

Figure 1

Conference evaluation framework (adapted from [[4]]). The bulleted components represent the most frequent examples utilized for each category (as reported in the literature). Components highlighted in red indicate which objectives, purpose, methods, and indicators were utilized in the International Forum's evaluation.

The International Forum was held in Addis Ababa, Ethiopia, from 27–31 August 2012. The meeting was hosted by Supporting the Use of Research Evidence in African Health Systems (SURE), the WHO Evidence-Informed Policy Network (EVIPNet), and the Regional East African Community Health (REACH) Policy Initiative. The local host was the Ethiopian Health and Nutrition Research Institute (ENHRI). The objectives of the International Forum were: i) to share experiences with (and resources for) evidence-informed health policymaking (EIHP) in low- and middle-income countries (LMICs); ii) to identify opportunities for improving country-level efforts to support EIHP in LMICs; and iii) to provide an opportunity for networking among initiatives to support EIHP[5]. This study aims to examine the strengths and weaknesses of the meeting as a whole (process measures) as well as the potential benefits reported and intent to utilize these benefits (outcome measures) based on the formal feedback from attendees.

Methods

The International Forum evaluation strategy was operationalized as an end-of-conference participant survey. The survey was adapted from another international meeting with similar attendee demographics and agenda structure. The development of the survey was guided by our previously published conference evaluation framework[4], as well as by consultations with various meeting organizers.

Data collection

We administered 100 hard copies of the evaluation questionnaire in English (Additional file1) and 50 copies in French. We placed the questionnaires on participants’ tables prior to the last session of the final day and distributed them at the back of the same conference hall. The questionnaires were collected by volunteers after the session or placed into drop boxes at the back of the hall.

The survey consisted primarily of Likert scale questions evaluating participant satisfaction with the content of the meeting, supplemented by boxes for comments on other strengths or weaknesses of the meeting (process measures). The survey also included checkboxes to collect data on the potential benefits participants received from the meeting and space to report intent to utilize any of these benefits (outcome measures). Finally, we collected information on participants’ previous activities with EIHP (e.g., collaboration between researchers and policy makers when making decisions), both as an indicator of attendees experience with EIHP and to be used as baseline data when following-up with attendees at future iterations of the International Forum to examine any progress in the field as these types of meetings continue.

Data analysis and synthesis

Two researchers entered the survey data into Microsoft Excel, with double entry of 30% of the data to check inter-rater reliability and reconcile any errors (none were identified). Given the few French responses received, answers written in French were translated into English upon entry and analyzed regardless of initial language. From the data, we calculated the mean and standard deviation of each Likert-scale rating for all survey participants and for the following demographic categories: i) first time at an international meeting on EIHP; ii) from a LMIC; iii) self-reported role; iv) years of experience in EIHP; and iv) gender. These statistics were organized into tables according to overall focus of questions and findings were examined within each demographic category as well as across categories. Qualitative data from comment boxes were analyzed thematically using a constant comparative method. Thematic codes were tallied for each question to highlight which perspectives were most often reported in the survey. Response rates were determined by comparing the self-reported role of participants in the evaluation to the role they reported when registering for the meeting.

We also conducted four multinomial logistic regressions to examine relationships between certain demographic categories of attendees and their intent to utilize the benefits they reported. Regressions were performed in SPSS, with each of the top four benefits reported as dependent variables and the five demographic categories as factors (independent variables). All demographic categories were coded as binary variables except years of experience and self-reported role, which were recoded with dummy variables before being entered into the regression.

Results

The International Forum was structured as a large consultative process, including hands-on workshops, knowledge sharing sessions, and facilitated discussion, supplemented by five plenaries (Figure 2). The sessions were organized around five themes: i) Evidence informed health policy in action, in particular around sharing experiences related to the process of preparing evidence briefs for policy; ii) Skills training for knowledge translation efforts; iii) Tools to assist in pursuing knowledge translation efforts; iv) People and forming networks to work with highly complex issues and broadly based stakeholders; and v) Collaboration and innovation to improve EIHP initiatives. More detailed names and topics of the meeting sessions can be found in the International Forum’s programme[5].

Figure 2

Condensed logic model for the International Forum evaluation. *List of all benefits reported can be found in Table 4.

Of the approximately 121 attendees, 67 filled out the survey (55%), of which 40/75 were researchers (53%), 11/23 policymakers (48%), 4/8 journalists (50%), and 12/15 (80%) attendees who did not report a role. In terms of experience with EIHP (Table 1), journalists had the most experience (by percentage) with the EIHP activities we measured but they were also the smallest grouping of stakeholders at the meeting. Participants, regardless of role, were more likely to undertake activities they could accomplish by themselves (i.e., utilizing research in their policy decision-making) versus approaching other stakeholders (i.e., talking to a researcher/journalist about the available evidence on a policy).

Table 1 Number and percentage of participants reporting past experience with evidence-informed health policymaking

Overall, the meeting was very well received, with 100% of participants reporting that they would attend the International Forum again in the future. The following sections analyse the specific components of the meeting based on the survey responses. Logistic regressions comparing the different types of sessions and reported benefits (including intent to utilize them) identified no significant correlations.

Programme

On a scale of 1–5 (very poor to very good), participants rated the overall programme (4.4 [0.7]) and the various types of sessions very highly, which was a consistent finding across all reported demographic categories (Table 2). Participants rated the poster presentations much lower than all other sessions (3.2 [0.9]). Specifically, less experienced attendees as well as journalists and researchers (compared to policymakers) gave lower ratings for the poster presentations.

Table 2 Average participant rating with [standard deviation] of programme components

Plenaries

Participants rated all five plenary sessions above 4.0 on a 1–5 scale, with the plenary on success stories and lessons learned given the highest rating (4.4 [0.7]) (Table 3). In terms of specific plenaries, LMIC participants rated the panel discussion a 3.2 [1.0] on average, the lowest score for that session across all categories and the lowest rating by LMIC participants for all plenaries. This corresponds to the feelings of certain participants noted in written comments that there were not enough LMIC speakers. In contrast, researchers were generally more positive in their ratings than policymakers or journalists. A few participants noted in their comments that including more current (“true”) policymakers would improve the plenary sessions.

Table 3 Average participant rating (with standard deviation) of plenary sessions

Benefits to participants

Overall, the top three benefits reported by participants were: i) sharing experiences and lessons learned (50 (75%)); ii) new opportunities for future collaboration (46 (69%)); and iii) new knowledge (45 (67%)). These three benefits were consistently the most frequently reported across all demographic categories (Table 4). Participants with the least experience with EIHP reported the most benefits, while the most experienced participants reported the least (by percentage). Most participants reported receiving multiple benefits from the meeting and no participant reported receiving no benefits, yet much fewer participants reported they intended to utilize the benefits in a meaningful way. Only 25% or less of meeting participants reported an intent to utilize any of the potential benefits highlighted in the questionnaire, with the exception of pursuing new opportunities for future collaboration (26 (39%)). Furthermore, the number of participants reporting a certain benefit did not directly correspond to participants’ intentions to utilize the benefit in their work in a meaningful way. For example, by comparison, few participants reported new skills as a benefit overall (31 (46%)), but 13 (19%) reported an intent to utilize new skills, the third highest percentage.

Table 4 Number and percentage of participants reporting benefits from attending the International Forum

Qualitative comments on meeting organization

Twelve participants noted that the logistics and the pre-meeting organization could have been strengthened, specifically through reducing last minute changes to the programme and being clearer about meeting objectives. Multiple participants also reported that there was too much in the programme and that “less is more”, which would have provided more time for discussions. In terms of sessions participants enjoyed the most, seven participants highlighted presentations on country experiences and six participants highlighted the impact evaluation/analysis sessions as their favourites. Four participants highlighted a need to explore the sustainability of EIHP initiatives and how to finance policy research and initiatives. Finally, four other participants suggested expanding the capacity building sessions at the International Forum to include topics such as training on writing evidence briefs for policy.

Discussion

This evaluation of the International Forum demonstrates that participants were highly satisfied with the various programme components. A majority of participants reported receiving multiple benefits but fewer participants reported an intent to utilize any of these benefits in their future work. It is interesting to note that there was no significant correlation between high ratings for any type of session (i.e., plenary, workshop, informal discussion) and benefits reported, suggesting that the meeting structure as a whole has a greater impact than the organization of one specific component. Overall, the evaluation findings suggest that the International Forum achieved its objectives of sharing experiences with EIHP and providing opportunities for networking among EIHP initiatives, though it is difficult to tell whether the meeting was successful in identifying opportunities to improve efforts to support EIHP in LMICs. There were some differences in findings between demographic categories, which were reported above, but generally, ratings were fairly consistent across categories. However, participants from LMICs did rate some of the plenary sessions lower and noted in their comments that more presentations from LMIC participants would have provided more relevant information. A similar concern was noted by policymakers and some comments suggested inviting more current policymakers.

Several strengths and weaknesses in this study should be considered. In terms of strengths, this is one of the first meeting evaluations to be developed based on a framework that incorporates theories and previous evaluative practices specific to large meetings. Further, our survey included both quantitative and qualitative components to increase the breadth of findings we could examine. We also achieved a fairly high response rate for an international meeting survey, at 55%. Finally, we conducted a formal analysis and synthesis of the data to add to a limited body of research on conference evaluations. A weakness of this study is that only one tool (a survey) was used for data collection, which could have been strengthened through intercept interviews to further explore some findings. Finally, though we achieved a high response rate for an international conference evaluation, the relatively small size of the meeting limited the total number of participants. As a result, performing reliable and generalizable regression analyses proved difficult.

This evaluation confirms that conferences and meetings can be successful in relation to traditional participant objectives such as sharing experiences, networking and gaining new knowledge but that there is still work to be done towards increasing meetings’ impact on its participants and associated sectors. In order to ensure that we are getting the most of out of these meetings, we need to continue to develop effective ways of evaluating them. Future evaluations of international meetings could build on the present study by expanding upon our methodology to include interviews or follow-up to supplement an end-of-conference survey. Furthermore, our previous scoping review on conference objectives and evaluations, as well as numerous other evaluation studies, highlight the importance of including not only participant perspectives but indicators of sector development in evaluations as well[4, 6–8]. This study takes the first steps in this process by collecting information on a number of important EIHP activities, which can be compared with evaluation data at future iterations of the International Forum to track any progress in the field as a whole. This being said, there remain unanswered questions about the effectiveness of these indicators in predicting the impact of the meeting itself on the field of EIHP.

Conclusions

The International Forum on Evidence Informed Health Policymaking successfully encouraged sharing experiences and lessons learned, and new opportunities for future collaboration and knowledge exchange between a variety of stakeholders. Furthermore, all participants said they would attend the International Forum again in the future. Meeting evaluations should continue to be made public to inform similar future meetings about successes and failures of the meeting itself, but also to strengthen and guide the development of evaluations themselves.