Email updates

Keep up to date with the latest news and content from HARPS and BioMed Central.

Open Access Research

Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in Thailand

Sitaporn Youngkong12*, Yot Teerawattananon1, Sripen Tantivess1 and Rob Baltussen2

Author Affiliations

1 Health Intervention and Technology Assessment Program (HITAP), Ministry of Public Health, Nonthaburi, Thailand

2 Nijmegen International Center for Health Systems research and Education (NICHE), Department of Primary and Community Care, Radboud University Nijmegen Medical Center, Nijmegen, The Netherlands

For all author emails, please log on.

Health Research Policy and Systems 2012, 10:6  doi:10.1186/1478-4505-10-6


The electronic version of this article is the complete one and can be found online at: http://www.health-policy-systems.com/content/10/1/6


Received:27 September 2011
Accepted:17 February 2012
Published:17 February 2012

© 2012 Youngkong et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

A wide range of preventive, treatment, and care programs for HIV/AIDS are currently available and some of them have been implemented in Thailand. Policy makers are now facing challenges on how the scarce resources for HIV/AIDS control can be spent more wisely. Although effectiveness and cost-effectiveness information is useful for guiding policy decisions, empirical evidence indicates the importance of other criteria, such as equity and the characteristics of the target population, also play important roles in priority setting. This study aims to experiment with the use of multi-criteria decision analysis (MCDA) to prioritise interventions in HIV/AIDS control in Thailand.

Methods

We used MCDA to rank 40 HIV/AIDS interventions on the basis of the priority setting criteria put forward by three groups of stakeholders including policy makers, people living with HIV/AIDs (PLWHA), and village health volunteers (VHVs). MCDA incorporated an explicit component of deliberation to let stakeholders reflect on the rank ordering, and adapt where necessary.

Results

Upon deliberation, policy makers expressed a preference for programs that target high risk groups such as men who have sex with men, injecting drug users, and female sex workers. The VHVs preferred interventions that target the youth or the general population, and gave lower priority to programs that target high risk groups. PLWHA gave all interventions the same priority. The rank order correlation between the priorities as expressed before and after deliberation was 37% among the policy makers and 46% among the VHVs.

Conclusion

This study documented the feasibility of MCDA to prioritize HIV/AIDS interventions in Thailand, and has shown the usefulness of a deliberative process as an integrated component of MCDA. MCDA holds potential to contribute to a more transparent and accountable priority setting process, and further application of this approach in the prioritisation of health interventions is warranted.

Keywords:
Multi-criteria decision analysis; Priority setting; HIV/AIDS interventions; Discrete choice experiment

Background

Since HIV/AIDS has long been recognized as a leading cause of death and a high burden of disease in Thailand [1-3], a wide range of preventive, treatment, and care programs have been implemented to combat the disease. Recently, it was suggested that funding decisions on these programs are not taken in a systematic manner and that the resulting mix of interventions is not offering the best value for money [4]. Consequently, Thai policy makers now face the challenge of how the scarce resources available for HIV/AIDS control can be spent more wisely.

A range of studies are available to guide Thai policy makers to prioritise HIV/AIDS interventions. International estimates are available on the effectiveness and cost-effectiveness of HIV/AIDS interventions [5-7], and a recent document has systematically reviewed this information - in combination with national estimates - to provide informed priorities for HIV/AIDS control [4]. Yet the analysis falls short of including other criteria that may also play important roles in effective decision-making, such as ethical and social concerns. For example, the preference of society to pursue not only efficiency goals (that could result in prevention-oriented strategies for the general population) but also equity goals (that could, for example, result in treatment-oriented strategies for the severely ill) may have a large impact on the choice of programs [8-10]. This indicates the need for multiple criteria decision analysis (MCDA) to account for other criteria beyond effectiveness and cost-effectiveness in the decision-making process [11-14].

Although MCDA is used in only a few applications to guide the making of resource allocation decisions on health, it is routinely used in environmental, agricultural and marketing sciences to set intervention priorities [14]. In those disciplines, MCDA has evolved as a response to the observed inability of people to effectively analyze multiple streams of dissimilar information. The analysis establishes preferences between interventions by reference to an explicit set of criteria that the decision-making body has identified. A key component of every MCDA is the performance matrix that describes the performance of the interventions against each criterion. The performance matrix may be the final product of the analysis, allowing the decision makers to qualitatively rank the interventions. Such intuitive processing of the data can be quick and effective, but it may also lead to the use of unjustified assumptions, causing an incorrect ranking of options. In analytically more sophisticated MCDA techniques, the information in the basic matrix is usually converted into consistent numerical values. The key idea is to construct scales representing preferences for the consequences, to weigh the scales for their relative importance, and then to calculate weighted averages across the preference scales [14]. In recent applications of MCDA [15-19], it has been criticized for its quantitative nature - studies typically rank ordered interventions on the basis of weighted averages and, in this way, consider quantifiable criteria only. To date, some attempts to capture non-quantifiable criteria to support the deliberative process have been reported [20,21]. This confirms that MCDA should rather include a deliberative process or other qualitative tools to also consider non-quantifiable concerns [20,22-25] and foster well-balanced judgments on intervention priorities [26,27].

The primary aim of this study is to experiment with the use of MCDA, including the use of a deliberative process to prioritise interventions in HIV/AIDS control in Thailand. This research follows up on a recent study that employed discrete choice experiments (DCE) to identify and measure the relative importance of various quantifiable and non-quantifiable criteria for the priority setting of HIV/AIDS interventions in Thailand among various stakeholders [28].

Methods

The MCDA in the present study includes three components. Firstly, we assessed the performance of interventions on the criteria as identified in the DCE (i.e. we constructed the performance matrix). Second, we ranked ordered interventions. Third, we engaged with the various stakeholders in a deliberative process to adapt the rank ordering where necessary. We also compared the rank order of interventions before and after the deliberative process. These components are discussed in turn.

Constructing the performance matrix

As a starting point, we identified a broad set of 40 HIV/AIDS interventions that are already implemented, or eligible for implementation, in Thailand. We then constructed the performance matrix, i.e. we scored each of the selected HIV/AIDS interventions as a function of their performance on a set of criteria as identified in a recent DCE study [28]. This study identified the criteria to be relevant to the priority setting of HIV/AIDS control in Thailand through group discussions with each group of stakeholders including policy makers, people living with HIV/AIDS (PLWHA), and community members represented by village health volunteers (VHVs). The resulting criteria from the three group discussions were compared and finally those that were identified by two or more discussion groups were selected. These included: target groups of interventions (i.e. children, teenagers, adults, and high-risk adults); gender of target groups (i.e. female versus male); type of interventions (i.e. prevention, treatment of patients with HIV, and treatment of patients with AIDS); effectiveness (i.e. low versus high effectiveness); and quality of evidence (i.e. weak versus strong evidence). In the performance matrix, '0' denotes the absence of and '1' indicates the presence of a criterion level (see Additional file 1: Appendix 1). Information on target group, gender of target group and type of intervention was identified from each intervention itself, whereas the information on effectiveness of intervention and quality of evidence on effectiveness were based on the review conducted by Pattanaphesaj and Teerawattananon [4].

Additional file 1. Appendix 1. The performance matrix of HIV/AIDS interventions. Appendix 2. Discrete choice model results by perspective.

Format: DOC Size: 144KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Rank ordering of interventions

Subsequently, we estimated the probability of selection of an intervention by using the logistic regression model derived from the DCE study [28]:

Logit(P) = ln [P/(1-P)] = β0 + β1-3 Target group + β4-5 Gender of target group + β6-7 Type of intervention + β8 Effectiveness + β9 Quality of evidence on effectiveness + ε

where P is the probability of an intervention being selected by the respondents, β0 is the constant term, βi (i = 1-9) are the coefficients of the model indicating the probability of selection relative to the reference criterion level, and ε is the unobservable error term. The regression coefficients for all criteria were obtained from each of the three groups of stakeholders - policy makers, PLWHA, and community members - represented by VHVs during the DCE survey (These are listed in Additional file 1: Appendix 2). Next, all interventions were ranked in order of their probability of selection.

Deliberative process

In the deliberative process, group discussions were independently organized between July and August 2009 with three groups of stakeholders: six policy makers at the national level who are heavily involved in health resource allocation decisions in Thailand specifically on HIV/AIDS ('policy makers'); six members of the Thai network for PLWHA, representing PLWHA groups at the regional level in Thailand ('PLWHA'); and six community members who have been trained by public health providers to be the VHVs in Samutprakan province ('VHVs'). Participants were selected purposively from each group of stakeholders on the basis of their participation in the previous DCE study to ensure that they were familiar with the DCE and the priority setting process. Each group discussion began with a brief introduction of the purpose of the meeting. Next, participants were presented with the rank ordering of the interventions, and they were then asked whether they agreed with the rank and to provide their justifications. We then asked them to re-classify all interventions into three categories - based on the traffic-light analogy: 'good candidates for implementation' (green), 'not good candidates for implementation' (red), and 'in-between' (yellow). This re-classification was done through consensus or, when necessary, through voting. In all steps, participants were encouraged by the researcher (SY) to discuss, bring in additional criteria, and share their opinions with justifications regarding their preferences.

Comparison of rankings

We compared the rank ordering of interventions before and after the deliberative process to explore the impact of deliberation by estimating the Spearman's rank correlation coefficient.

Research ethics

This study was approved by the Institute for the Development of Human Research Protections, Ministry of Public Health, Thailand. All participants provided their written informed consent.

Results

The rank ordering of the 40 HIV/AIDS interventions before and upon deliberation is presented in Table 1. As indicated by the ranking results before deliberation, the group of policy makers expressed a preference for preventive programs that are highly effective and target high risk groups such as men who have sex with men (MSM), injecting drug users (IDU), female sex workers (FSW), and HIV sero-discordant couples, with good quality of evidence on intervention effectiveness. The five interventions with the highest priority were voluntary counseling and testing (VCT) for IDU, street outreach for IDU, substitution treatment for IDU, improved sexual transmitted infection (STI) treatment services for IDU, and improved STI treatment services for HIV sero-discordant couples.

Table 1. The HIV/AIDS interventions' ranking based on DCE, and the ranking after group discussions

Upon deliberation, the group of policy makers reinforced their preference for highly effective programs that target high risk groups. Community-based education and programs that target the youth or the general population (with the exception for those aimed at the improvement of STI treatment services) were not preferred. In the deliberative process, a number of additional criteria were put forward in addition to those identified in the DCE. The policy maker group proposed cost-effectiveness as an important additional criterion. This group also added the criteria of whether an intervention could be used for multiple purposes, and of safety. For example, a policy maker argued that introducing nucleic acid test screening for blood testing enables the Thai Red Cross Society to simultaneously investigate the existence of Hepatitis B and C with detecting HIV in the same specimen, thus creating added value. Also, a reliable blood donation system is very important to secure safety in Thailand in this respect. The other criterion mentioned was the importance of targeting health care workers at risk as a way of encouraging them to work with PLWHA in hospitals. This led to a change from the rank 10th of the post-exposure prophylaxis for healthcare workers before deliberation to 'good candidate for implementation' category, upon deliberation.

The group of PLWHA expressed a strong preference for the treatment or care for AIDS patients i.e. highly active antiretroviral therapy, and treatment for opportunistic infection and other palliative care, as elicited by the DCE study (Table 1). However, upon deliberation, PLWHA gave almost all of the 40 interventions the same priority. They argued that every intervention was important and should be implemented together to prevent HIV infection. This group of PLWHA also asked that more budget possibilities be found from several sources of funding in order to secure the programs and that otherwise, HIV/AIDS programs should be smaller in scope so policy makers can cover all programs within their limited budgets.

PLWHA suggested the availability of alternatives as an additional criterion. For example, improving STI treatment services was not seen as a priority as alternative services were available in hospitals. PLWHA strongly disagreed with considering cost-effectiveness as a criterion - they argued that if an intervention is effective, it should be implemented, and that financial considerations should not be important. PLWHA also prioritized interventions that target the general population rather than high risk groups, because interventions for the general population cover a larger segment of the population, and reflect their notion that everyone has equal risk of HIV infection. One participant argued: "If these (interventions) are the national policy, they should be implemented to everyone not only the high risk groups. This is because everyone is at equal risk of HIV infection. We are all the same".

The preferences of VHVs cohered largely with those of policy makers except for the target group of the interventions: VHVs preferred interventions that target the youth rather than high risk populations. Consequently, community-based education and improvement of STI treatment services for the youth were the highest priority (Table 1). This preference was also confirmed upon deliberation. VHVs introduced the number of beneficiaries as an additional criterion. One volunteer mentioned: "Mass media campaigns have an impact on lots of people in society. So we think this intervention is beneficial for society at large". Furthermore, VHVs emphasized the need to adapt certain interventions to suit the groups targeted.

There was a significant correlation between the rank ordering before and after deliberation for policy makers (correlation coefficient 37%) and VHVs (46%). The correlation coefficient presents the consistency of results between the DCE ranking and deliberation ranking. No such significant correlation was found for the PLWHA. In addition, from the group discussions, we found that both policy makers and VHVs were generally positive about the ease of interpreting DCE results and the MCDA process, whereas PLWHA were generally negative because of the difficulty of the DCE questionnaire, which might lead to a misunderstanding of the exercise among the respondents.

Discussion

This study has experimented with the use of MCDA to guide the priority setting of HIV/AIDS interventions in Thailand, on the basis of consultations with the relevant stakeholders through a deliberative process.

This study revealed the importance of five criteria included in the DCE (i.e. target groups of interventions, gender of target groups, type of interventions, effectiveness, and quality of evidence on effectiveness), and a number of additional criteria raised during the deliberative process (i.e. ethical and social concerns, cost-effectiveness, (non)availability of alternatives; number of beneficiaries; and inappropriate use or abuse of interventions). This reflects that stakeholders consider multiple criteria in prioritising interventions.

The abovementioned results highlight that MCDA has good potential to be used for the making of explicit prioritisation decisions. Also, we observed that the group of policy makers and VHVs - although not PLWHA respondents - applauded the systematic approach for priority setting, including the development of relevant criteria, the presentation of the performance of interventions against these criteria, and the deliberative process. Although MCDA seems difficult for PLWHA as they may not be familiar or comfortable to make trade-off decisions, the considerable overlap of the rank ordering before and upon deliberation in the group of policy makers and VHVs indicates that the quantifiable criteria used in the DCE partly reflect the concerns that stakeholders have in their intervention priorities. We believe that, through its explicit approach, MCDA contributes to the transparency and accountability of the priority setting process. Moreover, the provision of the DCE ranking reduces the stream of information that stakeholders need to absorb when prioritising many interventions simultaneously. We therefore advocate that the identification and weighing of quantifiable criteria (whether through DCE or any other technique) should also be considered as an integrated MCDA component.

The present application of MCDA seems especially useful for policy planning in the long run as it can set priorities among a large set of interventions without defining the allocation of resources in a precise fashion. This use, also labeled generalized priority setting, can have far-reaching and constructive influences on policy formulation in the long term [26]. In contrast, the use of MCDA as presented in this study may not be useful for guiding highly contextualized decisions on the implementation of a single intervention, since this requires a higher level of detail in terms of financial and budgeting considerations.

This study has experimented with the inclusion of a process of deliberation in MCDA in a research environment. As of now, Thailand is stepping towards a routine application of MCDA to define its universal coverage benefit package. Observations of that process reveal that the inclusion of all relevant stakeholders right from the beginning of the MCDA process is imperative to its success [29].

Yet, we also observed a number of shortcomings in the use of MCDA in this study. First, DCE are cognitive demanding and may not be appropriate for all stakeholders. Most notably, PLWHA had difficulties in completing the DCE survey and interpreting the DCE findings. Further research is needed on the use of less cognitive demanding techniques than DCE that serve the same goal [30]. Second, our intervention set was relatively homogeneous in terms of the criteria covered in the DCE (e.g. effectiveness; quality of evidence on effectiveness; type of intervention), and this resulted in low variation in the probabilities of inclusion. The application of DCE across different health conditions [15-19] is, in that respect, more powerful. Third, we did not engage all stakeholders in a single deliberative process to arrive at a consensus on the rank ordering of interventions, an adaption which would represent the final stage of a successful priority setting process. However, the findings in this study can serve as a reflection of other stakeholders' preferences for policy decision making that may lead to greater acceptance of priority setting decisions. Moreover, this study can be considered a lesson learned process for other stakeholders, especially the general population who have never been involved in health policy decision-making, and can help them to understand how to set priorities for health interventions. In future priority setting research, it would therefore be valuable to incorporate these public perspectives.

Although the set of criteria for MCDA may vary by country and health system context, the approach is generalizable to other settings. Furthermore, the MCDA criteria may be different if priority setting is required across different health problems e.g. infectious diseases, cardiovascular conditions, and mental health problems. Therefore, further exploration is warranted.

Conclusion

This study has documented the feasibility of MCDA for prioritising HIV/AIDS interventions in Thailand, and has shown the usefulness of a deliberative process as an integrated component of MCDA. MCDA holds potential to contribute to a more transparent and accountable priority setting process, and further application of this approach in the prioritisation of health interventions is warranted.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

SY, RB, and YT conceptualized the study. SY wrote the first draft of the manuscript. RB, ST, and YT made substantial contributions to the data interpretation and writing of the manuscript. All authors read and approved the final manuscript.

Acknowledgements

We would like to thank our participants in Thailand for their willingness to participate. We are grateful for the joint financial support to the Health Intervention and Technology Assessment Program (HITAP) by the Thai Health Promotion Foundation, Health System Research Institute, and Bureau of Health Policy and Strategy, Ministry of Public Health. At the time this study was conducted the first author was supported by the World Health Organization under its Fellowship Program to study at Radboud University Nijmegen Medical Centre, The Netherlands.

References

  1. UNAIDS/WHO working on global HIV/AIDS and STI: Epidemiological fact sheet on HIV and AIDS: Core data on epidemiology and response. Geneva: World Health Organization; 2008. OpenURL

  2. Porapakkham Y, Rao C, Pattaraarchachai J, Polprasert W, Vos T, Adair T, Lopez AD: Estimated causes of death in Thailand, 2005: implications for health policy.

    Population Health Metrics 2010, 8:14. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  3. Tantivess S: Universal access to antiretroviral therapy in Thailand: an analysis of the policy process. University of London: PhD thesis; 2006. OpenURL

  4. Pattanaphesaj J, Teerawattananon Y: Reviewing the evidence on effectiveness and cost-effectiveness of HIV prevention strategies in Thailand.

    BMC Publ Health 2010, 10:401. BioMed Central Full Text OpenURL

  5. Hogan DR, Baltussen R, Hayashi C, Lauer J, Salomon JA: Cost effectiveness analysis of strategies to combat HIV/AIDS in developing countries.

    BMJ 2005, 331(7530):1431-1437. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  6. Creese A, Floyd K, Alban A, Guinness L: Cost-effectiveness of HIV/AIDS interventions in Africa: a systematic review of the evidence.

    Lancet 2002, 359:1635-1642. PubMed Abstract | Publisher Full Text OpenURL

  7. Cohen DA, Wu S-Y, Farley TA: Cost-effectiveness allocation of government funds to prevent HIV infection.

    Heal Aff 2005, 24:915-926. Publisher Full Text OpenURL

  8. Kaplan EH, Merson MH: Allocating HIV-prevention resources: Balancing efficiency and equity.

    Am J Public Health 2002, 92(12):1905-1907. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  9. Kenny N, Joffres C: An ethical analysis of international health priority-setting.

    Health Care Anal 2008, 16:145-160. PubMed Abstract | Publisher Full Text OpenURL

  10. Tantivess S, Walt G: Using cost-effectiveness analyses to inform policy: the case of antiretroviral therapy in Thailand.

    Cost Effectiveness and Resource Allocation 2006, 4:21. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  11. Musgrove P: Public spending on health care: how are different criteria related?

    Health Policy 1999, 47:207-223. PubMed Abstract | Publisher Full Text OpenURL

  12. Dolan P, Shaw R: A note on the relative importance that people attach to different factors when setting priorities in health care.

    Health Expectation 2002, 6:53-59. OpenURL

  13. Multi-criteria analysis manual [http://iatools.jrc.ec.europa.eu/public/IQTool/MCA/DTLR_MCA_manual.pdf] webcite

  14. Baltussen R, Niessen LW: Priority setting of health interventions: the need for multi-critieria decision analysis.

    Cost Effectiveness and Resource Allocation 2006, 4:14. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  15. Baltussen R, Stolk E, Chisholm D, Aikins M: Towards a multi-criteria approach for priority setting: an application to Ghana.

    Heal Econ 2006, 15:689-696. Publisher Full Text OpenURL

  16. Baltussen R, ten Asbroek A, Koolman X, Shrestha N, Bhattarai P, Niessen LW: Priority setting using multiple criteria: should a lung health programme be implemented in Nepal?

    Health Policy and Planning 2007, 22:178-185. PubMed Abstract | Publisher Full Text OpenURL

  17. Green C, Gerard K: Exploring the social value of health-care interventions: a stated preference discrete choice experiment.

    Heal Econ 2009, 18:951-976. Publisher Full Text OpenURL

  18. Jehu-Appiah C, Baltussen R, Acquah C, Aikins M, d'Almeida SA, Bosu WK, Koolman X, Lauer J, Osei D, Adjei S: Balancing equity and efficiency in health priorities in Ghana: the use of multicriteria decision analysis.

    Value in Health 2009, 11:1081-1087. OpenURL

  19. Koopmanschap MA, Stolk EA, Koolman X: Dear policy maker: Have you made up your mind? A discrete choice experiment among policy makers and other health professionals.

    Int J Technol Assess Health Care 2010, 26(2):198-204. PubMed Abstract | Publisher Full Text OpenURL

  20. Goetghebeur MM, Wagner M, Khoury H, Rindress D, Grégoire J-P, Deal C: Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decisionmking framework to growth hormone for Turner syndrome patients.

    Cost Effectiveness and Resource Allocation 2010, 8:4. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  21. Culyer AJ: Equity of what in healthcare? Why the traditional answers don't help policy- and what to do in the future.

    Healthcare Papers 2007, 8(Spec No):12-26. PubMed Abstract OpenURL

  22. Brock D, Wikler D: Ethical issues in resource allocation, research, and new products development. In Disease control priorities in developing countries. 2nd edition. Edited by Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, Evans DB, Jha P, Mills A, Musgrove P. New York: Oxford University Press and the World Bank; 2006. OpenURL

  23. Kenny N, Joffres C: An ethical analysis of international health priority setting.

    Health Care Analysis 2008, 16:145-160. PubMed Abstract | Publisher Full Text OpenURL

  24. Kapiriri L, Norheim OF, Martin DK: Fairness and accountability for reasonableness. Do the views of priority setting decision makers differ across health systems and levels of decision making?

    Social Science & Medicine 2009, 68:766-773. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  25. Berney L, Kelly M, Doyal L, Feder G, Griffiths C, Jones IR: Ethical principles and the rationing of health care: a qualitative study in general practice.

    Br J Gen Prac 2005, 55:620-625. OpenURL

  26. Baltussen R, Youngkong S, Paolucci F, Niessen LW: Multi-criteria decision analysis to prioritize health interventions: Capitalizing on first experiences.

    Health Policy 2010, 96:262-264. PubMed Abstract | Publisher Full Text OpenURL

  27. Johri M, Norheim OF: Can cost-effectiveness analysis integrate equity concerns? A systematic review of current approaches.

    In Paper presented at ISPOR 12th Annual European Congress Paris 2009. PubMed Abstract | PubMed Central Full Text OpenURL

  28. Youngkong S, Baltussen R, Tantivess S, Koolman X, Teerawattananon Y: Criteria for priority setting of HIV/AIDS interventions in Thailand: A discrete choice experiment.

    BMC Health Service Research 2010, 10:197. BioMed Central Full Text OpenURL

  29. The Intervention Health Policy Program, Health Intervention and Technology Assessment Program: Research for development of health benefit package under universal health care coverage scheme. Nonthaburi; 2011. OpenURL

  30. Dolan JG: Multi-criteria clinical decision support: A primer on the use of multiple criteria decision making methods to promote evidence-based, patient-centered healthcare.

    Patient 2010, 3(4):229-248. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL