Advertisement

Comparing Use of Health Technology Assessment in Pharmaceutical Policy among Earlier and More Recent Adopters in the European Union

Open ArchivePublished:October 11, 2018DOI:https://doi.org/10.1016/j.vhri.2018.08.002

      ABSTRACT

      Objectives

      To examine and compare the use of health technology assessment (HTA) for the reimbursement of new medicines in selected European Union member states with decades of experience in the use of HTA and in countries that have used it regularly since 2000.

      Methods

      The selected countries were categorized into “earlier” adopters (group A: England, Germany, France, and Sweden) and more “recent” adopters (group B: Poland, Bulgaria, Hungary, and Romania). A systematic review of published literature was performed. The analysis and comparison of HTA procedures were done by using an analytical framework.

      Results

      In all countries, the assessment criteria used include effectiveness, safety, relative effectiveness, and economic data. In group A countries, the main objectives are improving quality of care, ensuring equal access, and efficient use of resources. Group B countries have established HTA organizations with official guidelines but often seek the decisions of other developed countries. They place considerable emphasis on the budget impact of new therapies, and HTA is also used as a cost estimation tool for state budgets.

      Conclusions

      HTA organizations have been developed dynamically not only in high-income countries but also in countries with limited resources. The experience and evolution of both can be used by countries that are in the dawn of creating an HTA organization.

      Keywords

      Introduction

      Health technology assessment (HTA) is considered a key tool used for decision making in health care policy, which can support the efficient use of resources while rewarding innovation. A key purpose of HTA in decision making is to achieve greater value for the money spent [
      • Banta D
      What is technology assessment?.
      ]. Over the past 30 years, several European countries have established specific bodies and developed various programs for the implementation of HTA [
      • Sorenson C
      • Chalkidou K
      Reflections on the evolution of health technology assessment in Europe.
      ]. There are, however, considerable differences between national HTA agencies among European Union (EU) member states. The differing philosophy of these organizations is the result of political, social, and economic factors that have shaped European health systems [
      • Allen N
      • Pichler F
      • Wang T
      • et al.
      Development of archetypes for non-ranking classification and comparison of European National Health Technology Assessment systems.
      ].
      A systematic comparison of HTA processes applied in decision making on the pricing and reimbursement (P&R) of medicines can identify similarities and differences that provide important information about the stages of development of this complex and multifactorial process. The aim of the present study was to compare how HTA is implemented in the procedures for reimbursement of medicines in selected countries at different levels of maturity in the application of HTA. The purpose of this exercise was to contribute to the evidence base that can be used in the process of planning and introducing an HTA system in a country, as in the case of Greece, which has a constricted health care budget and is in the process of institutionalizing HTA in decision making for the reimbursement of pharmaceutical products. We aimed to provide a snapshot of the selected HTA systems’ organization, the procedures and evaluation criteria applied, and the role of HTA in the decision-making process; another question of interest was whether and in what way the characteristics of the HTA systems differ between countries that are at a different stage of HTA implementation. A detailed comparison of analytical methods and techniques applied during the HTA process as well as the actual result of the reimbursement decisions per se were out of the scope of the present analysis.

      Methods

       Criteria for the Selection of Countries

      The first criterion was the countries’ political-geographical position; the countries selected were members of the EU. Second, given that the Greek legislation provides for a centrally organized HTA organization, countries with regionally organized HTA procedures were excluded. To account for different levels of maturity in the application of HTA, the number of years of experience in HTA implementation (not limited to decision making on pharmaceuticals) was considered. Sweden, France, the United Kingdom, and Germany are considered leaders in the establishment of HTA in Europe and have also been very influential regarding the methods and tools applied in HTA and its use in policymaking [
      • Sorenson C
      • Chalkidou K
      Reflections on the evolution of health technology assessment in Europe.
      ]. Selected Central and Eastern European countries were included because they constitute “recent adopters” of HTA.
      On the basis of the aforementioned criteria, the following countries and their respective HTA agencies were selected: France (Haute Autorité de Santé [French National Authority for Health]), Germany (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen [German Institute for Quality and Efficiency in Health Care]), the United Kingdom (National Institute for Health and Care Excellence [NICE]), Sweden (Tandvårds- och läkemedelsförmånsverket [TLV; Dental and Pharmaceutical Benefits Agency]), Bulgaria (National Centre of Public Health Analysis), Hungary (Technology Appraisal Head Department), Poland (Agencja Oceny Technologii Medycznych i Taryfikacji [Agency for Health Technology Assessment and Tariff System]), and Romania (HTA unit of the National Drug Agency). Two groups were formed: group A included the “earlier” adopters (France, Germany, the United Kingdom, and Sweden), whereas group B included the “recent” adopters (Bulgaria, Hungary, Poland, and Romania).

       Collection of Information

      Information was collected through a systematic literature review that applied modified guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses. An extensive search using a structured search strategy was performed for peer-reviewed articles published in English during the last 15 years (from January 2000 to February 2015).
      English terms including specific conditions (i.e., Medical Subject Headings terms) combined with free-text terms were used: the selected “country,” with the phrases “health technology assessment” or “HTA,” “health policy,” “pharmaceutical policy,” “pricing and reimbursement,” “health reform,” “pharmaceuticals reimbursement,” “economic evaluation,” and “impact on health budget” [
      • Stafinski T
      • Menon D
      • Davis C
      • McCabe C
      Role of centralized review processes for making reimbursement decisions on new health technologies in Europe.
      ]. The first extensive search was done in PubMed, from which most of the included articles were recovered. An additional search was done in the following specialized journals: International Journal of Technology Assessment in Health Care and Value in Health (main and regional issues). Finally, manual search was performed by checking the list of references in the articles identified as satisfying all the inclusion criteria.
      All recovered abstracts were reviewed independently and, subsequently, full-text articles were identified on the basis of specific inclusion and exclusion criteria. Articles were included if they 1) were in English, 2) were related to the implementation of HTA for decisions on medicine reimbursement, and 3) were published between January 2000 and February 2015. Articles were excluded if 1) articles presented results of economic evaluations of medicines and medical devices, 2) HTA implementation was in a hospital setting, and 3) full text was not accessible. The independent review was performed online by using Covidence, which is a tool for the organization and evaluation of information gathered in the context of a systematic literature review. When there were disagreements, the final decision was made after a discussion among all authors. A search for supplementary information was performed in HTA organizations’ Web sites and guidelines (where available in English and in other cases with the use of Google Translate) and in the Web sites of the International Society for Pharmacoeconomics and Outcomes Research, the Organization for Economic Cooperation and Development, and the European Observatory on Health Systems and Policies.

       Methods of Comparative Analysis

      For the analysis and comparison of HTA processes in the selected countries, an adapted methodology developed by Hutton et al. [
      • Hutton J
      • McGrath C
      • Frybourg J-M
      • et al.
      Framework for describing and classifying decision-making systems using technology assessment to determine the reimbursement of health technologies (fourth hurdle systems).
      ] and modified by Franken et al. [
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      ] was used. According to the Hutton framework, data are displayed in tables where the main characteristics of the HTA systems are depicted so as to facilitate comparison [
      • Hutton J
      • McGrath C
      • Frybourg J-M
      • et al.
      Framework for describing and classifying decision-making systems using technology assessment to determine the reimbursement of health technologies (fourth hurdle systems).
      ,
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      ]. In this context, the organization of an HTA agency is split into two levels of analysis—the policy implementation level and the individual technology decision level. The policy implementation level concerns the way in which ΗΤΑ is embedded in the broader political system, the HTA agency's legal status, and its relationships with other public sector bodies and stakeholders (such as industry and patient groups) and also provides information as to what the purpose of the HTA organization is, whom does it advise, and to whom it is accountable [
      • Hutton J
      • McGrath C
      • Frybourg J-M
      • et al.
      Framework for describing and classifying decision-making systems using technology assessment to determine the reimbursement of health technologies (fourth hurdle systems).
      ,
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      ]. The technology decision level comprises the processes by which individual technologies are evaluated by the system, for example, assessment processes, how decisions are made, and how they are implemented [
      • Hutton J
      • McGrath C
      • Frybourg J-M
      • et al.
      Framework for describing and classifying decision-making systems using technology assessment to determine the reimbursement of health technologies (fourth hurdle systems).
      ,
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      ]. Franken et al. [
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      ] also differentiated between an assessment and an appraisal phase in the HTA process.

      Results

      A total of 1724 articles were identified for initial review. Of these, 309 were duplicates and thus excluded. By applying the inclusion and exclusion criteria, 1165 articles were excluded after the assessment of abstracts, resulting in 249 articles for full-text evaluation. Applying the same independent assessment process, 64 articles were found to fully satisfy the inclusion criteria. An additional nine articles were identified after checking the reference lists of included articles. Thus, finally 73 articles were used in the analysis. The collected information was analyzed and classified according to the two dimensions of the Hutton framework.
      A schematic representation of the search and selection process is shown in Figure 1.
      Fig 1
      Fig. 1Schematic representation of the systematic review based on the PRISMA model. HTA, health technology assessment; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

       Policy Implementation Level

      The main characteristics of the political implementation level for the countries of groups A and B are presented in Table 1.
      Table 1Elements of an HTA system: policy implementation level—group A and B countrie
      Sources: Franken et al.
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      , Stafinksi 2011 et al
      • Stafinski T
      • Menon D
      • Davis C
      • McCabe C
      Role of centralized review processes for making reimbursement decisions on new health technologies in Europe.
      , Sorenson and Chalkidou
      • Sorenson C
      • Chalkidou K
      Reflections on the evolution of health technology assessment in Europe.
      , Gulacsi et al
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      , Nuijten et al
      • Nuijten MJC
      • Szende A
      • Kosa J
      • et al.
      Health care reform in six Central European countries. A focus on health economic requirements in the drug pricing and reimbursement processes.
      , Weill and Banta
      • Weill C
      • Banta D
      Development of health technology assessment in France.
      , Greiner and von der Schulenburg
      • Greiner W
      • von der Schulenburg J-MG
      HTA in Germany: very special and specific.
      , Busse and Blümel
      • Busse R
      • Blumel M
      Germany: health system review.
      , Panteli et al.
      • Panteli D
      • Arickx F
      • Cleemput I
      • et al.
      Pharmaceutical regulation in 15 European countries review.
      , Haute Autorité de Santé -, Rochaix and Herri
      • Rochaix L
      • Xerri B
      National Authority for Health: France.
      , Chevreul et al
      • Chevreul K
      • Durand-Zaleski I
      • Bahrami SB
      • et al.
      France: health system review.
      , Annel et al
      • Anell A
      • Glenngård AH
      • Merkur S
      -, and National Institute for Health and Care Excellence

      National Institute for Health and Care Excellence. Technology appraisal and highly specialised technologies programmes: procedure for varying the funding requirement to take account of net budget impact. Available from: https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-technology-appraisals/TA-HST-procedure-varying-the-funding-direction.pdf. [Accessed December 18, 2017].

      .
      ElementGroup AGroup B
      EnglandFranceGermanySwedenPolandRomaniaHungaryBulgaria
      Establishment: Relationship with the MoH and other organizationsNICE (1999)

      Nondepartmental public body

      Established by the DoH
      HAS (2004)

      Autonomous public scientific authority Established by the MoH (replaced the National Agency for Accreditation and Evaluation of Health—ANAES)
      IQWiG (2004)

      Established by G-BA as an autonomous scientific institution of the Foundation for Quality and Efficiency in Health Care
      TLV (2002)

      Government agency
      AOTMiT (2005)

      Autonomous organization under the supervision of the MoH
      HTA unit (2014)

      Established inside the NDA
      Office HTA (2004)

      In 2012 it was renamed as TAHD as a department in the National Institute for Quality and Organizational Development in Healthcare
      NCPHA (2015)

      Under the supervision of the MoH
      Objective: Broader political objectivesDevelopment of evidence-based guidelines and advice, quality standards, and provision of information services to decision makersTo provide health authorities with the information required to make decisions on the reimbursement of health technologies; to improve quality of care; to provide information on the quality of care to the public; to provide health economics assessments and opinions on the most efficient strategies for health care, prescribing, or managementProvision of independent scientific assessments of the benefit and cost of new technologies; development of clinical practice guidelines and provision of information to the public on quality and efficiency of careDeciding on the pricing and reimbursement of medicines; re-evaluating medicines already reimbursed; improving the quality of provided pharmaceutical care; supervising certain areas of the pharmaceutical marketOptimization of resources provided for the reimbursement of health technologies, taking into account the social, economic, and ethical aspectsOptimization of the resources that are used for the reimbursement of services or medicinesProvides high-quality, safe, effective, and cost-effective medicines to the reimbursement scheme intended either for prevention or treatmentSets price limits for prescription drugs, records the maximum retail prices, and decides on the inclusion, amendments, or exclusion of pharmaceuticals from the positive list of medicinal products
      Implementation: ScopeMedicines selected via a topic selection procedure

      NICE aims to consider all new significant drugs and indications
      All newly marketed medicines (clinical appraisal)

      Since October 2013: health economic assessment for all new highly innovative (ASMR I to III) medicines with a likely significant impact on health expenditure or health care organization/provision
      All newly marketed medicines and new indicationsNewly marketed medicines applying for reimbursement

      All medicines included in the benefit scheme before 2002
      For medicines that will be included in the reimbursement list, the MAH must submit a full HTA applicationNewly marketed innovative medicines, new indications, and re-evaluation of medicines that are already on the reimbursement listNewly marketed innovative medicines, new indications, or new route of administration and in the case of a price increase requestFor newly marketed innovative medicines, products not included in the positive drug list
      Implementation: Role in the final decision makingFinal decision taken by NICE (Transparency Committee) with legal effectRecommendation

      Final decision on reimbursement: UNCAM and MoH
      Recommendation

      Final decision: G-BA
      Final decision with legal effectRecommendation

      Final decision: MoH
      Recommendation

      Final decision: MoH
      Recommendation

      Final decision: MoH and Ministry of Economy
      Recommendation

      Final decision: MoH
      TransparencyMost of the produced documents and the final appraisal report are publicly availableThe final assessment report and the decision are published on the HAS Web siteIQWiG publishes assessments on its Web site

      The G-BA assessment and the final decision are publicly available
      The decisions and assessment reports are publicly availableThe decision is publicly available and implemented if it is accepted by the MoHThe final assessment report is published on the Web site of the MoH; the publication of the decision is done at the Official GazetteThe decision is publicly available on the MoH Web site; the assessment report is not publicly availableThe decision and assessments are not publicly available
      ANAES; Agence nationale d'accréditation et d'évaluation en santé (National Agency for Accreditation and Evaluation of Health).; AOTMiT, Agencja Oceny Technologii Medycznych i Taryfikacji (Agency for Health Technology Assessment and Tariff); ASMR, Amélioration du Service Médical Rendu (Clinical Added Value); DoH, Department of Health; G-BA, Gemeinsamer Bundesausschuss (Federal Joint Committee); HAS, Haute Autorité de Santé (French National Authority for Health); HTA, health technology assessment; IQWiG, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (German Institute for Quality and Efficiency in Health Care); MAH, marketing authorization holder; MoH, Ministry of Health; NCPHA, National Center of Public Health Analyses; NDA, National Drug Agency; NICE, National Institute for Health and Care Excellence; TAHD, Technology Appraisal Head Department; TLV, Tandvårds- och läkemedelsförmånsverket (Dental and Pharmaceutical Benefits Agency); UNCAM, Union nationale des caisses d'assurance maladie (French National Union of Health Insurance Funds).
      In England, France, and Germany, the HTA bodies are public bodies that operate independently from the government, whereas in Sweden the TLV is a governmental agency. All four HTA bodies in group A countries were established by government bodies in the context of broader reforms toward evidence-based medicine, improvement of safety and quality of care, as well as promotion of equity and efficiency in the use of health care budgets [

      Bidgood E, Clarke E. Healthcare Systems: France. 2013. Based on the 2001 Civitas Report by David Green and Benedict Irvine Updated by Emily Clarke (2012) and Elliot Bidgood (January 2013).

      ,

      Boyle S. United Kingdom (England): Health System Review. Health Systems in Transition Eur. Obversatory Heal Syst Policies 2011;Vol 13.

      ,
      • Jonsson E
      History of health technology assessment in Sweden.
      ,
      • Busse R
      • Blumel M
      Germany: health system review.
      ,
      • Perleth M
      • Gibis B
      • Gohlen B
      A short history of health technology assessment in Germany.
      ]. HTA was seen as a tool toward achieving these goals while rewarding innovation and has an important role in the decision-making process [
      • Jonsson E
      History of health technology assessment in Sweden.
      ,
      • Perleth M
      • Gibis B
      • Gohlen B
      A short history of health technology assessment in Germany.
      ,
      • Weill C
      • Banta D
      Development of health technology assessment in France.
      ,
      • Gloekler S
      • Traupe T
      • Stoller M
      • et al.
      The effect of heart rate reduction by ivabradine on collateral function in patients with chronic stable coronary artery disease.
      ,
      • Ivandic V
      Requirements for benefit assessment in Germany and England—overview and comparison.
      ,

      National Institute for Health and Care Excellence. Technology appraisal and highly specialised technologies programmes: procedure for varying the funding requirement to take account of net budget impact. Available from: https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-technology-appraisals/TA-HST-procedure-varying-the-funding-direction.pdf. [Accessed December 18, 2017].

      ]. This is reflected in the organizations’ objectives and scope of activities, which are broader than the assessment of medicines. These organizations also place emphasis on operating with processes and procedures that are considered best practice (e.g., independence, transparency, and openness) [
      • Sorenson C
      • Chalkidou K
      Reflections on the evolution of health technology assessment in Europe.
      ,
      • Stafinski T
      • Menon D
      • Davis C
      • McCabe C
      Role of centralized review processes for making reimbursement decisions on new health technologies in Europe.
      ,
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      ,
      • Ivandic V
      Requirements for benefit assessment in Germany and England—overview and comparison.
      ,
      • Drummond M
      • Sorenson C
      Nasty or nice? A perspective on the use of health technology.
      ,
      • Luce BR
      • Drummond M
      • Jonsson B
      • et al.
      EBM, HTA, and CER: clearing the confusion.
      ,
      • Drummond MF
      • Schwartz JS
      • Jonsson B
      • et al.
      Key principles for the improved conduct of health technology assessments for resource allocation decisions.
      ,

      Sorenson C, Drummond M, Kanavos P. Ensuring Value for Money in Health Care: The Role of Health Technology Assessment in the European Union. 2008, Observatory Studies Series No 11. European Observatory on Health Systems and Policies.

      ].
      In group B countries, the HTA process was integrated into the health system through reforms of existing legislation and in the framework of system restructuring [
      • Corabian P
      • Hailey D
      • Harstall C
      • et al.
      Mentoring a developing health technology assessment initiative in Romania: an example for countries with limited experience of assessing health technology.
      ,
      • Gaal P
      • Szigeti S
      • Csere M
      • et al.
      Hungary health system review.
      ,
      • Gaal P
      • Szigeti S
      • Panteli D
      • et al.
      Major challenges ahead for Hungarian healthcare.
      ,
      • Kolasa K
      • Kalo Z
      • Zah V
      • Dolezal T
      Role of health technology assessment in the process of implementation of the EU Transparency Directive: relevant experience from Central Eastern European countries.
      ]. In Hungary and Poland, the HTA organizations were established as autonomous organizations under the Ministry of Health (MoH) in 2004 and 2005, respectively. In Bulgaria and Romania, the respective agencies were established in 2013 [
      • Vladescu C
      • Olsavszky V
      • Scintee G
      Health Systems in Transition: Romania.
      ,
      • Dimova A
      • Rohova M
      • Moutafova E
      • et al.
      Bulgaria health system review.
      ] and consist of a dependent unit or a division within the MoH. In all group B countries, the HTA process is centralized with no reassessment of the decision in other regions. The purpose of the HTA process in group B countries is mainly the optimization of resources for technologies such as medicines, for which the marketing authorization holder (MAH) applies for reimbursement [
      • Gulacsi L
      • David T
      • Dozsa C
      Pricing and reimbursement of drugs and medical devices in Hungary.
      ,

      Sima M. Capacity building in pharmacoeconomics and HTA in CEE: opportunities in education and CEE context HTA implementation in CEE countries—opportunities HTA implementation in CEE countries—challenges. 2014:1–14. ISPOR 17th Annual European Congress 8-12 November 2014 Amsterdam RAI Amsterdam.

      ]. Especially for Poland and Hungary, the HTA process has a history of about 10 years, and there are efforts to take into account “social, economic and ethical aspects” in decision making [
      • Kolasa K
      • Schubert S
      • Manca A
      • Hermanowski T
      A review of health technology assessment (HTA) recommendations for drug therapies issued between 2007 and 2009 and their impact on policymaking processes in Poland.
      ,
      • Kolasa K
      • Dziomdziora M
      • Fajutrao L
      What aspects of the health technology assessment process recommended by international health technology assessment agencies received the most attention in Poland in 2008?.
      ]. The HTA bodies play an important role in the decision-making process; nevertheless, their recommendations are not always adopted, because specific confidential agreements with the MoH or other political implications may lead to a different decision [
      • Ozieranski P
      • McKee M
      • King L
      The politics of health technology assessment in Poland.
      ].

       Technology Decision Level

      The main findings on the technology implementation level for both group A and B countries are presented in Tables 2 and 3.
      Table 2Elements of an HTA system: technology decision level—group A and B countries: assessment and appraisal
      Sources: Gulacsi et al
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      , Nicod and Kanavos
      • Nicod E
      • Kanavos P
      Commonalities and differences in HTA outcomes: A comparative analysis of five countries and implications for coverage decisions.
      , Stafinksi
      • Stafinski T
      • Menon D
      • Davis C
      • McCabe C
      Role of centralized review processes for making reimbursement decisions on new health technologies in Europe.
      , Nuijten et al
      • Nuijten MJC
      • Szende A
      • Kosa J
      • et al.
      Health care reform in six Central European countries. A focus on health economic requirements in the drug pricing and reimbursement processes.
      , Kolasa
      • Kolasa K
      • Schubert S
      • Manca A
      • Hermanowski T
      A review of health technology assessment (HTA) recommendations for drug therapies issued between 2007 and 2009 and their impact on policymaking processes in Poland.
      ,
      • Kolasa K
      • Dziomdziora M
      • Fajutrao L
      What aspects of the health technology assessment process recommended by international health technology assessment agencies received the most attention in Poland in 2008?.
      ,
      • Kolasa K
      • Wasiak R
      Health technology assessment in Poland and Scotland: comparison of process and decisions.
      , Franken et al.
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      -, Panteli et al.
      • Panteli D
      • Arickx F
      • Cleemput I
      • et al.
      Pharmaceutical regulation in 15 European countries review.
      Bridges et al
      • Bridges JFP
      • Cohen JP
      • Grist PG
      • Mühlbacher AC
      International experience with comparative effectiveness research: case studies from England/Wales and Germany.
      , Haute Autorité de Santé [78], Rochaix and Herri
      • Rochaix L
      • Xerri B
      National Authority for Health: France.
      , Annel et al
      • Anell A
      • Glenngård AH
      • Merkur S
      , NICE 2017a,- and- National Institute for Health and Care Excellence

      National Institute for Health and Care Excellence. Technology appraisal and highly specialised technologies programmes: procedure for varying the funding requirement to take account of net budget impact. Available from: https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-technology-appraisals/TA-HST-procedure-varying-the-funding-direction.pdf. [Accessed December 18, 2017].

      .
      ElementEnglandFranceGermanySwedenPolandRomaniaHungaryBulgaria
      Assessment
      Competent organization (preparation, processing, and reporting)NICEHASIQWiG (commissioned by G-BA)TLV/and/or Swedish Agency for Health Technology Assessment and Assessment of Social ServicesAOTMiTHTA Division in the MoHTAHD (former OHTA)NCPHA
      Competent bodyEvidence Review GroupHAS Medicines Assessment Department (clinical assessment)

      HAS Division of Economic and Public Health (health economic assessment)
      IQWiG (consults with external medical experts and patient organizations in the assessment phase). If necessary, its Scientific Advisory Board is involvedTLV working groupsTransparency Council (Appraisal Committee)Specialist Committee/ΗΤΑ DivisionTACExpert opinion: HTA Committee
      Appraisal
      Competent organization/committeeAppraisal Committee (independent advisory body)Transparency Committee (reimbursement)

      Economic and Public Health Assessment Committee CEPS (pricing)
      G-BATLV Board on Pharmaceutical BenefitsTransparency CouncilSpecialist Committee/ΗΤΑ DivisionTAC/NHIFExpert opinion
      Evidence and evaluation criteria
      Clinical evidence/safetyYesYesYesYesYesYesYesYes
      Relative effectivenessYesYesYesYesYesYesYesYes
      Economic analysisCUA/cost comparisonCEA/CUA (pricing)CBA in specific casesCUA, cost comparison, cost benefit depending on the treatmentCEA/CUACEA/CUACEA/CUACEA/CUA
      Appraisal criteriaQuality and certainty of clinical evidence

      Benefits and adverse events from the patient perspective

      Cost effectiveness

      Budget impact

      Equality issues

      Disease severity

      Degree of innovation

      Costs and benefits incurred in other sectors

      Nonhealth benefits of the technology (only when specifically requested)
      Therapeutic benefit

      Disease severity

      Availability of other treatments

      Purpose of use (preventive, symptomatic, or curative)

      Impact on public health

      Cost effectiveness

      Prices in European countries
      Therapeutic benefit

      Patient-relevant therapeutic effect

      Certainty of the evidence

      Budget impact for social insurance

      Annual treatment costs of the comparator

      Prices in European countries
      Human value principle

      Need and solidarity principle

      Cost-effectiveness principle

      Disease severity

      Managed entry agreement between the county councils and the pharmaceutical company
      Clinical effectiveness and safety

      Economic evidence

      Affordability (budget impact)

      Disease severity

      Burden of disease
      Scorecard with specific criteria:

      • Budget impact analysis

      • Decisions of HAS/NICE/SMC/IQWiG–G-BA

      • Reimbursement in other EU countries
      Clinical effectiveness and safety

      Economic evidence

      Budget impact analysis

      Disease severity

      Burden of disease
      Scorecard with specific criteria:

      • Clinical effectiveness and safety

      • Economic evidence

      • Budget impact analysis

      Opinion in France, the United Kingdom, and Germany G-BA

      HTA assessments of other EU countries
      Threshold cost/QALY£20,000–£30,000 (implicit)

      A medicine with an ICER >£30,000 should make reference to the factors taken into consideration
      NoNoNo3 times the GDP/capitaNo2–3 times the GDP/capitaNo
      Appraisal report publicly availableYesYesYesYesYesYesNoNo
      AOTMiT, Agencja Oceny Technologii Medycznych i Taryfikacji (Agency for Health Technology Assessment and Tariff); CBA, cost-benefit analysis; CEA, cost-effectiveness analysis; CEPS, Comité économique des produits de santé (Economic Committee on Healthcare Products); CUA, cost-utility analysis; EU, European Union; G-BA, Gemeinsamer Bundesausschuss (Federal Joint Committee); GDP, gross domestic product; HAS, Haute Autorité de Santé (French National Authority for Health); HTA, health technology assessment; ICER, incremental cost-effectiveness ratio; IQWiG, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (German Institute for Quality and Efficiency in Health Care); MoH, Ministry of Health; NCPHA, National Center of Public Health Analyses; NHIF, National Health Insurance Fund; NICE, National Institute for Health and Care Excellence; OHTA: Office of Health Technology Assessment; QALY, quality-adjusted life-year; SMC, Scottish Medicine Consortium; TAC, Technology Appraisal Committee; TAHD, Technology Appraisal Head Department; TLV, Tandvårds- och läkemedelsförmånsverket (Dental and Pharmaceutical Benefits Agency).
      Table 3Elements of an HTA system: technology decision level—group A and B countries: decision, appeal, and implementation
      Sources: Gulacsi [
      • Gulacsi L
      • Brodszky V
      • Pentek M
      • et al.
      History of health technology assessment in Hungary.
      ,
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      ], Stafinksi 2011 et al
      • Stafinski T
      • Menon D
      • Davis C
      • McCabe C
      Role of centralized review processes for making reimbursement decisions on new health technologies in Europe.
      , Nuijten et al
      • Nuijten MJC
      • Szende A
      • Kosa J
      • et al.
      Health care reform in six Central European countries. A focus on health economic requirements in the drug pricing and reimbursement processes.
      , Kolasa
      • Kolasa K
      • Schubert S
      • Manca A
      • Hermanowski T
      A review of health technology assessment (HTA) recommendations for drug therapies issued between 2007 and 2009 and their impact on policymaking processes in Poland.
      ,
      • Kolasa K
      • Dziomdziora M
      • Fajutrao L
      What aspects of the health technology assessment process recommended by international health technology assessment agencies received the most attention in Poland in 2008?.
      ,
      • Kolasa K
      • Wasiak R
      Health technology assessment in Poland and Scotland: comparison of process and decisions.
      , Franken et al.
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      , National Institute for Health and Care Excellence

      National Institute for Health and Care Excellence. Technology appraisal and highly specialised technologies programmes: procedure for varying the funding requirement to take account of net budget impact. Available from: https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-technology-appraisals/TA-HST-procedure-varying-the-funding-direction.pdf. [Accessed December 18, 2017].

      , Panteli et al.
      • Panteli D
      • Arickx F
      • Cleemput I
      • et al.
      Pharmaceutical regulation in 15 European countries review.
      , Bridges et al
      • Bridges JFP
      • Cohen JP
      • Grist PG
      • Mühlbacher AC
      International experience with comparative effectiveness research: case studies from England/Wales and Germany.
      .
      ElementEnglandFranceGermanySwedenPolandRomaniaHungaryBulgaria
      Decision-making bodyNICE/Appraisal CommitteeUNCAM

      MoH

      CEPS
      G-BA

      GKV-SV
      TLVMoHMoH according to the scorecardMoHMoH according to the recommendation of NCPHA
      Decision typeRecommendation on the medicine's use in the NHSUNCAM/reimbursement level

      MoH/inclusion into the positive list

      CEPS/pricing
      G-BΑ/reimbursement

      GKV-SV/reimbursed price
      TLV/joint decision on reimbursement and pricingAOTMiT/reimbursementHTA Division/reimbursementTAC/reimbursementNCPHA/reimbursement and pricing
      Stakeholder involvementYesYesYesYesYesNoYesNo
      Possibility of restricted reimbursement

      (i.e., specified indications, patient groups, and settings)/or managed entry agreements
      Yes/YesYes/YesYes/YesYes/YesYes/YesYes/Yes

      Price: volume agreements
      Yes/Yes

      Price: volume agreements
      Yes/No
      Appeal/dissentYesYesYesYesNo information foundYesYesYes
      Revisions/reassessmentYesYesYes (depends on the case)Yes (medicines before 2002)Every 5 yNoNo information foundNo information found
      AOTMiT, Agencja Oceny Technologii Medycznych i Taryfikacji (Agency for Health Technology Assessment and Tariff); CEPS, Comité économique des produits de santé (Economic Committee on Healthcare Products); G-BA, Gemeinsamer Bundesausschuss (Federal Joint Commission); GKV-SV, GKV-Spitzenverband lth technology assessment; MoH, Ministry of Health; NCPHA, National Centre of Public Health Analysis; NHS, National Health Service; NICE, National Institute for Health and Care Excellence; TAC, Technical Appraisal Committee; TLV, Tandvårds- och läkemedelsförmånsverket (Dental and Pharmaceutical Benefits Agency); UNCAM, Union nationale des caisses d'assurance maladie (French National Union of Health Insurance Funds).

       Assessment and appraisal

      The type of evidence requested is similar across jurisdictions. Evidence on clinical effectiveness, relative clinical effectiveness, safety, target population, disease characteristics, availability of other treatments for the same indication, and so forth is part of the evidence base. All countries have published guidelines in which the documentation and methodological requirements for the application are described [
      • Iskrov GG
      • Raycheva RD
      • Stefanov RS
      Insight into reimbursement decision-making criteria in Bulgaria: implications for orphan drugs.
      ]. Nevertheless, differences exist in critical points, such as the criteria for selecting or prioritizing technologies, the quality of the required documentation, and the methodological approaches used. The differences in the methodological requirements of group A HTA agencies are discussed in detail elsewhere [
      • Ivandic V
      Requirements for benefit assessment in Germany and England—overview and comparison.
      ,
      • Makady A
      • Ham RT
      • de Boer A
      • et al.
      Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies.
      ,
      • Angelis A
      • Lange A
      • Kanavos P
      Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries.
      ]. Health economic analysis has recently been included as a requirement in the HTA process of newly marketed medicines that are characterized as innovative and are expected to have a significant impact on health care expenditure and provision in France. The results of the health economic analysis, in addition to the decision on the added therapeutic benefit, form the basis of price negotiations with the MAH. In Germany, since 2011, all newly marketed medicines and new indications are assessed for their added therapeutic benefit, but health economic analysis is performed only in those cases in which an agreement on the reimbursed price is not reached.
      In group A countries, the assessment report is prepared either by internal staff (in France, Germany, and Sweden) or by external academic organizations (in England) on the basis of the dossier submitted by the MAH; additional evidence may, however, be collected via a systematic literature review and/or consultation with stakeholders. The appraisal is done by committees or boards integrated within the HTA organization [
      • Jonsson E
      History of health technology assessment in Sweden.
      ,
      • Barron AJG
      • Klinger C
      • Shah SMB
      • Wright JSF
      A regulatory governance perspective on health technology assessment (HTA) in France: the contextual mediation of common functional pressures.
      ,
      • Greiner W
      • von der Schulenburg J-MG
      HTA in Germany: very special and specific.
      ,
      • Rochaix L
      • Xerri B
      National Authority for Health: France.
      ] or by the decision-making body (e.g., in Germany) [
      • Drummond M
      • Sorenson C
      Nasty or nice? A perspective on the use of health technology.
      ]. In France and Germany, the main criterion for a positive recommendation is the medicine's therapeutic benefit. In England and Sweden, cost effectiveness is an important criterion explicitly considered in the appraisal phase [
      • Sorenson C
      • Chalkidou K
      Reflections on the evolution of health technology assessment in Europe.
      ,
      • Shah SMB
      • Barron A
      • Klinger C
      • Wright JSF
      A regulatory governance perspective on health technology assessment (HTA) in Sweden.
      ]. HTA is also applied for the review of already marketed medicines. This approach allows the assessment of the therapeutic value of a large number of medicines, which may result in price modifications, changes in the reimbursement level, or even delisting [
      • Sorenson C
      • Chalkidou K
      Reflections on the evolution of health technology assessment in Europe.
      ].
      In group B countries, the concepts of assessment and appraisal are intertwined and are mainly based on the review of the evidence provided by the applicant. In Poland and Hungary, the applicant should present data demonstrating the cost effectiveness of the technology. The Agency for Health Technology Assessment and Tariff System in Poland and the Technology Appraisal Head Department Committee in Hungary perform de novo analysis of the submitted data. The main limitation of this analysis is that there is a lack of local data such as costs, costs per unit, the general health state of the population, epidemiological data per disease, and quality-of-life data for the weighting of financial data. There are also a limited number of registries [
      • Kolasa K
      • Kalo Z
      • Zah V
      • Dolezal T
      Role of health technology assessment in the process of implementation of the EU Transparency Directive: relevant experience from Central Eastern European countries.
      ,
      • Gagnon M-P
      • Gagnon J
      • St-Pierre M
      • et al.
      Involving patients in HTA activities at local level: a study protocol based on the collaboration between researchers and knowledge users.
      ,
      • Kolasa K
      • Wasiak R
      Health technology assessment in Poland and Scotland: comparison of process and decisions.
      ]. In Bulgaria, the HTA Committee critically evaluates the application and provides a recommendation advising the final decision-making body. A common key feature is the evaluation of the added therapeutic value when compared with existing alternatives. In all group B countries, the decisions of other European HTA bodies are consulted. Moreover, in Romania and Bulgaria, the final decisions of HTA bodies in the United Kingdom, France, and Germany are considered to be important for the final positive or negative decision. In these countries, a scorecard is the primary tool used for the final decision.
      The results of budget impact analysis are also important for the final appraisal decision in all countries. The budgetary impact of the introduction of a new medicine in the health system is valuable in estimating the financial implications and, hence, whether the product will be included in the health benefit basket. In group B countries, although specific appraisal criteria representative of their health system are set, their final objectives often remain unclear, which could lead to a lack of transparency [
      • Iskrov GG
      • Raycheva RD
      • Stefanov RS
      Insight into reimbursement decision-making criteria in Bulgaria: implications for orphan drugs.
      ,
      • Iskrov G
      • Miteva-Katrandzhieva T
      • Stefanov R
      Challenges to orphan drugs access in Eastern Europe: the case of Bulgaria.
      ].

       Decision

      Sweden is the only country where there is a joint decision on P&R. In France and Germany, P&R decisions are interlinked, whereas in England, NICE provides guidance on the use and reimbursement of the technology. In France and Germany, the final decision is taken by another body, taking into consideration the result of the assessment phase. In France, the assessment of clinical benefit by the French National Authority for Health is used by the Union Nationale des Caisses d'Assurance Maladie (French National Union of Health Insurance Funds), which is responsible for reimbursement decisions [
      • Chevreul K
      • Durand-Zaleski I
      • Bahrami SB
      • et al.
      France: health system review.
      ]. In Germany, Gemeinsamer Bundesausschuss (Federal Joint Committee) is the body that takes the final decision on reimbursement after the finalization of the assessment report of the German Institute for Quality and Efficiency in Health Care. In Sweden and England, the decisions of the HTA organizations that conduct the appraisal are final and mandatory for regional or local authorities [
      • Anell A
      • Persson U
      Reimbursement and clinical guidance for pharmaceuticals in Sweden: Do health-economic evaluations support decision making?.
      ,
      • Walley T.
      Health technology assessment in England: assessment and appraisal.
      ]. In all the countries, the decision can be positive or negative, or reimbursement can be granted with restrictions (e.g., for specific indications or for specific patient groups) [
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      ,

      Espin J. Rovira J. Analysis of differences and commonalities in pricing and reimbursement systems in Europe. DG Enterprise and Industry of the European Commission. Final report June 2007, available from https://ec.europa.eu/docsroom/documents/7605?locale=en, Accessed January 17, 2016.

      ]. There may also be a decision on temporary reimbursement because of the uncertainty about the evidence provided, conditional upon the collection of additional evidence [
      • Franken M
      • le Polain M
      • Cleemput I
      • Koopmanschap M
      Similarities and differences between five European drug reimbursement systems.
      ]. In England, France, and Sweden, risk-sharing agreements (coverage with evidence development) with the manufacturers have been introduced [
      • Ferrario A
      • Kanavos P
      Dealing with uncertainty and high prices of new medicines: a comparative analysis of the use of managed entry agreements in Belgium, England, the Netherlands and Sweden.
      ]. Finally, it is worth noting that all four countries of group A have developed faster assessment processes for the early inclusion of innovative technologies in the reimbursement list, with a commitment from the MAH to submit documentation attesting the benefit of the medicine after it is marketed [
      • Ferrario A
      • Kanavos P
      Dealing with uncertainty and high prices of new medicines: a comparative analysis of the use of managed entry agreements in Belgium, England, the Netherlands and Sweden.
      ,
      Department of Health; Association of the British Pharmaceutical Industry
      The Pharmaceutical Price Regulation Scheme.
      ].
      In group B countries, the HTA agency has a consultative character and the final decision is taken by the MoH. Often a negative opinion or a bad score will result in negotiations between the MAH and the MoH regarding restrictions or risk-sharing agreements. In most cases, these are price-volume agreements [
      • Gulacsi L
      • Brodszky V
      • Pentek M
      • et al.
      History of health technology assessment in Hungary.
      ,
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      ,
      • Kolasa K
      • Kalo Z
      • Hornby E
      Pricing and reimbursement frameworks in Central Eastern Europe: a decision tool to support choices.
      ]. Specific timelines for the HTA procedure that are in line with the European Transparency Directive are foreseen in all group B countries. In Poland and Hungary, the HTA has a long history of implementation, and specific scientific tools that are developed by international scientific societies such as the International Society for Pharmacoeconomics and Outcomes Research are used for the critical appraisal of the applications [
      • Gaal P
      • Szigeti S
      • Csere M
      • et al.
      Hungary health system review.
      ,
      • Gulacsi L
      • Brodszky V
      • Pentek M
      • et al.
      History of health technology assessment in Hungary.
      ,
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      ,
      • Panteli D
      • Sagan A
      • Mckee M
      • et al.
      Poland: health system review.
      ]. In the appraisal phase, the medicines are evaluated from social, ethical, and organizational perspectives [
      • Gaal P
      • Szigeti S
      • Panteli D
      • et al.
      Major challenges ahead for Hungarian healthcare.
      ,
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      ,
      • Panteli D
      • Sagan A
      • Mckee M
      • et al.
      Poland: health system review.
      ]. In Romania and Bulgaria, the critical appraisal is done by a specific committee that considers comparative efficacy, pharmacoeconomic and financial data, and ethical implications [
      • Vladescu C
      • Olsavszky V
      • Scintee G
      Health Systems in Transition: Romania.
      ,
      • Dimova A
      • Rohova M
      • Moutafova E
      • et al.
      Bulgaria health system review.
      ,
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      ,
      • Radu C-P
      • Chiriac ND
      • Pravat AM
      The Development of the Romanian Scorecard HTA System.
      ,
      • Aleksandrova S
      The Bulgarian Health Care Reform and Health Act 2004.
      ].

       Result and implementation

      In all the countries, the decision implementation predominantly relates to the medicine's reimbursement rate or price. In Germany, medicines with no added therapeutic benefit are included in the reference price system. For medicines with a positive benefit assessment, a reimbursement price is set after negotiations between the Federal Joint Committee and the MAH. The negotiations must be completed within 1 year of the launch of the product. In Germany, there is only a negative reimbursement list; when a medicine is granted marketing authorization, it is considered reimbursable, unless it is included in the negative reimbursement list. In France, the medicine is included in a positive reimbursement list and price negotiations are initiated. In England, when a medicine is recommended as an option by NICE, clinical commissioning groups must ensure that it is available to patients within 3 months (funding requirement). Nevertheless, since April 2017, National Health Service (NHS) England may request for a variation to this requirement. To be specific, manufacturers are required to include a budget impact analysis in their submission. For medicines with a financial impact exceeding £20 million in any of the first 3 years of their use, NHS England will proceed to commercial discussions on behalf of clinical commissioning groups. If discussions fail, NHS England can request a variation to the funding requirement up to a maximum of 3 years [

      National Institute for Health and Care Excellence. Technology appraisal and highly specialised technologies programmes: procedure for varying the funding requirement to take account of net budget impact. Available from: https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-technology-appraisals/TA-HST-procedure-varying-the-funding-direction.pdf. [Accessed December 18, 2017].

      ]. In Sweden, the county councils have some discretion as to the implementation of decisions made by TLV and can reach a more restrictive reimbursement decision, mainly because of budgetary considerations [
      • Anell A
      • Persson U
      Reimbursement and clinical guidance for pharmaceuticals in Sweden: Do health-economic evaluations support decision making?.
      ].
      Reimbursement restrictions are common in all the studied countries, with variations observed as to the type of restriction: reimbursement for specific indications, patient groups, or prescriber groups. In the case of reimbursement under specific conditions, managed entry agreements are implemented. In Poland, Romania, Hungary, and Bulgaria, the medicine is included in a positive reimbursement list [
      • Gaal P
      • Szigeti S
      • Csere M
      • et al.
      Hungary health system review.
      ,
      • Vladescu C
      • Olsavszky V
      • Scintee G
      Health Systems in Transition: Romania.
      ,
      • Dimova A
      • Rohova M
      • Moutafova E
      • et al.
      Bulgaria health system review.
      ,
      • Panteli D
      • Sagan A
      • Mckee M
      • et al.
      Poland: health system review.
      ,
      • Panteli D
      • Arickx F
      • Cleemput I
      • et al.
      Pharmaceutical regulation in 15 European countries review.
      ]. In Poland, an update and a reassessment of the data are requested every 5 years [
      • Ozieranski P
      • McKee M
      • King L
      The politics of health technology assessment in Poland.
      ]. In Romania, if the reimbursement is under specific conditions, a reassessment is done after 1 year and the MAH is required to submit additional data. Nevertheless, no other details are given in the relevant law [
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      ]. For Hungary and Bulgaria, no information could be found on a reassessment procedure of all the reimbursed medicinal products.

      Discussion

      The results of the present review confirm that the two groups of countries are going through different phases in the evolution of HTA. Group A countries (England, France, Germany, and Sweden) have a long experience of more than two decades in the implementation of HTA [
      • Shah SMB
      • Barron A
      • Klinger C
      • Wright JSF
      A regulatory governance perspective on health technology assessment (HTA) in Sweden.
      ,
      • Schwarzer R
      • Siebert U
      Methods, procedures, and contextual characteristics of health technology assessment and health policy decision making: comparison of health technology assessment agencies in Germany, United Kingdom, France, and Sweden.
      ] and have made a significant investment in the development of the HTA process. Results of the HTA process are used in P&R decisions and also as input in the development of clinical guidelines (e.g., in England) [
      • Sorenson C
      • Chalkidou K
      Reflections on the evolution of health technology assessment in Europe.
      ,
      • Ciani O
      • Jommi C
      The role of health technology assessment bodies in shaping drug development.
      ]. The main objective of the introduction of HTA in group A countries was improving quality of care, ensuring equal access to care, and assessment of the value for money of reimbursed medicines [
      • Sorenson C
      • Chalkidou K
      Reflections on the evolution of health technology assessment in Europe.
      ,
      • Henshall C
      • Schuller T
      • Mardhani-Bayne L
      Using health technology assessment to support optimal use of technologies in current practice: the challenge of “disinvestment.
      ]. Although health care cost considerations have led to the systematic assessment of innovative medicines recently in Germany and France, the assessment results are used to negotiate “fair” prices for social insurance. The HTA process in group A countries has reached a high level of maturity, and therefore the question is how the institutionalization of HTA has influenced policy and what was its impact on health care delivery. At the same time, these countries are trying to convey their expertise and knowledge at the European and international levels. They are in what is characterized by Battista and Hodge [
      • Battista RN
      • Hodge MJ
      The “natural history” of health technology assessment.
      ] the “expansion phase.”
      HTA has developed over the last two decades and has been institutionalized and implemented in Poland, Romania, Hungary, and Bulgaria. Poland and Hungary have a longer history of HTA compared with Romania and Bulgaria. These group B countries have adopted the formal conditions for implementing HTA and have established relevant HTA agencies, although these differ in organizational structure and operations, size of human resources, and extent of influencing decision making [
      • Gulácsi L
      • Péntek M
      HTA in Central and Eastern European countries; the 2001: a space odyssey and efficiency gain.
      ,
      • Inotai A
      • Pékli M
      • Jóna G
      • et al.
      Attempt to increase the transparency of fourth hurdle implementation in Central-Eastern European middle income countries: publication of the critical appraisal methodology.
      ]. These countries have generally followed the standards of other more mature HTA organizations, and occasionally they have established advisory relationships with them. In Romania and Bulgaria, it is officially accepted to use the appraisal decision of other European countries (the United Kingdom, France, and Germany), and this plays an important role in the final decision. The key question in this case is whether this information can be used as such, and to what extent it can reflect the conditions of the third country that uses them [
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      ,
      • Kaló Z
      • Landa K
      • Doležal T
      • Vokó Z
      Transferability of National Institute for Health and Clinical Excellence recommendations for pharmaceutical therapies in oncology to Central-Eastern European countries.
      ,
      • Gulácsi L.
      The time for cost-effectiveness in the new European Union member states: the development and role of health economics and technology assessment in the mirror of the Hungarian experience.
      ,
      • Gulacsi L
      • Rencz F
      • Pentek M
      • et al.
      Transferability of results of cost utility analyses for biologicals in inflammatory conditions for Central and Eastern European countries.
      ]. Group B countries, having completed the first phase of the standardization and institutionalization of HTA, are now in the stage of further developing the HTA process, using local data as input for decision making so that final conclusions are based on national priorities and values [
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      ].
      The present review provides an overview of the HTA procedure in countries with a different degree of maturity in the use of HTA. The commonalities and differences between these HTA systems can inform the development of short- and long-term plans for the introduction and subsequent development of the HTA procedure in countries that are planning to introduce a “fourth hurdle” in their P&R system. During the planning phase, the new HTA comers should first decide on the type of their HTA agency and establish specific rules governing its relationship with other organizations and decision-making bodies. The vision and the main scope of the organization should be clearly defined. Another important point to be decided upon is whether the HTA agency will have an advisory or regulatory role. While designing a new HTA organization, the total available budget, the available personnel with appropriate training and knowledge, the availability of data, and the ability of the health care system to use the results should be taken into account [
      • Gulácsi L
      • Boncz I
      • Drummond M
      Issues for countries considering introducing the “fourth hurdle”: the case of Hungary.
      ]. Second, at the technology level, clear rules and guidelines for the applicant should be available, describing also the evaluation criteria to be applied.
      Independence and transparency are key issues to be considered. In most jurisdictions the data are submitted by the MAHs of the technology, and thus the independence of the assessment is of great importance [
      • Gulácsi L
      • Boncz I
      • Drummond M
      Issues for countries considering introducing the “fourth hurdle”: the case of Hungary.
      ]. In all the studied countries, the assessment of the submitted evidence is performed by an independent committee. It is also suggested that the appraisal report and the final decision be fully justified, transparent, and publicly available. A procedure of appeal could also be considered and described. Finally, a reassessment of the decision must be a part of the HTA procedure because when a new medicine is marketed the information about its long-term benefit is limited, whereas the initial economic assessment has been performed mainly with data from phase 3 clinical studies. The collection of real-world evidence can be a requirement in some cases, and its reassessment over a predetermined period (e.g., 3 or 5 years later) should be considered.
      Another crucial point is the transferability of HTA results across countries, specifically whether the data on the effectiveness and costs can be transferred from one country to another. The differences that may exist in epidemiological data, such as disease incidence or mortality, prove to be a barrier in the transfer of results between countries [
      • Gulacsi L
      • Brodszky V
      • Pentek M
      • et al.
      History of health technology assessment in Hungary.
      ,
      • Gulácsi L
      • Rotar AM
      • Niewada M
      • et al.
      Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
      ,
      • Kaló Z
      • Landa K
      • Doležal T
      • Vokó Z
      Transferability of National Institute for Health and Clinical Excellence recommendations for pharmaceutical therapies in oncology to Central-Eastern European countries.
      ]. In general, clinical data may be transferred, particularly within Europe, because there are no significant genetic or other differences between European nations [
      • Kaló Z
      • Landa K
      • Doležal T
      • Vokó Z
      Transferability of National Institute for Health and Clinical Excellence recommendations for pharmaceutical therapies in oncology to Central-Eastern European countries.
      ,
      • Gulacsi L
      • Rencz F
      • Pentek M
      • et al.
      Transferability of results of cost utility analyses for biologicals in inflammatory conditions for Central and Eastern European countries.
      ]; the target population, however, should be concretely defined. On the contrary, cost-effectiveness data are not easily transferred from one country to another because of factors such as variations in prices of medicines and health services, the organization of health care, and the availability of financial resources. The question of other countries’ data transferability may be partly answered by using appropriate testing tools (extensive checklist of critical and noncritical factors) to support an objective conclusion [
      • Goeree R
      • Diaby V
      Introduction to health economics and decision-making: Is economics relevant for the frontline clinician?.
      ]. Taking into account available resources, each country should invest in capacity building to proceed to a stage of HTA development that entails using local data as input for decision making and basing the final conclusions on national priorities and values [
      • Danko D.
      Health technology assessment in middle-income countries: recommendations for a balanced assessment system.
      ,
      • Dankó D
      • Petrova G
      Health technology assessment in the Balkans: opportunities for a balanced drug assessment system.
      ]. This would fulfill a basic goal of HTA implementation—early access, within available resources, for patients to new, safe, and effective technologies [
      • Velasco Garrido M
      • Gerhardus A
      • Rottingen J-A
      • Busse R
      Developing health technology assessment to address health care system needs.
      ]. The next step is that of engaging the main stakeholders, especially the patients in the decision-making process [
      • Abelson J
      • Giacomini M
      • Lehoux P
      • Gauvin F-P
      Bringing “the public” into health technology assessment and coverage policy decisions: from principles to practice.
      ,
      • Gauvin F-P
      • Abelson J
      • Giacomini M
      • et al.
      Moving cautiously: public involvement and the health technology assessment community.
      ,
      • Gauvin F-P
      • Abelson J
      • Giacomini M
      • et al.
      “It all depends”: conceptualizing public involvement in the context of health technology assessment agencies.
      ].
      While introducing the preliminary steps of the HTA evaluation, the use of calibrated criteria should be considered [
      • Berntgen M
      • Gourvil A
      • Pavlovic M
      • et al.
      Improving the contribution of regulatory assessment reports to health technology assessments—a collaboration between the European Medicines Agency and the European network for Health Technology Assessment.
      ,
      • Moharra M
      • Espallargues M
      • Kubesch N
      • et al.
      Systems to support health technology assessment (HTA) in member states of the European union with limited institutionalization of HTA.
      ]. The result of the assessment by other countries should not be the sole criterion for the decision of reimbursing a new pharmaceutical product but could be part of a wider scoring framework, as in the case of Romania and Bulgaria. Finally, opportunities for international cooperation should be exploited. The European Network for Health Technology Assessment aims at increasing the use of the HTA process in decision making among European countries, strengthening the relationship of HTA and health policymakers in the EU and its member states, and supporting countries with limited experience in HTA by providing tools for increasing transferability and decreasing duplication of work [
      • Kristensen FB
      • Makela M
      • Neikter SA
      • et al.
      European network for health technology assessment, EUnetHTA: planning, development, and implementation of a sustainable European network for health technology assessment.
      ,
      • Kristensen FB
      • Lampe K
      • Chase DL
      • et al.
      Practical tools and methods for health technology assessment in Europe: structures, methodologies, and tools developed by the European Network for Health Technology Assessment, EUnetHTA.
      ]. In the framework of the European Network for Health Technology Assessment Joint Actions, the possibility of cooperation between HTA agencies was studied by implementing joint HTA evaluations [
      • Huic M
      • Nachtnebel A
      • Zechmeister I
      • et al.
      Collaboration in health technology assessment (EUnetHTA joint action, 2010–2012): four case studies.
      ].
      The present analysis has certain limitations. The main search was specifically done in the PubMed database and in some cases full access to some articles was not possible. An effort was made to validate and complement the information retrieved through searches in HTA agencies’ Web sites and reports published by the European Observatory on Health Systems and Policies. Nevertheless, the data collected and the comparative analysis included in the review can be used as a source of information on the basic aspects that countries with different degrees of maturity have in common and thus are important for the initiation of a successful HTA process.

      Conclusions

      Among the selected countries, there are various implementation modes of HTA, leading to the conclusion that there is no specific model for the development and implementation of an HTA process. Nevertheless, it is estimated that HTA will remain the main tool for the evaluation of new medicines for P&R decisions. Countries that are at the dawn of the implementation of an HTA process should take advantage of the experience of other countries.
      Source of financial support: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

      REFERENCES

        • Banta D
        What is technology assessment?.
        Int J Technol Assess Health Care. 2009; 25: 7-9
        • Sorenson C
        • Chalkidou K
        Reflections on the evolution of health technology assessment in Europe.
        Health Econ Policy Law. 2012; 7: 25-45
        • Allen N
        • Pichler F
        • Wang T
        • et al.
        Development of archetypes for non-ranking classification and comparison of European National Health Technology Assessment systems.
        Health Policy. 2013; 113: 305-312
        • Stafinski T
        • Menon D
        • Davis C
        • McCabe C
        Role of centralized review processes for making reimbursement decisions on new health technologies in Europe.
        Clin Outcomes Res. 2011; 3: 117-186
        • Hutton J
        • McGrath C
        • Frybourg J-M
        • et al.
        Framework for describing and classifying decision-making systems using technology assessment to determine the reimbursement of health technologies (fourth hurdle systems).
        Int J Technol Assess Health Care. 2006; 22: 10-18
        • Franken M
        • le Polain M
        • Cleemput I
        • Koopmanschap M
        Similarities and differences between five European drug reimbursement systems.
        Int J Technol Assess Health Care. 2012; 28: 349-357
      1. Bidgood E, Clarke E. Healthcare Systems: France. 2013. Based on the 2001 Civitas Report by David Green and Benedict Irvine Updated by Emily Clarke (2012) and Elliot Bidgood (January 2013).

      2. Boyle S. United Kingdom (England): Health System Review. Health Systems in Transition Eur. Obversatory Heal Syst Policies 2011;Vol 13.

        • Jonsson E
        History of health technology assessment in Sweden.
        Int J Technol Assess Health Care. 2009; 25: 42-52
        • Busse R
        • Blumel M
        Germany: health system review.
        Health Syst Transit. 2014; 16 (xxi): 1-296
        • Perleth M
        • Gibis B
        • Gohlen B
        A short history of health technology assessment in Germany.
        Int J Technol Assess Health Care. 2009; 25: 112-119
        • Weill C
        • Banta D
        Development of health technology assessment in France.
        Int J Technol Assess Health Care. 2009; 25: 108-111
        • Gloekler S
        • Traupe T
        • Stoller M
        • et al.
        The effect of heart rate reduction by ivabradine on collateral function in patients with chronic stable coronary artery disease.
        Heart. 2014; 100: 160-166
        • Ivandic V
        Requirements for benefit assessment in Germany and England—overview and comparison.
        Health Econ Rev. 2014; 4: 12
      3. National Institute for Health and Care Excellence. Technology appraisal and highly specialised technologies programmes: procedure for varying the funding requirement to take account of net budget impact. Available from: https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-technology-appraisals/TA-HST-procedure-varying-the-funding-direction.pdf. [Accessed December 18, 2017].

        • Drummond M
        • Sorenson C
        Nasty or nice? A perspective on the use of health technology.
        Value Health. 2009; 12: 8-13
        • Luce BR
        • Drummond M
        • Jonsson B
        • et al.
        EBM, HTA, and CER: clearing the confusion.
        Milbank Q. 2010; 88: 256-276
        • Drummond MF
        • Schwartz JS
        • Jonsson B
        • et al.
        Key principles for the improved conduct of health technology assessments for resource allocation decisions.
        Int J Technol Assess Health Care. 2008; 24: 244-248
      4. Sorenson C, Drummond M, Kanavos P. Ensuring Value for Money in Health Care: The Role of Health Technology Assessment in the European Union. 2008, Observatory Studies Series No 11. European Observatory on Health Systems and Policies.

        • Corabian P
        • Hailey D
        • Harstall C
        • et al.
        Mentoring a developing health technology assessment initiative in Romania: an example for countries with limited experience of assessing health technology.
        Int J Technol Assess Health Care. 2005; 21: 522-525
        • Gaal P
        • Szigeti S
        • Csere M
        • et al.
        Hungary health system review.
        Health Syst Transit. 2011; 13: 1-266
        • Gaal P
        • Szigeti S
        • Panteli D
        • et al.
        Major challenges ahead for Hungarian healthcare.
        BMJ. 2011; 343: d7657
        • Kolasa K
        • Kalo Z
        • Zah V
        • Dolezal T
        Role of health technology assessment in the process of implementation of the EU Transparency Directive: relevant experience from Central Eastern European countries.
        Expert Rev Pharmacoecon Outcomes Res. 2012; 12: 283-287
        • Vladescu C
        • Olsavszky V
        • Scintee G
        Health Systems in Transition: Romania.
        European Observatory on Health Systems and Policies, Copenhagen, Denmark2008: 10
        • Dimova A
        • Rohova M
        • Moutafova E
        • et al.
        Bulgaria health system review.
        Health Syst Transit. 2012; 14: 1-186
        • Gulacsi L
        • David T
        • Dozsa C
        Pricing and reimbursement of drugs and medical devices in Hungary.
        Eur J Health Econ. 2002; 3: 271-278
      5. Sima M. Capacity building in pharmacoeconomics and HTA in CEE: opportunities in education and CEE context HTA implementation in CEE countries—opportunities HTA implementation in CEE countries—challenges. 2014:1–14. ISPOR 17th Annual European Congress 8-12 November 2014 Amsterdam RAI Amsterdam.

        • Kolasa K
        • Schubert S
        • Manca A
        • Hermanowski T
        A review of health technology assessment (HTA) recommendations for drug therapies issued between 2007 and 2009 and their impact on policymaking processes in Poland.
        Health Policy. 2011; 102: 145-151
        • Kolasa K
        • Dziomdziora M
        • Fajutrao L
        What aspects of the health technology assessment process recommended by international health technology assessment agencies received the most attention in Poland in 2008?.
        Int J Technol Assess Health Care. 2011; 27: 84-94
        • Ozieranski P
        • McKee M
        • King L
        The politics of health technology assessment in Poland.
        Health Policy. 2012; 108: 178-193
        • Iskrov GG
        • Raycheva RD
        • Stefanov RS
        Insight into reimbursement decision-making criteria in Bulgaria: implications for orphan drugs.
        Folia Med (Plovdiv). 2013; 55: 80-86
        • Makady A
        • Ham RT
        • de Boer A
        • et al.
        Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies.
        Value Health. 2017; 20: 520-532
        • Angelis A
        • Lange A
        • Kanavos P
        Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries.
        Eur J Health Econ. 2018; 19: 123-152
        • Barron AJG
        • Klinger C
        • Shah SMB
        • Wright JSF
        A regulatory governance perspective on health technology assessment (HTA) in France: the contextual mediation of common functional pressures.
        Health Policy. 2015; 119: 137-146
        • Greiner W
        • von der Schulenburg J-MG
        HTA in Germany: very special and specific.
        Eur J Health Econ. 2010; 11: 1-3
        • Rochaix L
        • Xerri B
        National Authority for Health: France.
        Issue Brief (Commonw Fund). 2009; 58: 1-9
        • Shah SMB
        • Barron A
        • Klinger C
        • Wright JSF
        A regulatory governance perspective on health technology assessment (HTA) in Sweden.
        Health Policy. 2014; 116: 27-36
        • Gagnon M-P
        • Gagnon J
        • St-Pierre M
        • et al.
        Involving patients in HTA activities at local level: a study protocol based on the collaboration between researchers and knowledge users.
        BMC Health Serv Res. 2012; 12: 14
        • Kolasa K
        • Wasiak R
        Health technology assessment in Poland and Scotland: comparison of process and decisions.
        Int J Technol Assess Health Care. 2012; 28: 70-76
        • Iskrov G
        • Miteva-Katrandzhieva T
        • Stefanov R
        Challenges to orphan drugs access in Eastern Europe: the case of Bulgaria.
        Health Policy. 2012; 108: 10-18
        • Chevreul K
        • Durand-Zaleski I
        • Bahrami SB
        • et al.
        France: health system review.
        Health Syst Transit. 2010; 12 (xxi–xxii): 1-291
        • Anell A
        • Persson U
        Reimbursement and clinical guidance for pharmaceuticals in Sweden: Do health-economic evaluations support decision making?.
        Eur J Health Econ. 2005; 6: 274-279
        • Walley T.
        Health technology assessment in England: assessment and appraisal.
        Med J Aust. 2007; 187: 283-285
      6. Espin J. Rovira J. Analysis of differences and commonalities in pricing and reimbursement systems in Europe. DG Enterprise and Industry of the European Commission. Final report June 2007, available from https://ec.europa.eu/docsroom/documents/7605?locale=en, Accessed January 17, 2016.

        • Ferrario A
        • Kanavos P
        Dealing with uncertainty and high prices of new medicines: a comparative analysis of the use of managed entry agreements in Belgium, England, the Netherlands and Sweden.
        Soc Sci Med. 2015; 124: 39-47
        • Department of Health; Association of the British Pharmaceutical Industry
        The Pharmaceutical Price Regulation Scheme.
        Department of Health, London2014
        • Gulacsi L
        • Brodszky V
        • Pentek M
        • et al.
        History of health technology assessment in Hungary.
        Int J Technol Assess Health Care. 2009; 25: 120-126
        • Gulácsi L
        • Rotar AM
        • Niewada M
        • et al.
        Health technology assessment in Poland, the Czech Republic, Hungary, Romania and Bulgaria.
        Eur J Health Econ. 2014; 15: S13-S25
        • Kolasa K
        • Kalo Z
        • Hornby E
        Pricing and reimbursement frameworks in Central Eastern Europe: a decision tool to support choices.
        Expert Rev Pharmacoecon Outcomes Res. 2015; 15: 145-155
        • Panteli D
        • Sagan A
        • Mckee M
        • et al.
        Poland: health system review.
        Health Syst Transit. 2011; 13: 1-193
        • Radu C-P
        • Chiriac ND
        • Pravat AM
        The Development of the Romanian Scorecard HTA System.
        Value Heal Reg Issues. 2016; (10:41-7. https://doi.org/:10.1016/J.VHRI.2016.07.006)
        • Aleksandrova S
        The Bulgarian Health Care Reform and Health Act 2004.
        Med Law. 2007; 26: 1-14
        • Panteli D
        • Arickx F
        • Cleemput I
        • et al.
        Pharmaceutical regulation in 15 European countries review.
        Health Syst Transit. 2016; 18: 1-122
        • Schwarzer R
        • Siebert U
        Methods, procedures, and contextual characteristics of health technology assessment and health policy decision making: comparison of health technology assessment agencies in Germany, United Kingdom, France, and Sweden.
        Int J Technol Assess Health Care. 2009; 25: 305-314
        • Ciani O
        • Jommi C
        The role of health technology assessment bodies in shaping drug development.
        Drug Des Devel Ther. 2014; 8: 2273-2281
        • Henshall C
        • Schuller T
        • Mardhani-Bayne L
        Using health technology assessment to support optimal use of technologies in current practice: the challenge of “disinvestment.
        Int J Technol Assess Health Care. 2012; 28: 203-210
        • Battista RN
        • Hodge MJ
        The “natural history” of health technology assessment.
        Int J Technol Assess Health Care. 2009; 25: 281-284
        • Gulácsi L
        • Péntek M
        HTA in Central and Eastern European countries; the 2001: a space odyssey and efficiency gain.
        Eur J Health Econ. 2014; 15: 675-680
        • Inotai A
        • Pékli M
        • Jóna G
        • et al.
        Attempt to increase the transparency of fourth hurdle implementation in Central-Eastern European middle income countries: publication of the critical appraisal methodology.
        BMC Health Serv Res. 2012; 12: 332
        • Kaló Z
        • Landa K
        • Doležal T
        • Vokó Z
        Transferability of National Institute for Health and Clinical Excellence recommendations for pharmaceutical therapies in oncology to Central-Eastern European countries.
        Eur J Cancer Care (Engl). 2012; 21: 442-449
        • Gulácsi L.
        The time for cost-effectiveness in the new European Union member states: the development and role of health economics and technology assessment in the mirror of the Hungarian experience.
        Eur J Health Econ. 2007; 8: 83-88
        • Gulacsi L
        • Rencz F
        • Pentek M
        • et al.
        Transferability of results of cost utility analyses for biologicals in inflammatory conditions for Central and Eastern European countries.
        Eur J Health Econ. 2014; 15: S27-S34
        • Gulácsi L
        • Boncz I
        • Drummond M
        Issues for countries considering introducing the “fourth hurdle”: the case of Hungary.
        Int J Technol Assess Health Care. 2004; 20: 337-341
        • Goeree R
        • Diaby V
        Introduction to health economics and decision-making: Is economics relevant for the frontline clinician?.
        Best Pract Res Clin Gastroenterol. 2013; 27: 831-844
        • Danko D.
        Health technology assessment in middle-income countries: recommendations for a balanced assessment system.
        J Mark Access Health Policy. 2014; 1: 1-10
        • Dankó D
        • Petrova G
        Health technology assessment in the Balkans: opportunities for a balanced drug assessment system.
        Biotechnol Biotechnol Equip. 2014; 28: 1181-1189
        • Velasco Garrido M
        • Gerhardus A
        • Rottingen J-A
        • Busse R
        Developing health technology assessment to address health care system needs.
        Health Policy. 2010; 94: 196-202
        • Abelson J
        • Giacomini M
        • Lehoux P
        • Gauvin F-P
        Bringing “the public” into health technology assessment and coverage policy decisions: from principles to practice.
        Health Policy. 2007; 82: 37-50
        • Gauvin F-P
        • Abelson J
        • Giacomini M
        • et al.
        Moving cautiously: public involvement and the health technology assessment community.
        Int J Technol Assess Health Care. 2011; 27: 43-49
        • Gauvin F-P
        • Abelson J
        • Giacomini M
        • et al.
        “It all depends”: conceptualizing public involvement in the context of health technology assessment agencies.
        Soc Sci Med. 2010; 70: 1518-1526
        • Berntgen M
        • Gourvil A
        • Pavlovic M
        • et al.
        Improving the contribution of regulatory assessment reports to health technology assessments—a collaboration between the European Medicines Agency and the European network for Health Technology Assessment.
        Value Health. 2014; 17: 634-641
        • Moharra M
        • Espallargues M
        • Kubesch N
        • et al.
        Systems to support health technology assessment (HTA) in member states of the European union with limited institutionalization of HTA.
        Int J Technol Assess Health Care. 2009; 25: 75-83
        • Kristensen FB
        • Makela M
        • Neikter SA
        • et al.
        European network for health technology assessment, EUnetHTA: planning, development, and implementation of a sustainable European network for health technology assessment.
        Int J Technol Assess Health Care. 2009; 25: 107-116
        • Kristensen FB
        • Lampe K
        • Chase DL
        • et al.
        Practical tools and methods for health technology assessment in Europe: structures, methodologies, and tools developed by the European Network for Health Technology Assessment, EUnetHTA.
        Int J Technol Assess Health Care. 2009; 25: 1-8
        • Huic M
        • Nachtnebel A
        • Zechmeister I
        • et al.
        Collaboration in health technology assessment (EUnetHTA joint action, 2010–2012): four case studies.
        Int J Technol Assess Health Care. 2013; 29: 323-330
        • Nuijten MJC
        • Szende A
        • Kosa J
        • et al.
        Health care reform in six Central European countries. A focus on health economic requirements in the drug pricing and reimbursement processes.
        Eur J Health Econom. 2003; 4: 286-291
        • Bridges JFP
        • Cohen JP
        • Grist PG
        • Mühlbacher AC
        International experience with comparative effectiveness research: case studies from England/Wales and Germany.
        Adv Health Econ Health Serv Res. 2010; 22: 29-50
      7. (Haute Autorité de Santé. Assessment of medicinal products [WWW Document], n.d.) (Accessed September 25)
        • Anell A
        • Glenngård AH
        • Merkur S
        Health Syst. Transit. 2012; 14: 187
        • Nicod E
        • Kanavos P
        Commonalities and differences in HTA outcomes: A comparative analysis of five countries and implications for coverage decisions.
        Health Policy (New York). 2012; 108: 167-177