Bachelor Thesis

The current state of cooperation in the area of selected

emergency services in the Euregio Meuse-Rhine

- Assessment of the practical and theoretical suitability of evaluation tools -

Tobias Becking

Student ID I6080319

UM Supervisors: Prof. Dr. Thomas Krafft

External Supervisor: Marian Ramakers

Bachelor of European Public Health

Faculty of Health, Medicine and Life science

Maastricht University

29th June 2016

2

Acknowledgement:

First, I would like to express my sincere appreciation to my external supervisor

Marian Ramakers and to my two university supervisors Prof. Dr. Thomas Krafft and

Eva Pilot for their patience, motivation and knowledge. Their guidance and support

helped me during the whole research and writing of my bachelor thesis.

Second, besides my supervisors, I would like to thank my two colleagues in the

EMRIC bureau, Kim Worseling and Nina Albrecht for their insightful comments and

mental support during my research.

Last but not least, I would like to thank all experts who participated in my study by

providing me with rich data during the interviews.

3

Table of Contents List of Figures: .......................................................................................................................... 4

List of Tables: ............................................................................................................................ 4

Abstract: .................................................................................................................................... 5

1. Introduction ................................................................................................... 6 1.2 Cross-border Emergency care in the Euregio Maas-Rhein ................................................. 7

1.3 Brief description of existing arrangements .......................................................................... 8

1.4 Reasons for research, purpose and research question .......................................................... 9

1.5 Research objective and research questions ........................................................................ 10

2. Theoretical Concepts ....................................................................................... 10 2.1 Theory ............................................................................................................................... 11

2.2 Conceptual Model ............................................................................................................. 12

3. Research Methods .......................................................................................... 13 3.1 Research design and methodology .................................................................................... 13

3.2 Data collection ................................................................................................................... 14

3.2.1 Literature review ........................................................................................................ 14 3.2.2 Analysis of evaluation tools ....................................................................................... 14 3.2.3 Expert interviews ........................................................................................................ 15

3.3 Methods for analysis.......................................................................................................... 15

4.0 Results .......................................................................................................... 16 4.1 Existing Cooperation Arrangements ................................................................................. 16

4.1.1 Information exchange................................................................................................. 17 4.1.2 EUMED ...................................................................................................................... 18 4.1.3 EMRIC ....................................................................................................................... 20

4.2 Relevant Stakeholders ....................................................................................................... 21

4.3 Evaluation Design ............................................................................................................. 22

4.3.1 Requirements of an Evaluation Tool in Theory ......................................................... 23 4.3.2. Requirements of an Evaluation Tool in Practice ...................................................... 26 4.3.3 Evaluation Process .................................................................................................... 26

5. Discussion ....................................................................................................... 36 5.1 The Requirements/Standards ............................................................................................. 37

5.2 The Evaluation Tool .......................................................................................................... 39

5.4 Limitations ......................................................................................................................... 40

6. Conclusion and Recommendations .................................................................. 41

References .......................................................................................................... 43

Appendix ............................................................................................................ 47

4

List of Figures: Figure 1: Administrative organization of the EMR(Euregio Maas-Rhein, 2015) ......... 7

Figure 2: Steps in the CDC Framework for Evaluation in Public Health (Centers for

Disease Control and Prevention, 2014a) .............................................................. 12

Figure 3: Hierachic distribution of informaion in the event of a disaster (EMRIC

Lenkungsgruppe Euregio Maas-Rhein in Crisismanagement, 2015) (blue box:

Information exchange in EMR) ........................................................................... 18

Figure 5: Example of a Logic Model for the Emric bureau in the EMR ..................... 27

Figure 6: Illustration where in the policy process implmentation evaluation is focused

(Centers for Disease Control and Prevention, 2014b) ......................................... 28

Figure 7: Amount of Control Exercised/Determines Places on Interview Continuum

(Harrell & Bradley, 2009) .................................................................................... 29

List of Tables: Table 1: Eumed Alarm keywords and scope (EUMED, October 2015)...................... 18

Table 2: Emric alarm keywords and scope (EMRIC, 2015b) ...................................... 20

Table 3: Overview of advantages and disadvantages of evaluation tools ................... 34

5

Abstract: Background: In border regions, a disaster might have an impact on a neighbouring

country and foreign emergency services can often be at the scene quicker. Therefore,

there is a need for cross-border emergency collaboration. The collaboration is shaped

by bilateral agreements between countries, which encourage the operational services

to reach their own arrangements. However, there is a lack of research regarding the

practical implementation. Therefore, there is a need to gather particular knowledge

about the current state of implementation of such cooperation arrangements.

Objective: This study aims to first analyse and describe the existing cooperation

arrangements in the EMR in the field of fire services, CBRN, (Emric) acute medical

care (Eumed) and information exchange. Second, this study aims to select an

appropriate evaluation tool, which could be applied to assess the current state of

implementation of cooperation arrangements in practice.

Method: A multi-method approach is used to answer the research questions. First, a

non-systematic literature review was conducted to determine the current state of

cooperation arrangements and available evaluation tools. Second, a qualitative

assessment of available evaluation tools was conducted. Data was collected by

reviewing databases, and internal documents. Furthermore, experts were interviewed

by using semi-structured interviews.

Results and Discussion: Three major documents (Emric, Eumed, Information

exchange) define standards for cross-border emergency collaboration. With regard to

the evaluation tools, there are theoretical and practical requirements for a successful

evaluation. In theory it is important to take standards such as utility, feasibility,

property and accuracy into account. During the interviews, stakeholders mentioned

requirements regarding time, language barrier and the fact that an external agent

would be preferred to decrease the risk of biased evaluation. Based on these practical

and theoretical criteria, the advantages and disadvantages of the selected evaluation

tools were compared to get an overview which tool is the most appropriate. Practical

needs comply with theoretical requirements and the literature provides further support

for successful evaluation. As a result, a mixed method approach, using a combination

of tools is required, in order to determine the gap between the cooperation

arrangements and the implementation in practice.

6

1. Introduction In the European Union (EU), border regions are often economically disadvantaged

because of their differences in population density, socio economic status (SES) and

their economic characteristics (Brand, Hollederer, Wolf, & Brand, 2008). According

to Brand et al. (2008), cross-border cooperation between countries is used to reduce

the burden of these disadvantages. Therefore, already before the development of the

EU there was a need for cross-border collaboration in order to improve the safety

situation for the population in these areas (Appendix 1). The EMR is an area of

approximately 10.000 km2 with a population of 4 million inhabitants along the border

triangle of Belgium, the Netherlands and Germany (EMRIC Lenkungsgruppe Euregio

Maas-Rhein in Crisismanagement, 2012). The EMR geographically includes the

following regions: Belgium with the province of Limburg, Liege and the German

speaking community of Belgium, in Germany the region of Aachen and in the

Netherlands the southern part of Limburg (Ramakers, Jabakhanji, & Thönis, 2009).

The territory is characterised by a lot of industry, a very high volume of traffic as well

as air-traffic. In addition, tourism plays a crucial role in the area. Additional,

peculiarities of the EMR are the 3 different languages (German, Dutch and French)

and the 12 different government levels (EMRIC Lenkungsgruppe Euregio Maas-

Rhein in Crisismanagement, 2012).

In the Euregio Meuse-Rhein (EMR) collaboration regarding emergency healthcare

started approximately 40 years ago with the working group “Öffentliche Sicherheit

und Katastrophenschutz” (public safety and disaster control). This was the foundation

for cross-border cooperation between the different organizations and authorities

responsible for emergency services in the EMR. Today the responsible partners for

emergency services in the EMRIC Program are: Städteregion Aachen, Stadt Aachen,

Bezirksregierung Köln, Kreis Heinsberg, Kreis Düren, Kreis Euskirchen, Province de

Liege, Provincie Limburg, Veiligheisregio Zuid-Limburg and GGD Zuid Limburg

(Ramakers et al., 2009).

7

Figure 1: Administrative organization of the EMR(Euregio Maas-Rhein, 2015)

1.2 Cross-border Emergency care in the Euregio Maas-Rhein

Despite the complex environment, collaboration, on-going training and joint exercises

improved cross-border emergency care in the EMR and led to more than 1000 cross-

border activities in 2015 (EMRIC Lenkungsgruppe Euregio Maas-Rhein in

Crisismanagement, 2015). According to Ramakers (2014), the EMR can be seen as a

European example of well-functioning cross-border collaboration.

Due to the fact that borders are so close together (Figure 1) in the EMR it is possible

that first, the accident/disaster might have an impact on a neighbouring country and

second, that foreign help is often closer and therefore quicker at the scene than

national services. Due to the short distances between the countries it is even possible

to increase the quality of emergency care, as countries can support each other by

pooling their resources. An example of this is helicopter Christoph Europa 1 located

in Würselen, Germany, which flies to Belgium and the Netherlands. Furthermore,

countries can request special materials for technical aid from a foreign fire

8

department. Consequently, fast and reliable exchange of data between the 7 dispatch

centres in the EMR is important in order to prevent negative consequences to the

population (Ramakers et al., 2009, p. 11).

1.3 Brief description of existing arrangements The scope of cross-border emergency care in the EMR includes large-scale disasters,

daily emergency situations, information exchange between dispatch centres, crisis

teams and the coordination of several disciplines from fire brigades and ambulance

services. Nonetheless, the focus of this study will be on large-scale disasters.

The basis of the collaboration is shaped by bilateral agreements that have been signed

by all three countries (Germany/Netherlands, Netherlands/Belgium,

Belgium/Germany).1 The purpose of these conventions is that the operational services

such as the dispatch centres or crisis teams reach their own arrangements regarding

the organisation and mutual support in the event of disasters (EMRIC, 2015b)

In addition to the bilateral agreement, all three countries are working with the same

definition of “disaster” in order to prevent misunderstandings. In an international

context where organisations or operational services are working together, it is

important to agree on common definitions regarding certain terminology or

deployment keywords to improve cross-border collaboration (EMRIC, 2015b).

The basis for successful cross-border emergency healthcare is the exchange of

knowledge and the implementation of cooperation arrangements. In order to achieve

these goals, the EMRIC bureau (Programme Office) coordinates regular meetings

with the steering group consisting of responsible directors of each organization

dealing with safety issues (Ramakers et al., 2009). Under the steering group there are

multiple focus groups, which are specialized in either fire service and CBRN

(Chemical, biological, radiological, nuclear) (Focus Group BUKS2), acute medical

care (Focus Group Eumed) or information exchange (Task Force Communication and

1 - Agreement between the Netherlands and Belgium about mutual assistance in large-scale disaster settings from 14. November 1984 - Agreement between The Netherlands and Germany about mutual assistance in large-scale disaster settings from 7. June 1988 - Agreement between Belgium and Germany about mutual assistance in large-scale disaster settings from 6. November 1980

2 BUKS: Brand und Katastrophenschutz (Fire Fighting and Disaster Management)

9

Dispatch Centre). Within these groups, experts developed documents for information

exchange to manage cross-border collaboration. (Ramakers et al., 2009). This also

includes the use of standardized deployment keywords such as “EMRIC” and

“EUMED” in the case of a cross-border deployments. Furthermore, foreign assistance

needs to be picked up at a previously defined location, which should be close to the

border in order to prevent unnecessary phone or radio contact. Moreover, most of the

staff works within their own organizational structure and language to minimize

misunderstandings (EMRIC, 2015b) (EUMED, October 2015)

This only reflects a small part of the cooperation arrangements that are in place and

should give a brief overview on how complex the coordination of resources and

collaboration of different organizational structures is within the EMR. However, since

the implementation of the arrangements, no research has been done to identify the gap

between policy development and implementation in practice. Therefore, the overall

objective of this study is to fill this gap and to find a tool which tests the current state

of cooperation arrangements and how they are implemented in practice.

1.4 Reasons for research, purpose and research question

Due to the complexity of cross-border emergency care and the increasing

interdependency between the countries in the EMR, it is important to ensure that all

cooperation arrangements are fully implemented. There is a need to gather particular

knowledge about the current state of implementation of cooperation arrangements in

the fields of fire services, CBRN, acute medical care and information exchange, in

order to measure the gap between theoretical foundation and implementation in

practice. Additionally, an evaluation of the implementation of these arrangements is

necessary to assess the achievements of the EMR cross-border emergency care

initiative. With an evaluation tool, it would be possible to develop an on-going

surveillance system to enhance the work of the EMRIC bureau in the future. This

would be beneficial for all involved stakeholders at the national and international

level. In addition, it could be beneficial on a European dimension and furthermore

ensure quality of care for patients.

10

1.5 Research objective and research questions The objective of this study is first to analyse and describe the existing cooperation

arrangements. Second, it is important to select an appropriate evaluation tool, which

could be applied to assess the current state of implementation of cooperation

arrangements in practice in the future. Third, it is essential to asses the practical

suitability.Therefore, key stakeholders such as crisis team members, emergency

physicians and mayors are interviewes in order to present the evaluation tools that

could be identified in the literature. The overall goal of the interviews is to discuss

pros and cons of different tools regarding feasibility and utility.

For that purpose, it is necessary to further investigate the differences in the

organizational structure in the different countries in the EMR, in order to understand

the working procedures and to get a deeper insight in the work of EMRIC.

To reach these objectives, the following research questions will be used:

1. Which cooperation arrangements are in place in the field of fire service,

CBRN, acute medical care and information exchange in the EMR?

a. What are the most relevant stakeholders in these fields?

2. How can evaluation tools be used in order to better understand the current gap

between the cooperation agreement and their implementation in the fields of

CBRN, fire service, acute medical care, and information exchange in the

EMR?

a. What are the requirements of a tool to measure implementation of

cooperation arrangements?

b. Which tools can be identified in the literature to measure the state of

implementation and what are their advantages and disadvantages?

c. Which tool is appropriate in the current literature that fits these

requirements?

2. Theoretical Concepts According to the research objectives, it is crucial to determine an evaluation tool to

assess the gap between the cooperation arrangements and the implementation in

11

practice. Therefore, the Centres for Disease Control and Prevention (CDC) Evaluation

Framework for policy evaluation will be used.

2.1 Theory The Evaluation Framework developed by the CDC is using the following definition:

“A Policy is a law, regulation, procedure, administrative action, incentive or voluntary

practice of governments and other institutions” (Centers for Disease Control and

Prevention, 2014a, p. 1). According to the Centers for Disease Control and Prevention

(2014a), policies in general aim at improving a certain system by focusing on changes

on the system-level. This policy approach might lead to further improvement of the

health and safety of the population. There are different ways of analysing a set of

policies or a whole program. This study is going to use the “six-step CDC Framework

for Evaluation in Public Health” for the purpose of finding a tool to determine the

current state of the implementation of cooperation arrangements in the field of fire

services, CBRN, acute medical care and information exchange in the EMR Centers

for Disease Control and Prevention (2014a).

As a basis, the Framework uses a certain set of standards in order to conduct the

evaluation and to support choices during the process. The four categories of standards

are: “Utility” (who is the contracting authority for the evaluation and what is the

objective?), “Feasibility” (is it possible to conduct the research regarding time,

resources, participants?), “Propriety” (is the evaluation ethical and fair?) and

“Accuracy” (does the research meet the demand of stakeholder needs and is the

evaluation accurate?).

The first step in the Framework is to engage the stakeholders. This is very important

because cross-border collaboration has a multi stakeholder approach and includes

stakeholders from different levels (e.g paramedic, member crisis team, major). The

second step is the description of the policy being evaluated. Here it is important to

describe the purpose and the underlying logic of the policy precisely. The third step is

crucial and the focus of this research. During this step, the selection of the appropriate

evaluation design takes place and furthermore it is necessary to select/develop the

evaluation questions (Centers for Disease Control and Prevention, 2014b).

The subsequent steps are not part of this research, because it would exceed the scope

of the thesis. To gather credible evidence is a time intensive step due to the fact that

approximately 50.000 people work in the EMR. Therefore, this research focuses on

12

the first three steps to find an appropriate evaluation design together with the most

important key stakeholders (Centers for Disease Control and Prevention, 2014c).

To be more specific on how the CDC Framework will be applied in this specific case,

it must be said that every step can be seen as a guide to answer the research questions.

For instance, the fist two steps in the CDC concept (Engage Stakeholders, Describe

the Program) will guide the researcher to answer reseach question one. All the

existing arrangements will be reviewed and summarized in order to give an overview

of the current state. Afterwards, it is necessary to determine the most important

stakeholders regarding crisis management in the EMR, to identify interest

representatives for the interviews. To answer the second research question, the third

step (focus the evaluation design) will guide the researcher to find a tool and to adjust

it to the needs of the evaluation for the EMR.

2.2 Conceptual Model The overall aim of this study is to find a suitable evaluation design to describe or

determine the gap between policy development and implementation in practice, which

if necessary will be adjusted to this specific setting (step 3). The other steps will be

supportive in developing such an evaluation tool and moreover provide a theoretical

basis for this study. All steps included in the CDC Framework can be found in figure

2.

Figure 2: Steps in the CDC Framework for Evaluation in Public Health (Centers for Disease

Control and Prevention, 2014a)

13

3. Research Methods The following section will describe the research design and methodology.

Furthermore, it will explain how the researcher is going to collect the data and which

method will be used for the analysis of the data.

3.1 Research design and methodology

In order to answer the research questions, different research methods will be used

within this study. First, a non-systematic literature review will be conducted to

determine the current state of cooperation arrangements in the EMR. Furthermore,

legal documents and policies will be reviewed to get an overview of the different

organizational structures and working procedures of the different organizations in

each country.

According to Polit and Beck (2012), research starts with a question and then moves in

a reasonably linear sequence of steps to the end in order to obtain a answer. As this

approach fits in the research setting, the second step would be a qualitative

assessment (semi-structured interviews) of the evaluation tools, which could later be

used to identify the current state of the implementation of cooperation arrangements.

Due to the fact that this study follows a mixed method approach of different research

tools, the research during the literature review will be descriptive when reviewing the

current cooperation arrangements. While finding and adjusting an appropriate

evaluation tool the research is going to be exploratory and evaluative.

Because developing an own scale would exceed the scope of this thesis, it is crucial to

make use of existing tools. Thus, according to (Streiner, Norman, & Cairney, 2015),

one should search for literature to critically review and judge whether the scales found

are appropriate. Furthermore, one should check if there is enough evidence to support

the particular scale in terms of reliability, internal consistency and validity. Only if

there is no suitable tool available, an existing tool has to be adjusted. After

determining an appropriate evaluation tool, it is necessary to conduct the first

interviews with key stakeholders in order to determine feasibility and utility of the

identified tools and simultaneously strengthen the support for the evaluation.

14

3.2 Data collection In this paragraph methods for data collection are discussed separately for the literature

review, the analysis of the evaluation tools and for the expert interviews.

3.2.1 Literature review

The data is collected by different means. Therefore, the use of different databases and

sources is necessary to identify important documents available in the field of cross-

border emergency care. Data was extracted from current laws, policies and

regulations. To complement the findings, it is necessary to make use of internal

documents and scientific databases such as “PubMed, “Google Scholar” and “ the

Web of Science”. In order to get an overview of the current laws and policies, the

“EUR-LEX” database from the EU will be used. In addition, to identify earlier

research on the same topic, the ancestry approach will be used (Polit & Beck, 2012).

The following search terms will be included to search for scientific articles and

information. The “Boolean Method” is used as a search strategy.

- “cross border” AND “emergency” AND “cooperation” OR “cross border” AND

“emergency” AND “collaboration”

- “evaluation” AND “cross border projects” AND policy evaluation OR “assessment”

AND “cross border projects”

- cross border projects AND policy evaluation AND gab between policy and practice

These Keywords were combined with terms such as “Euregio Maas-Rhein”,

“Germany”, “The Netherlands”, “Belgium” and “Europe”. To specify the research it

is necessary to determine some inclusion and exclusion criteria. Examples are that the

literature should not be older than 10 years, only literature in English and German will

be reviewed and the abstract of a paper should contain one or more of the key search

terms identified.

3.2.2 Analysis of evaluation tools

During the third step of the CDC Evaluation Framework (“Focus the Evaluation

Design”), the researcher reviewed and assessed a variety of available evaluation tools

appropriate for the purpose of the research and moreover also suitable for the EMRIC

15

Project and the EMR. Therefore, it might be possible that available tools have to be

slightly adapted to the research setting.

3.2.3 Expert interviews

To answer the second research question, semi-structured interviews will be conducted

in order to gain more insight into the practical application of the evaluation tools. The

type of qualitative sampling used for these interviews is the so called purposive

sampling method (judgemental sampling). According to Polit and Beck (2012), this

type of qualitative sampling is suitable to pre-test newly developed tools effectively

and it makes use of the researchers knowledge to identify sample members.

Furthermore, researchers often use this type of sampling when only a sample of

experts from a specific field is needed. In order to develop a keen understanding of

the current state of implementation of cooperation arrangements, the interviews

carried out will be semi-structured. These interviews will be recorded after the

interviewee has given the consent. The general informed consent to conduct the

research including the interviews was already given by the steering group of the

EMRIC office. According to Louise Barriball and While (1994), one major advantage

of using a semi-structured interview is that the whole interviews structure can be more

flexible. The identification of the most relevant stakeholders will take place at the

beginning of the research period together with the program manager and expert in

cross-border emergency collaboration Marian Ramakers. In general, the plan would

be to approach the sample members (approximately 20) via e-mail in order to arrange

an appointment for the beginning of May 2016. After the interviews have been

conducted and recorded, the researcher is going to transcribe and analyse the results.

3.3 Methods for analysis For the analysis, a deductive approach will be used. This is a suitable approach as the

focus will be on what other researchers have done so far, to evaluate the current state

of implementation of cooperation arrangements (Blackstone, 2016). Based on

reviewing and assessing their work, the researcher is going to find an evaluation tool

which is appropriate for the research setting and the purpose of the thesis. In addition,

the findings from the literature review will be clustered and systematically assessed.

To gain more insight into the current cooperation arrangements, a policy analysis will

16

be conducted. In general, there are two different approaches of policy analysis. This

study will use the analysis “of” policies because this approach is more descriptive and

the goal is to understand the policies. Furthermore, with respect to the five-step policy

cycle this analysis will focus on the last step “evaluation”. This step is a vital part in

the whole policy cycle and is crucial to provide the policy maker with necessary

feedback in order to improve their own work and give them a basis for their reflection

process(Barkenbus, 1998).

4.0 Results

The following section describes the findings from this study. Here, the focus is on the

first three steps of the CDC Framework for Evaluation in Public Health (Engage

Stakeholder, Describe the Program and Focus the evaluation design). This Section is

structured according to the research questions as described in section 1.5

4.1 Existing Cooperation Arrangements For the cooperation arrangements developed by the EMRIC bureau together with the

regional authorities responsible for emergency services, the parties agreed on five

starting points:

1) Simplicity in a disaster setting and the use of structured cooperation concepts

is the key to successful collaboration

2) Tasks should be carried out close to the daily practice of the staff

3) The language barrier should not form a disadvantage for care of casualties

4) Radio communication takes place within the scope of available resources of

one organisation

5) There is no legal obligation for a member state to offer cross-border assistance

These starting points can be seen as a basis for further arrangements. Based on these

five points the involved stakeholders agreed on the following fundamental

arrangements. First, the dispatch centres use a standardised alerting keyword in each

country to make the deployed personnel aware of the fact that they have to deal with a

cross-border deployment. Second, the foreign providers of aid are always picked up at

the gathering point, which are defined in advance. If possible, the gathering point

17

should be close to the border in order prevent unnecessary radio contact between

emergency vehicles. Third, emergency personnel has to work with the same

procedures as they do in their country of affiliation. Furthermore, in a large scale

disaster settings they work in their own work section, which means that a crew is

responsible for a certain task or part of the incidence (EMRIC, 2015b).

In the past, the EMRIC-bureau developed three major documents that define how

cross-border collaboration should work. The first one is the EMRIC document, which

provides a framework for cross-border collaboration regarding the fire departments in

the EMR. The second one is the EUMED document, which deals with cross-border

acute medical care in the EMR. The third document describes the arrangements

regarding information exchange between the partners in the EMR.

The following sections will describe these three arrangements in more detail, to get an

overview about the arrangements in the different disciplines.

4.1.1 Information exchange

The overall objective of the information exchange plan is to define a joint

arrangement under the consideration of the national laws 3 in each of the three

Member States Germany, Belgium and The Netherlands. The strategic goals of the

arrangement are the exchange of relevant information needed for the prevention of

incidents or disasters. Furthermore, this agreement should give the basis for

information exchange regarding incidents that might have a cross-border implication

for neighbouring countries. With respect to the communication, there are some crucial

success factors. It is important that each country has some basic knowledge about the

system of the neighbouring country and that involved stakeholders understand the

received information (language barrier). As a disaster setting often requires a

multidisciplinary approach, it is also highly important that the communication takes

place between the correct officials (EMRIC, 2015b).

As shown in figure 3, there are differences in terms of crisis communication in the

event of a disaster. Every country has different organisational structures to pass on

information and to successful manage the coordination of required emergency service.

3 - BHKG (Brandschutz, die Hilfeleistung und den Katastrophenschutz) (Die Landesregierung Nordrhein-Westfalen, 2015) - Wet Veiligheidsregio's (Ministerie van Veiligheid en Justitie, 2013) - Civile Bescherming and Koninklijk besluit

18

Figure 3: Hierachic distribution of informaion in the event of a disaster (EMRIC Lenkungsgruppe Euregio Maas-Rhein in Crisismanagement, 2015) (blue box: Information

exchange in EMR)

4.1.2 EUMED

The Eumed document defines goals for acute medical care during large-scale

emergencies. The objective of this document is that emergency vehicles (ambulances)

and the deployed personnel (paramedics) reach the quality of supply as soon as

possible, as it would be the case in individual medical care (EUMED, October 2015).

To ensure that all countries have the same understanding in the case of a cross-border

deployment, the dispatch centres use a standardized alarm keyword “EUMED”. There

are three different stages with a different scope and time frame for the emergency aid,

which are depicted in table 1.

EUMED 1:

Transport aid (small) /

Emergency aid

Time to provision < 30

min

EUMED 2:

Transport aid (large)

Time to provision >30 min

< 60 min

EUMED 3:

Treatment Aid

Time to provision >60 min

Table 1: Eumed Alarm keywords and scope (EUMED, October 2015)

The EUMED 1 alarm keyword is used to request two to three vehicles for emergency

aid. Those vehicles are immediately available and they are responsible for the first

treatment at the deployment location and/or the transport of maximum three patients.

19

Within Eumed 1 operating resources are not picked up at a gathering point, but they

immediately drive to the accident scene.

The EUMED 2 alarm keyword is used to request not only resources that are

immediately available, but also requests additional resources from the reserve.

Thereby, it is possible to double the capacity compared to EUMED 1. However, this

also increases the organisational time and effort and the time to provision can be up to

60 minutes. The operating unit will be picked up at a gathering point and is suitable to

transport up to 10 patients (Germany: PTZ-104)

The EUMED 3 alarm keyword is used in order to request comprehensive rescue units,

which are needed during large-scale emergencies. The deployed personnel builds up

an own independent treatment unit to treat approximately 50 patients. Due to the

organizational effort, the time to provision will be up to 60 minutes and can increase

to 4 h to become fully operational (Germany: BHP-B 505) (EUMED, October 2015).

Only the dispatch centre that is responsible for the coordination of a disaster is

allowed to request cross-border assistance. To save time, the first request can be done

via phone and should follow the METHANE-Method:

1) Major incident

2) Exact location

3) Type of incident

4) Hazards

5) Access

6) Number of casualties

7) Emergency Services

In general, medical operations need to be distinguished in three different categories.

The first category comprises only transportation orders for single emergency vehicles

(ambulances). The second category is the inpatient treatment where the deployed

personnel is responsible for a defined damage zone at the scene. The last category

includes not only the inpatient treatment at the scene, but the rescue unit is also

responsible for the transport to an appropriate hospital (EUMED). The patients should

be distributed to the hospitals according to the type of injury, available capacities in 4 PTZ-10 (Patiententransport-Zug 10): Patiententransport crew which is able to transport up to 10 patients to the hospital. 5 BHP-B 50 (Behandlungsplatz-Bereitschaft 50): Treatment Unit - On-Call Service, which is able to treat up to 50 patients. Does not provide transport unites.

20

the hospitals, available emergency vehicles and according to the patient´s condition

(triage category).

At the end of every Eumed deployment, the whole emergency chain and the

organization should be evaluated and assessed in order to improve the cross-border

collaboration during large-scale emergencies (EUMED). However, it is still important

to display the major differences in the level of training, responsibilities and

capabilities of medical professionals in each country in order to get aware of the fact

that each ambulance system is different and mutual assistance might be difficult under

specific circumstances (Appendix 2)

4.1.3 EMRIC

The Emric 6 document describes the arrangements made regarding cross-border

collaboration in the field of fire services and CBRN. The overall goal of these

arrangements is to achieve a high quality standard in case of disasters and to make

optimal use of the available resources for a successful emergency response. After the

dispatch centre gathered all the important information (size of fire/technical

assistance, special circumstances etc.), it is possible to ask the neighbouring country

for assistance. Similar to the Eumed arrangements, the fire departments in the EMR

agreed on the standardized alarm keyword EMRIC. There are three different alarm

stages available, as shown in table 2.

EMRIC 1:

− small/medium damage event

− specialised assistance in the form of special vehicles or units (e.g. heights rescuer or heavy rescue vehicle)

− time to provision: <30 min

− officer in charge not always needed

EMRIC 2:

− as a basis 4 fire fighting vehicles for fire fighting or technical aid including support unit

− independent formation with own operation section

− time to provision: < 60 min

− officer in charge picks up work order at command post

EMRIC 3:

− as a basis 8 fire fighting vehicles for fire fighting or technical aid including support unit

− independent formation with own operation section

− time to provision: < 60 min

− officer in charge picks up work order at command post

Table 2: Emric alarm keywords and scope (EMRIC, 2015b) 6 The name EMRIC is used in two different contexts: First, EMRIC is the name of the program and the whole cooperation. Second, the document for fire fighting cooperation is also called EMRIC.

21

The request from the responsible dispatch centre will be carried out via telephone due

to the urgency of the situation. Immediately after the phone call, the dispatch centre

has to send a standardized E-mail/Fax document in order to prevent communication

problems. The most important information that should be communicated includes:

1) Which package (Emric 1,2,3) is requested

2) Location of gathering point

3) Type and location of the incident

4) Special instructions

For disasters including chemical, biological radioactive and nuclear (CBRN)

hazardous substances, the Emric+ Project developed uniform measurement concepts

to implement a standardized interpretation of measurement results in the EMR. The

service for cross-border collaboration regarding CBRN is also defined in different

packages with a different scope (Appendix 3). Therefore, the EMRIC office

developed a document in German, Dutch and French to simplify the information

exchange between the dispatch centres in the EMR (EMRIC, 2015b).

4.2 Relevant Stakeholders Taking into account the documents mentioned above, one could assume that cross-

border emergency collaboration implies a multidisciplinary approach. In addition, this

also includes a multi-stakeholder approach, where key stakeholders from different

disciplines and different specialisation have to work together. Therefore, it is

important to identify the most important stakeholder in order to get an overview and

to identify potential interview partners.

According to Ramakers (2016), approximately 50.000 people work with the EMRIC

and EUMED plans. As shown in Appendix 4, stakeholders with different levels of

responsibilities from different institutions and countries are involved in the

emergency care chain. On the one hand, there are stakeholders who are responsible

for administrative tasks, who focus more on the organizational level like the members

of the crisis team. On the other hand, there are stakeholders like paramedics and fire

fighters, who work at the scene with a more practical oriented job. Independent of

their own function, people involved in emergency management must familiarize

22

themselves with the different types of stakeholders/resources available in the EMR

(Lindell, Prater, & Perry, 2006)

Figure 3 provides an overview of the different levels/stages available in each country

and it visualises the differences between each country. The first stage is always the

dispatch centre (Leistelle/Meldkamer), where all important information about the

incident scene is gathered and assessed. In the second stage, operational leadership is

mostly dependent on the scale of the incident, it is called Einsatzleitung in Germany,

Coördinatie Plaats Incident (COPI) (responsible for source region) or Regionaal

operational team (ROT) (responsible for effect region) in The Netherlands and

Commandopost Opertaties (CP-OPS) in Belgium. Here, different stakeholders from

different disciplines (Fire Department, Ambulance Service, Police, information

manager) come together to coordinate the units and resources on the scene (EMRIC,

2015a). The third and therefore also the highest stage is the crisis team of the region.

There are always two different crisis teams available, which operate on different

levels. In Germany, this can be on the “Gemeinde” or “Kreis” level, in the

Netherlands on the “Gemeente” or “Regionaal” and in Belgium on the “Gemeente” or

“Provincie” level. Here, it depends on the scale of the incident which crisis team is

responsible for the emergency management. Nonetheless, members of the crisis team

are stakeholders who are responsible for public safety and security on municipal or

regional level.

The workforce on the deployment location regarding the ambulance service must be

distinguished, as there are major differences in competences and responsibilities.

Appendix 2 gives an overview of the different medical teams available in the EMR.

With regard to the fire service, there are minor differences in the organizational

structure, however, the fire fighters in the EMR have similar or almost the same

responsibilities and tasks. It is therefore not necessary to distinguish between every

institution (EMRIC, 2015b).

4.3 Evaluation Design Defining the evaluation design is a crucial step and has a major impact on the

evaluation result. Every evaluation design should balance utility and feasibility,

however, it is important to keep in mind that there is no ”perfect” or “right” design.

The goal should be to balance pros and cons of the evaluation tools and to determine

23

the most appropriate design to answer the evaluation questions (Centers for Disease

Control and Prevention, 2014b).

4.3.1 Requirements of an Evaluation Tool in Theory

The CDC Evaluation Framework includes different attributes that give an indication

of the quality of an evaluation in their framework. The four attributes are utility,

feasibility, propriety and accuracy. All attributes are important in order to guide

choices in the development process of an evaluation tool (Centers for Disease Control

and Prevention, 2014a).

Utility refers to the usefulness of an evaluation tool and its results (Joint Committee

on Standards for Educational Evaluation, 1994). According to Taut (2000), utility can

be subdivided into seven categories. “Stakeholder Identification” is the first category,

which takes into account that stakeholders, who are involved in the project or affected

by the outcomes of the evaluation, have an important role and should be identified.

This is a crucial factor in order to address the different needs of the stakeholders,

especially in an international or cross-border evaluation context. The second category

is the “Evaluator Credibility”. This category recommends that the evaluator should be

competent and reliable in order to increase the acceptance of the outcomes and results

of the evaluation. The next category “Information Scope and Selection” is a very

critical and sometimes difficult process. Each stakeholder should have an opportunity

to participate (input) and have access to the results (output). Sometimes this is not

feasible or compliance is not intended due to a hierarchical structure in the

organization. “Value Identification” is the fourth category and points out the necessity

of developing certain values to interpret the findings. This is especially important in

an international context as it allows identification of common values, as a basis for

value judgements. The fifth category is “Report Clarity”, which means that the

program being evaluated should be clearly described including the purpose, procedure

and context. The sixth category “Report Timeliness and Dissemination” states that the

outcome/result of the evaluation should be disseminated to the intended stakeholders,

so that they can make use of the findings and implement new or improve old policies.

The last category is the “Evaluation Impact” where Taut (2000) points out that the

evaluator should promote follow-through by stakeholders. This might increase the

likelihood that the evaluation will have a beneficial impact.

24

The second attribute mentioned in the CDC Evaluation Framework is feasibility.

According to the Joint Committee on Standards for Educational Evaluation (1994),

feasibility can be influenced costs, politics, available resources, power and to what

extent they have an impact on the evaluation design. These factors should be taken

into account before the implementation of an evaluation tool. However, it is vital to

increase or maintain feasibility once the evaluation tool has been implemented, in

order to achieve the objective of the evaluation.

According to Taut (2000), feasibility can be divided in three categories. First, the

evaluation process should be practical (“Practical Procedure”) in order to keep

interferences to a minimum and at the same time it is important to obtain the

necessary information from stakeholders or participants. The second category is

“Political Viability”. This is especially important when the evaluation takes place in

different countries. In this case, political viability means that the different positions,

and also interest groups, should cooperate with each other in order to prevent bias or

the misapplication of the results. The last category of feasibility is “Cost

Effectiveness”. In other words, the evaluation process should be efficient and the

outcome of the evaluation should be from sufficient value to justify the expenses.

The third attribute for conducting an evaluation is propriety and describes the rights,

responsibilities and legal concerns of all parties (Joint Committee on Standards for

Educational Evaluation, 1994). This attribute is partitioned in eight categories.

Because not all categories are relevant for the current topic, the following part will

describe the four most important categories. “Service Orientation” is the first one and

describes that the evaluation should be designed to support the organisations. In other

words “Service Orientation” asks for serving “program participants, community and

society” (Joint Committee on Standards for Educational Evaluation, 1994, p. 83).

Moreover, it is important to formulate “formal agreements” due to the fact that all

obligations should be written down and signed by all parties (what is done, when, by

whom etc.). A written agreement is necessary to obligate the involved stakeholders to

abide to all arrangements and responsibilities. The third category is, a “Complete and

Fair Assessment”, which is the basis for identifying both strengths and weaknesses of

a program or the implementation of an agreement. Only then, it will be possible to

build upon the strengths and address the weaknesses. The last category of propriety

“Conflict of Interest” can be seen as a sore point, due to the fact that an evaluation

program where multiple stakeholder are involved always include different interests or

25

points of views. The “Conflict of Interest” should be discussed openly and honestly to

prevent a negative impact on the evaluation process (Taut, 2000).

The last attribute of the CDC Evaluation Framework is accuracy. During this step it is

important to determine whether each step of the evaluation process was conducted

accurate, taking into account the stakeholder´s needs and the evaluation purpose

(Centers for Disease Control and Prevention, 2014a). Therefore, accuracy considers

validity, reliability and also the reduction of bias and errors. As shown in Figure 4 the

measurement of reliability and validity is crucial to build a strong basis for a credible

evaluation.

Therefore, it can be said that evaluation requires a valid measure that is going to be

collected in a reliable way. “Measurement Validity” describes the certainty of a

measurement. The overall objective here is that a measure accurately assesses the

intention of the evaluator. At this point, the assessment of the data collection process

should take place to make sure that the evaluation tool provides reasonable and

accurate information and to test the measure for accuracy (Wholey et al., 2010).

“Reliability refers to the extent to which a measure can be expected to produce similar

results on repeated observations of the same condition or event” (Wholey et al., 2010,

p. 13). Therefore, reliability in a cross-border evaluation process is especially

important when the evaluation tool needs to be translated into multiple languages. It is

crucial to assess whether the questions still provoke comparable results after the tool

was translated. In order to pre-test the reliability of data collection and procedures, it

Figure 4: Design Evaluation studies to provide credible findings: The Pyramide of Strength(Wholey,

Harty, & Newcomer, 2010)

26

is required to use statistical indicators such as the Cronbach´s alpha, which indicates

the statistical strength of a test (Tavakol & Dennick, 2011).

4.3.2. Requirements of an Evaluation Tool in Practice

This paragraph gives an overview about practical requirements of an evaluation tool

based on the interviews conducted in the EMR. A summary of the interviews is given,

which makes it possible to compare these finding with the findings from the literature

as described in section 4.3.1.

Some requirements, which were mentioned by all stakeholders, are relevant for all

three countries. These requirements include a clear target group, which must be

defined in advance, the evaluation should not be time consuming, it should be

available in each language (German, Dutch, French) and a moderator such as an

external agent would be preferred to decrease the risk of a biased evaluation (I2/P17,

Personal Communication, 11.05.2016).

Furthermore, there are also country specific requirements. According to respondent

I1/P1 (Personal Communication, 10.5.2016), an intermediate step before the

implementation phase would be needed in Germany, due to a different evaluation

culture. Employees have a rather negative image of evaluation, therefore, a training

where such evolutions will be described and presented would increase a broad

acceptance and thereby the response rate. Respondent I2/P1 (Personal

Communication, 11.05.2016) recommended, to use the evolution tool as a tool for

training purposes, so that participants receive feedback, could learn more and improve

their knowledge, which would increase the utility of this tool. Motivation would be

higher, because there is a direct effect for the participants. Another important aspect is

the timing of the start of the evaluation. In the Netherlands, an evaluation process is

currently running on a national level. Therefore, the implementation of another

evaluation tool at the same time might overburden staff and thereby might decrease

the response rate (I3/P2, Personal Communication, 12.05.16).

4.3.3 Evaluation Process

The following paragraph describes the different steps of an evaluation process in

general. To illustrate this process, the two best known approaches will be discussed 7 I1/P1: Interview 1/Participant 1

27

and a variety of evaluation tools will be presented. One step in the evaluation process

that is essential within every evaluation is the “Evaluability assessment”. Therefore, it

will first be described what an evaluability assessment is and some examples are

provided referring to the EMR.

4.3.3.1 Evaluability assessment

The first step in the evaluation process should be an evaluability assessment. The

evaluator starts with developing a logic model based on program documents and

interviews with key stakeholders in the program. In addition, a logic model displays

the gathered information in a simple flow chart including the needed resources,

intended activities and expected outputs (Figure 5). The final goal is to define the

desired outcome of a project. In general, it can be said that a logic model offers

support for the evaluator in order to focus the data collection on important activities

including their outcomes. Furthermore, it will help to organize and interpret data from

multiple methods or sources. This helps to display patterns of relationships in

complex matters. After the evaluator gathered all information according to the logic

model and the assumptions underlying the program seem logical, it is possible to

collect data about the program in action. If the project described in the logic model

corresponds with the project and the implementation, it is possible to start measuring

the effectiveness of a program (Cooksy, Gill, & Kelly, 2011).

Figure 5: Example of a Logic Model for the Emric bureau in the EMR

4.3.3.2 Formative vs. Summative Assessment

The range of different types of evaluation tools that are available, can be devided into

two main philosophical approaches. These two approaches are formative and

summative evaluation. It is however not simply possible to draw a clear line between

these two approaches. According to Taras (2005), it can be said that the process of an

assessment automatically leads to an summative evaluation approach. This approach

28

includes all findings and evidence from a defined point of time, and can be seen as the

judgement at a final stage of implementation. Furthermore, a summative approach has

various functions, which do not have a significant impact on the general process.

Moreover, Taras (2005) mentions that a summative assessment is a single process

where it is possible to make judgements according to predetermined criteria or

standards.

For a formative assessment, it is necessary to have feedback that indicates the

occurrence of a gap. This gap would most likely be between the current

implementation of the cooperation arrangements and the in theory required standard

of those arrangemnts. Moreover, it is required to determine how the work can be

improved, in order to reach the required standards/criteria. To sum up, it can be said

that both approaches can be seen as a process and it is possible to only conduct a

summative evaluation where the judgement is the final point. However, it is not

possible for an evaluation tool to only be formative, the summative judgement can be

seen as a basis and is required for each type of assessment. Therefore, every formative

evaluation also implies aspects from a summative approach (Taras, 2005).

In addition, an assessment or evaluation can take place at different stages of a project.

Therefore, Figure 6 illustrates where in the policy process implementation evaluation

is focused.

Figure 6: Illustration where in the policy process implmentation evaluation is focused (Centers for Disease Control and Prevention, 2014b)

4.3.3.3 Overview of Evaluation Tools

Considering the above-mentioned information, it was possible to identify a variety of

different evaluation tools. The following evaluation tools where preselected by the

researcher according to predetermined requirements as described in section 4.3.2. The

tools that were found tobe unfeasible with regard to the required needs/goals, are not

discussed within this section.

29

1. Semi-Structured Interviews

A semi-structured interview (SSI) is a type of qualitative data collection, which is

commonly used in policy research. Data collection is a crucial step to perform high

quality research. It is therefore important to use proper techniques to make sure that

the data being collected are accurate, valid and reliable (Harrell & Bradley, 2009).

The interview in general is a managed verbal exchange where communications skills

have a crucial role. As shown in Figure 7, the control an interviewer has on the course

of the interview depends on the type of interview. A SSI balances the amount of

control and makes it possible for the interviewer to both have perceptions about the

order of questions that are asked, and also to conduct a relatively free-flowing

interview (Harrell & Bradley, 2009).

Figure 7: Amount of Control Exercised/Determines Places on Interview Continuum (Harrell & Bradley, 2009)

The approach of using SSI interview with rather open-ended questions should

encourage a discussion with the opportunity for the interviewer to discuss topics that

may have not been considered. Due to the fact that a SSI implies a face-to-face

interviewing approach it is easier for the researcher to gather insight and

understanding about an assigned issue (Newton, 2010). Some weaknesses and

strength are going to be presented in the next paragraph.

According to Denscombe (2007), semi structured face-to-face interviews have the

disadvantage that respondents are very likely to respond differently depending on how

the participant perceive the interviewer. Nevertheless, this disadvantage might vary

depending on the nature of the issue/topic being discussed. Furthermore, it must be

said that it is relatively difficult to compare the interviews because the wording of the

individual will probably differ between each interview (Patton, 2002). Moreover, it is

important to take into consideration that SSI is time consuming and labour intensive,

not only the interview itself, but also the preparation and analysis requires a lot of

30

effort. Therefore, this design is not ideal to evaluate a large sample or institution

(Wholey et al., 2010).

There are also advantages associated with SSIs. First, this type of data collection is

suitable when the researcher is interested in the independent knowledge/thoughts of

an individual in a group. Furthermore it is also an appropriate tool for conducting a

formative program evaluation by using one-on-one interviews with relevant

stakeholders (Wholey et al., 2010). Second, the semi-structured nature of the

interview allows the interviewee to develop ideas, which put more emphasis on the

participant elaborating points of interest (Denscombe, 2007).

2. Focus Groups

Focus groups are interview groups with approximately 6-12 participants. The overall

objective of a focus group is to encourage the participants to share their experiences

or perceptions. This method is a type of qualitative data collection with descriptive

outcome data, which cannot be measured numerically (Centers for Disease Control

and Prevention, 2008a). According to Krueger and Casey (2010b), this method is

suitable for evaluation with a summative and formative approach.

With a focus group as an evaluation tool, it is possible to gather in depth information

about previous experiences, attitudes, perceptions and beliefs. Therefore, it is possible

to gain subjective perspectives from a previously defined group of key stakeholders.

Moreover, focus groups are able to gain additional information for quantitative data

collection. Accordingly, this method is useful in delivering on top information or

more in depth information for a quantitative study (Centers for Disease Control and

Prevention, 2008a). As a result it can be said that this tool is commonly used to

develop, test and improve ideas for a program (Krueger & Casey, 2010a).

The advantages of a focus group evaluation are that within a group brainstorming it is

possible to bring out new ideas. Moreover, it is less time consuming because the input

of several key stakeholders can be collected in one session. However, there are also

some disadvantages. First, it is difficult to bring important key stakeholder together

and, which might proof to be a time consuming challenge. Only a limited number of

questions can be asked depending on the group size and the time available.

Furthermore, depending on the group setting and on the topic, participants might feel

uncomfortable which could result in a biased or distorted answer (M. Jones, Carson-

Cheng, & Lezin, 2013). Overall it can be said that this method might be a suitable tool

31

to provide valuable insights about a program or policy evaluation. Moreover, it is

possible to only conduct a focus group evaluation or to combine it in a mixed method

approach with other tools (Krueger & Casey, 2010b).

3. Project Diary

A project diary is a written record from participants/individuals to gather information

about a process or state of implementation of a project (Evaluation Toolbox, 2010a).

The project diary evaluation can be conducted in two different ways, online or in

written form. Due to the fact that most of the work nowadays is done with a computer

this description will focus on the online diary for qualitative research.

An online diary is an approach used to evaluate the implementation of a

project/agreement and furthermore provides real-time communication between the

evaluator, the bureau staff and the participants. If this tool is used in a mixed method

approach and combined with other tools, it is possible to get an in-depth

understanding of the program implementation (Cohen, Leviton, Isaacson, Tallia, &

Crabtree, 2006)

The methods behind the interactive online diaries involve the collection of data,

which are written by key stakeholders over a defined period of time. It is important

that every key stakeholder writes his/her own diary in order to identify who wrote the

feedback. There are two major differences in the characteristic of an interactive online

diary. It can be distinguished between structured and unstructured approaches or what

R. K. Jones (2000) refers to as the solicited and unsolicited approaches of online

diaries. The solicited approach has a highly structured template developed by the

evaluator, whereby the unsolicited approach is completely unstructured which gives a

lot of freedom to the writer to determine which topic to focus on. Advantages of this

tool, include that participants are able to reflect on their own intervention experiences

in a short time and provide evidence and feedback for the evaluator. A drawback

however is, that that the outcome is not necessarily easy to analyse. Moreover, the

values of the diary keeper might influence or bias the outcome.

In conclusion using this tool for data collection is a time consuming activity and the

effort involved in the process of analysing the data are relatively high (Cohen et al.,

2006).

32

4. Post-Activity Questionnaire:

A questionnaire is a common form of data collection in an evaluation. It is a useful

tool to gather information from large groups. On the one hand, a well-designed

questionnaire is a powerful tool for the evaluator to gather credible data, however, on

the other hand, an insufficiently designed questionnaire makes it difficult for the

participant to complete the questionnaire. Furthermore, it is difficult for the evaluator

to analyse or compare the outcome data (Evaluation Toolbox, 2010b). It is possible to

administer the questionnaires via mail, as handouts or electronically (e.g online, e-

mail) (Centers for Disease Control and Prevention, 2008b).

One should use questionnaires to gather data from a large number of people and there

is only a limited access to financial or time resources. Moreover, questionnaires are a

suitable tool when the evaluator is interested in individual attitudes, knowledge,

behaviour and beliefs. In addition, as questionnaires can be administered and

collected anonymously, it is possible to protect the privacy of each participant. To

maintain the privacy can be of major significance when dealing with sensitive

information, because this might increase the response rate (Centers for Disease

Control and Prevention, 2008b).

There are different types of questionnaires available, however this research is going to

focus on post-activity questionnaires. After having considered other types of

questionnaires it must be said that for the purpose of analysing how arrangements are

implemented, it is the most suitable tool taking the information from section 4.3.1 into

account. In general, this type of questionnaires consist of a limited number of

quantitative closed-end questions in order to reduce the amount of time needed to

complete the survey. It is also possible to include or combine it with open-ended

questions, but this will increase the time effort and will also make data analysis and

reporting more difficult. The overall objective of a post-activity questionnaire is that

participants are able to rate the effectiveness of an activity. Nonetheless, it is also

possible to test the knowledge of the participants after implementing a new policy if

the right set of questions are used (Evaluation Toolbox, 2010b)

5. Stakeholder Analysis

The fifth tool this research is going to discuss has a slightly different approach

compared to the other four tools. The focus of this tool is to identify relevant key

stakeholder and does not aim at a large number of participants. A stakeholder analysis

33

is an important step for the project coordinator to provide insights into the relationship

between a project and the stakeholders. It is a reliable tool to identify which

stakeholders have an impact on the project/policy implementation success (Kennon,

Howden, & Hartley, 2009).

A stakeholder analysis in general can be described as a process, which systematically

gathers data for analysing qualitative information. During this process, the evaluation

focuses on different stakeholder characteristics such as: “policy, interests related to

the policy, position for or against the policy, potential alliances with other

stakeholders, and ability to affect the policy process (through power and/or

leadership)” (Schmeer, 2000, p. Stakeholder Analysis at a Glance). Therefore, a

stakeholder analysis is a suitable tool to analyse and identify the key stakeholders,

assess their current knowledge and in addition to that also their alliances. For policy

makers or an institution, which implemented a new policy, these is crucial

information that allows to act more effectively and to increase the support for the

program/policy (Schmeer, 2000).

4.3.2.4 Overview advantages and disadvantages in Theory

To provide a better overview, table 3 shows advantages and disadvantages of the

selected evaluation tools described in section 4.3.3.3.

Advantages Disadvantages

Semi-Structured Interviews

- gain independent knowledge of an individual - can provide unexpected findings - suitable to discuss sensitive issues -encourages two-way communication

-different respond depending on interviewer -interview skills required -time consuming/labour intensive

Focus Groups - gather information about experiences, attitudes, perceptions and beliefs of a group - less time consuming and economically efficient - large sample sizes

-difficult to bring important stakeholder together -participant might feel uncomfortable die to group size - strong voiced vs. passive

34

possible participants

Project Diary - can provide insights about working procedure - gather information about the process of change -participants reflect on their own intervention experiences

- labour intensive and time consuming - difficult to analyse - value of diary highly depended on participant - required project mediator to keep diary up to date

PA Questionnaire - provides quantitative data for statistical analysis - a standardized questionnaire allows comparison - less time consuming for participant (closed-end) - suitable tool for a large group of people

- risk of low response rate - wording can result in biased response - risk of incomplete response - risk that participants forget important issues

Stakeholder Analysis - improve relationship to stakeholder - possible to make better decisions and strategies

- analysis may be subjective - not always representative

Table 3: Overview of advantages and disadvantages of evaluation tools

4.3.2.5 Overview advantages and disadvantages in Practice

This paragraph describes the advantages and disadvantages of each tool based on the

interviews conducted with the different key stakeholders and experts in the EMR.

During the interviews, all participants were introduced to the tools as described in

section 4.3.3.3 and afterwards asked about their opinion on these tools.

According to respondent I5/P1 (Personal Communication, 25.05.2016), the semi-

structured approach has a certain degree of flexibility during the interview, which is

an advantage when it comes to gathering more information and insights about a

topic/issue that might not have been considered before. Another advantage, is that the

rich data are gathered and the response rate are relatively high compared to other

tools. Respondent I1/P1 (Personal Communication, 10.05.2016) stated that the

response rate compared to other is relatively high when conducting individual face-to-

face interviews with personnel in a leading position. However, there are also

disadvantages in practice. First, the preparation and the interviews itself are labour

intensive and time consuming. According to respondent I2/P1 (Personal

35

Communication, 11.05.2016), performing SSI´s is time consuming and from an

economical perspective not feasible for a large cohort in the EMR. Conducting SSI’s

is therefore only feasible when the sample size is small.

Focus Groups have the advantage that it is possible to save time and reach more

participants simultaneously. According to respondent I3/P2 (Personal

Communication, 12.05.2016), a group session where it is possible to interview a

group of employees or personnel with a leading position makes the evaluation process

more efficient and data collection faster. The disadvantage of this tool is that it is

quite difficult to organize and find a common date, where all stakeholders can be

present. In most organizations people work in shifts and generally 50 % of the

employees are on call (I1/P2, Personal Communication, 10.05.2016; I2/P1, Personal

Communication, 11.05.2016).

Most respondents considered the project diary to be a suitable tool to evaluate

individual working procedures. Respondent I2/P1 (Personal Communication,

11.05.16) stated: “ I can imagine that this tool is an appropriate evaluation tool to

gather individual feedback from employees”. Moreover, a project diary can be used

for a follow-up and to monitor a change process. I5/P3 (Personal Communication,

25.05.2016) describes that they have implemented a similar tool a long time ago in

the dispatch centre in the Netherlands. This allowed to gather crucial and individual

data over time to evaluate disruptions of critical processes. However, a project diary

also has some disadvantages. I1/P1 (Personal Communication, 10.05.2016) stated that

this tool is a time and labour intensive activity with regard to implementation and

maintenance. Therefore, implementation of a project diary in practice for the

emergency services will be difficult due to the time pressure on the scene and during

shifts. In addition, respondent I2/P1 (Personal Communication, 11.05.16) mentioned

that this tool would only be suitable for the evaluation in the dispatch centre because

this is the place where all information comes together so that the personnel there gets

an overview about the whole scene.

A post-activity questionnaire has the advantage that it has the possibility to reach

many people, so that the data can be gathered from a large sample. I3/P2 (Personal

Communication, 12.05.2016) explained, the distribution of questionnaires in practice

is relatively simple; this can be done online via intranet, via e-mail and also as a

hardcopy via mail. In addition, this tool would be suitable to combine with other

evaluation methods due to the fact that this tool can gather addition information with

36

relatively little effort. Therefore, respondent I3/P1 (Personal Communication,

12.05.2016) gave the example to conduct face-to-face interviews with stakeholders in

leading positions first and then distribute (online or hardcopy) questionnaires to vast

majority if the key stakeholder supports such an evaluation. However, according to

I1/P2 (Personal Communication, 10.05.2016) post-activity questionnaires are a waste

of effort, if questionnaires are handed out to the vast majority of employees, because

it is a frequently used tool and the response rate is expected to be low. Moreover, due

to the fact that this tool is common in evaluation processes, the motivation of

employees to participate is relatively low, especially if this is carried out

anonymously (I6/P1, Personal Communication, 02.06.2016).

According to I2/P1 (Personal Communication, 11.05.16), stakeholder interviews can

save a lot of time and it is possible to take a sample that can represent a large groups,

if samples are drawn carefully. However, it is very likely that important information

from the basis is missing because key stakeholders are generally in higher positions.

In addition, it is possible that some key stakeholders have an information advantage,

which makes them unsuitable to serve as a representative sample. Respondent I6/P1

(Personal Communication, 02.06.2016) stated: “ I´m part of the steering group and

some focus groups, of course I know the documents quite well, but this does not

represent the average knowledge of the employees here”.

Multiple respondents indicated that it would be useful to make use of a mixed-method

approach. Therefore, I5/P3 stated that an evaluation consisting of small steps

including different tools for different target groups would be appropriate. As an

example, the expert said that a questionnaire for the vast majority of employees could

give a basic overview of knowledge and experiences, which could be used as a

foundation for SSI with people in leading position.

5. Discussion

To evaluate the current state of implementation of cooperation arrangements in the

EMR in the fields of fire services, CBRN, acute medical care and information

exchange, the “CDC Framework for Evaluation in Public Health” will be used as a

reference. This framework provides a step-by-step approach to find an appropriate

tool, to determine the gap between the cooperation arrangements and implementation

37

in practice. According to this framework, one should describe the arrangements and

therefore the desired outcome of the evaluation, which for the EMR include three

arrangements that were identified in this study. Arrangements regarding fire services

and CBRN are described in the EMRIC document, arrangement regarding acute

medical care are described in the EUMED document, and arrangements regarding

information exchange are described in the document called “Arrangements for

information exchange between Euregional partners in the event of a disaster”. In

addition, the framework identifies a set of standards (utility, feasibility, propriety and

accuracy), which provide an indication of the quality of the evaluation tool.

Interviews with key stakeholders also determined a set of requirements regarding the

practical application of such an evaluation tool. As the CDC Evaluation Framework

provides a theoretical input and the interviews with key stakeholders are focusing on

the practical application, it is crucial to compare the theoretical knowledge with the

practical implementation, in order to identify the most appropriate tool.

Therefore, this chapter focuses on comparing results extracted from the literature with

the requirements mentioned by the respondent during the interviews, as it is crucial to

meet the theoretical requirements while taking the suggestions mentioned by the key

stakeholders into account.

5.1 The Requirements/Standards

The first paragraph compares the standards extracted from the literature with the

requirements stated by all stakeholders in each country during the interviews. The

second paragraph focuses on country specific requirements.

All stakeholders indicated that a moderator such as an external agent would be

preferred in order to reduce the risk of a biased evaluation and provide a reliable and

credible outcome. A similar theme was discussed in the literature, where “Evaluator

Credibility” is recommended as a crucial standard to increase acceptance of the

evaluation outcome in practice. The suggested moderator, which according to the

respondents would ideally be an external party, would ensure the evaluator credibility,

to which is referred in the literature.

Furthermore, the stakeholders mentioned that an evaluation should not be time

consuming in order to keep the expenses to a minimum. This can be linked to “Cost

effectiveness” as mentioned in the literature, which is seen as an important standard in

38

order to increase the feasibility of an evaluation tool. Both time and expenses are

important factors when considering the cost-effectiveness of an activity. Therefore, it

can be assumed that both the literature and the interviewed stakeholder refer to the

same criterion, that needs to be considered when choosing an evaluation tool (I2/P1,

Personal Communication, 11.05.2016).

Another issue, which is applicable in a cross-border setting, is the language barrier

between the countries. The stakeholders made clear that the evaluation tool must be

available in each language (German, Dutch, French) in order to increase the response

rate. This might however require reliability to be checked for each language

separately, as described by Wholey et al. (2010) who state that after translating an

evaluation tool, it is crucial to assess whether the questions still provoke comparable

results. As a result it is likely that this step will be more time consuming than

expected, in order to ensure reliability of the translated tool.

Hence, it can be said that the requirements put forward by all stakeholders in the

EMR are similar to those described in the literature. However, there are also country

specific requirements regarding a possible evaluation tool. The following paragraph

therefore compares the country specific requirements with the attributes (utility,

feasibility, propriety, accuracy) mentioned in section 4.3.2.

Respondent I2/P1 recommended using the evaluation tool for training purposes so

that participants receive feedback after they participated in the evaluation. This is a

crucial point with regard to the utility of the evaluation tool. The literature states that

the “Information Scope and Selection” is a critical process. However, in some

organisations it is not desired to distribute the outcome/result of the evaluation due to

their hierarchical structure. Even though the statement from respondent I2/P1 clearly

describes the opposite. In the Netherlands, stakeholders have the intention to

distribute the outcome and provide feedback for the participants. This might result in

increasing the usefulness or utility of this specific evaluation tool. Moreover, in

Belgium and Germany the key stakeholders did not mention something similar.

Another crucial aspect mentioned by respondent I3/P2 is the timing of the start of the

evaluation. Comparing this to the feasibility of the evaluation tool, “Practical

Procedure” and “Political Viability” should be considered. In order to keep

interferences to minimum, the evaluation tool should be practical and the evaluation

should not take place together with another evaluation on the national level for

39

instance. This might decrease the response rate and the motivation for participation

and could furthermore result in interferences due to political discrepancies.

Another finding of this study is, that there are discrepancies regarding the evaluation

culture between the three countries. In Germany, respondent I1/P2 mentioned that an

intermediate step would be needed in order to increase the acceptance of the

evaluation tool. Therefore, it is crucial to provide information about the purpose,

procedure and the context of the evaluation for the stakeholders/participants.

Overall, it can be said that the practical needs identified through stakeholder

interviews comply with the criteria identified in the literature. The literature, however,

provides some additional criteria besides the requirements mentioned by the interview

partners. The reason for that might be that stakeholders are more focused on the

practical application of the tool in their organization instead of theoretical standards.

However, the additional requirements mentioned in the literature should be taken into

account because they provide crucial information to increase reliability and validity of

the evaluation. One example is to develop certain values, which are defined in

advance to interpret findings. This is especially important in a cross-border evaluation

due to different standards and procedures.

5.2 The Evaluation Tool Regarding the different evaluation tools, it can be said, that all tools have different

advantage and disadvantages. It is therefore important to select the most appropriate

tool depending on the purpose of the evaluation. In order to identify the gap between

the cooperation arrangements and their implementation in practice and to define how

the implementation can be improved in order to reach the required standards, it can be

concluded that for the purpose of this evolution, a formative approach would be

suitable (Taras, 2005).

However, taking all advantages and disadvantages of each tool into account, it can be

said that a perfect tool cannot be found. The objective is to balance the pros and cons

to determine the most appropriate design. Consequently, this study points out that

making use of one tool, might not be sufficient. Taking the feedback from the key

stakeholders into account it might not be feasible to conduct the evaluation with one

tool (e.g semi-structured interview). In order to determine the gap between the

cooperation arrangements and the implementation in practice, it is crucial to gather

detailed and individual data. On the one hand, this is only possible with individual

40

interviews. On the other hand, in the EMR interviewing approximately 50.000 people

is not feasible, as it would be to time consuming and too expensive. With

questionnaires, it is possible to reach a large group of people and save a lot of time,

which would increase cost effectiveness during the evaluation process. However,

making use of this evaluation tool implies a high risk of low or incomplete responses,

which would result in low quality data and less validity. Therefore, as two

respondents and the literature recommended, a multi method approach might be the

most suitable approach in order to assess the current gap between cooperation

arrangements and their implementation in practice (I5/P3, Personal Communication,

25.05.2016),(Centers for Disease Control and Prevention, 2014b). In addition, a multi

method approach has the benefit that advantages from different tools can be mixed in

order to balance out the disadvantages and to create an individual tool suitable for the

purpose of the EMRIC bureau.

With regard to the “CDC Framework for Evaluation in Public Health”, it can be said

that the first three steps are in a logical order to find an appropriate evaluation tool.

Moreover the four standards (utility, feasibility, propriety and accuracy) are

supportive and provide a theoretical background, which can be used as a guideline for

the evaluator. However, the framework is missing out cross-border dimension in the

EMR. The framework does not take into account the different evaluation culture in

each country. Furthermore, this framework does not take into account the willingness

and the ability to participate or cooperate regarding the evaluation, which could also

differ between countries and might have a major impact of the evolution.

5.4 Limitations This study had several limitations, which could have affected the reliability and

validity of the study. First, the researcher had never conducted semi-structured

interviews before. This might have caused the first interview to be less flexible in

terms of reacting to the interviewees answers, compared to interviews that followed

the first one. In future studies, this could be prevented by using more experienced

interviewers, or by training the interviewer beforehand. Second, a language barrier,

was present as French and Dutch speaking people were asked to speak in German or

English. This might have limited them when replying to the questions, as they were

insecure in a different language or had difficulties expressing themselves. This might

41

affect the reliability due to possible occurrence of misunderstandings during the

interview. To limit this, a second interviewer was asked in two interviews to join the

meeting who could translate from French to English or from Dutch to English.

However, next time it would be more appropriate to ask someone else to conduct and

transcribe the interviews. Another issue is that the knowledge produced in this study

might not be generalizable to other settings, as this study focussed on developing an

evaluation tool for the EMR and only stakeholders from this region participated in

this research. However, this study used multiple methods for data collection and

provides individual information from key stakeholders in the EMR. Therefore it can

be said, that the qualitative approach of this study is especially responsive to local

conditions and the stakeholders´ need.

6. Conclusion and Recommendations The study was set out to find an evaluation tool to determine the current state of

implementation of cooperation arrangements in the fields of fire service, CBRN, acute

medical care and information exchange. This tool should aim to measure the gap

between theoretical foundation of the underlying arrangements and their

implementation in practice. The evaluation of the current state of implementation of

cooperation arrangements is crucial and of great importance to further improve cross-

border collaboration in the EMR. Furthermore, it is beneficial for the stakeholders

involved in the emergency care chain, as improved collaboration of these stakeholders

might ensure higher quality of care for patients.

With regard to the existing cooperation arrangements, a lot of work has been done. In

the past, the cooperation made it possible to clearly define standards and objectives

regarding cross-border collaboration in disaster settings. Therefore, the overall goal

regarding these arrangements should not be to further improve them, the focus should

be on the implementation for the future. Hence, this study can be seen as a foundation

for achieving this objective.

With regard to the evaluation tools, the requirements for an evaluation stated by key

stakeholders in the EMR can be fulfilled by using theoretical requirements mentioned

in the literature. To further support the practical application by taking the theoretical

requirements into account, a combination of two tools would encourage a high quality

evaluation to gather valid and reliable data. Therefore, to practically determine the

42

current gap between the cooperation arrangements and their implementation, a mixed

method approach should be used. In practice, this study would suggest to use SSI for

people in leading positions and post-activity questionnaires for the vast majority of

the staff. With these tools, it is possible to first gain independent knowledge of key

stakeholders and to assess their personal support or interest for cross-border

collaboration, while encourage a two-way discussion about sensitive issues. Second,

with post-activity questionnaires a large group of people can be assessed to provide

quantitative data for statistical analysis.

The scale of this research on a local level is comprehensive, however to generate a

feasible evaluation, there is still a need for further research. Therefore, taking the

following steps for further research into account, will facilitate the attainment of

determining the gap between arrangements in theory and implementation in practice.

Practical exercises would first provide an opportunity for staff to practice the

procedures and second, provide additional information besides the theoretical

evaluation. In addition, more detailed research regarding the suggested evaluations

tools is needed in order to specify the research setting

Overall, with the selected evaluation tools it is possible to determine where

improvements in the implementation process may be needed. Furthermore, tackling

this issue, would further enhance the cooperation to jointly overcome obstacles and

manage possible disasters in the EMR effectively.

43

References

Barkenbus, J. (1998). Expertise and the Policy Cycle. Tennessee: Energy,

Environment, and Resources Center The University of Tennessee.

Blackstone, A. (2016). Principles of Sociological Inquiry: Qualitative and

Quantitative Methods, v. 1.0. Retrieved 27.01.2016, from

http://catalog.flatworldknowledge.com/bookhub/reader/3585?e=blackstone_1.

0-ch02_s03

Brand, H., Hollederer, A., Wolf, U., & Brand, A. (2008). Cross-border health

activities in the Euregios: good practice for better health. Health Policy,

86(2-3), 245-254. doi: 10.1016/j.healthpol.2007.10.015

Centers for Disease Control and Prevention. (2008a). Data Collection Methods

for Program Evaluation: Focus Groups Retrieved 18.05, 2016, from

http://www.cdc.gov/healthyyouth/evaluation/pdf/brief13.pdf

Centers for Disease Control and Prevention. (2008b). Data Collection Methods

for Program Evaluation: Questionnaires Retrieved 19.5, 2016, from

http://www.cdc.gov/healthyyouth/evaluation/pdf/brief14.pdf

Centers for Disease Control and Prevention. (2014a). Brief 1: Overview of Policy

Evaluation. Retrieved 26.01.2016, 2016, from

http://www.cdc.gov/injury/pdfs/policy/Brief 1-a.pdf

Centers for Disease Control and Prevention. (2014b). Brief 2: Planning For Policy

Evaluation. Retrieved 26.01.2016, 2016, from

http://www.cdc.gov/injury/pdfs/policy/Brief 2-a.pdf

Centers for Disease Control and Prevention. (2014c). Brief 6: Policy Evaluation

Data Considerations. Retrieved 26.01.2016, 2016, from

http://www.cdc.gov/injury/pdfs/policy/Brief 6-a.pdf

Cohen, D. J., Leviton, L. C., Isaacson, N., Tallia, A. F., & Crabtree, B. F. (2006).

Online Diaries for Qualitative Evaluation - Gaining Real-Time Insights

American Journal of Evaluation, 27(2), 163-184.

44

Cooksy, L. J., Gill, P., & Kelly, A. (2011). The program logic model as an integrative

framework for a multimethod evaluation. Evaluation and Program

Planing, 24(2), 119-128.

Denscombe, M. (2007). THE GOOD RESEARCH GUIDE - for small-scale social

research projects (3 ed.). Buckingham: Open Univerity Press.

Die Landesregierung Nordrhein-Westfalen. (2015). Gesetz zur Neuregelung des

Brandschutzes, der Hilfeleistung und des Katastrophenschutzes.

Retrieved 14.06, 2016, from

https://recht.nrw.de/lmi/owa/br_vbl_detail_text?print=1&anw_nr=6&val=&ve

r=0&vd_id=15416&keyword=

EMRIC. (2015a). Gegenüberstellung der Funktionen bei Großunfällen.

Maastricht: Lenkungsgruppe Emric.

EMRIC. (2015b). Vereinbarung bezüglich der grenzüberschreitenden

feuerwehdienstlichen, technischen und spezialisierten Hilfeleistung in der

Euregio Maas-Rhein (Vol. 3). Maastricht, The Netherlands: EMRIC+.

EMRIC Lenkungsgruppe Euregio Maas-Rhein in Crisismanagement. (2012).

Weiternentwicklung der grenzüberschreitenden Hilfeleistung in der

Euregio Maas-Rhein Mehrjahresplan 2014-2019. Simmerath: EMRIC+.

EMRIC Lenkungsgruppe Euregio Maas-Rhein in Crisismanagement. (2015).

Absprache zum Informationsaustausch zwscihen den Euregionalen

Partnern im Falle einer Katastrophe oder Krise. Maastricht, The

Netherlands: EMRIC+.

EUMED. (October 2015). Grenzüberschreitende mendizinische Hilfe bei

Großschadenereignissen in der Euregio Maas-Rhein (Vol. 4).

Euregio Maas-Rhein. (2015). Administrative Gliederung Euregio Maas-Rhein.

Retrieved 26.01.2016, from http://www.euregio-

mr.com/intern/pdf/Administr. Gliederung_Administr. Indeling_Division

administr. EMR

European Union. (2015). The EU in brief. Retrieved 19.01.2016, from

http://europa.eu/about-eu/basic-information/about/index_en.htm

Evaluation Toolbox. (2010a). Project Diary. Retrieved 18.05, 2016, from

http://evaluationtoolbox.net.au/index.php?option=com_content&view=article

&id=34&Itemid=141

45

Evaluation Toolbox. (2010b). Types of Questionaires Retrieved 19.5, 2016, from

http://evaluationtoolbox.net.au/index.php?option=com_content&view=article

&id=58&Itemid=154

Harrell, M. C., & Bradley, M. A. (2009). Data Collection Methods Semi-Structured

Interviews and Focus Groups. Santa Monic National Defense Research

Insitute

Joint Committee on Standards for Educational Evaluation. (1994). The Program

Evaluation Standards: How to assess evaluations of educational programs

(2nd Edition ed.). Newbury Park: Sage

Jones, M., Carson-Cheng, E., & Lezin, N. (2013). Using Focus Groups to Enhance

Your Evaluation. Seattle, WA: Center for Community Health and

Evaluation.

Jones, R. K. (2000). The unsolicited diary as a qualitative research tool for

advanced research capacity in the field of health and illness. Qual Health

Res, 10(4), 555-567.

Kennon, N., Howden, P., & Hartley, M. (2009). Who really matters? A stakeholder

analysis tool. Extension Farming Systems Journal, 5(2).

Krueger, R. A., & Casey, M. A. (2010a). Focus Groups. A Practical Guide for Applied

Research (3rd Edition). Thousand Oaks: Sage Publications.

Krueger, R. A., & Casey, M. A. (2010b). Handbook of practical program evaluation

Focus Group Interviewing. San Francisco: Jossey-Bass.

Liga.Fokus 1. (2008). Evaluation of Border Regions in the European Union

(EUREGIO). Düsseldorf: Liga.NRW.

Lindell, M. K., Prater, C. S., & Perry, R. W. (2006). Fundamentals of Emergency

Management: Federal Emergency Management Agency.

Louise Barriball, K., & While, A. (1994). Collecting data using a semi-structured

interview: a discussion paper.

Ministerie van Veiligheid en Justitie. (2013). Wet Veiligheitsregio´s. Den Haag:

Rijskoverheid.

Newton, N. (2010). The use of semi-structured interviews in qualitative

research: strengths and weaknesses. Retrieved 17.05, 2016, from

http://www.academia.edu/1561689/The_use_of_semi-

structured_interviews_in_qualitative_research_strengths_and_weaknesses

46

Nickless, J. (2001). The Impact of the EU Internal Market on Social Health Care

Conference on European Integration and Health Care Systems: A

Challenge for Social Policy: European Union.

Patton, M. Q. (2002). Qualitative research and evaluation methods (Vol. 2). Calif:

Sage Publications.

Perkmann, M. (2010). Cross-Border Regions in Europe-Significance and Drivers

of Regional Cross-Border Co-Operation. European Urban and Regional

Studies, 10(2).

Polit, D. F., & Beck, C. T. (2012). NURSING RESEARCH: GENERATING AND

ASSESSING EVIDENCE FOR NURSING PRACTICE (Vol. 9): Wolters Kluwer

Health.

Ramakers, M. (2014). Working Paper-The Academy Maastricht, The Netherlands.

Ramakers, M. (2016). Grenzüberschreitende Zusammenarbeit in der Euregio

Maas-Rhein. Power Point Presentation. EMRIC.

Ramakers, M., Jabakhanji, S., & Thönis, T. (2009). Projektbuch 2009/2013.

Maastricht: EMRIC+.

Schmeer, K. (2000). Stakeholder Analysis Guidelines. Section 2 of Policy Toolkit

for Strengthening Health Reform. Washington, DC: Partners for Health

Reform.

Streiner, D. L., Norman, G. R., & Cairney, J. (2015). Health Measurement Scales - A

practical guide to their development and use (Vol. 5): Oxford Univerity

Press.

Taras, M. (2005). Assessment-Summative and Formative-Some Theoretical

Relfections. British Journal of Educational Studies, 53(4), 466-478.

Taut, S. (2000) The Program Evaluation Standards in International Settings.

Cross-Cultural Transferability of The Program Evaluation Standards. The

Evaluation Center Occasional Papers Series.

Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach's alpha. Int J Med

Educ, 2, 53-55. doi: 10.5116/ijme.4dfb.8dfd

Wholey, J. S., Harty, H. P., & Newcomer, K. E. (2010). Handbook of practical

program evaluation. San Francisco: Jossey-Bass.

47

Appendix

1) Cross-boarder healthcare and the European Union

The overall aim in the EU is to create a solid “Union” on different issues among their

28 Member States (European Union, 2015). Due to the improvement of the single

market, free movement of goods, services and professionals, (Article 114 TFEU) the

EU reduced barriers regarding trade and cross border cooperation. Kohl and Decker

were the first who applied these principles to healthcare. A new form of cross border

cooperation in healthcare has been established as a consequence of the judgement of

the European court of justice regarding the Kohl and Decker case in 1998 (Nickless,

2001). Based on the judgement of the European Court of Justice (ECJ), the EU

implemented new laws and legislations regarding cross border health care, such as the

Directive 2011/24/EU on patient´s rights in cross border healthcare and the directive

on cross border care (2014/24/EU). Due to the fact that hazards like toxic clouds, fire

or communicable diseases do not stop at boarders and therefore have an impact on

more than one country the cross border dimension needs to be taken into account in

order to combat possible threats to public safety especially in border regions(EMRIC

Lenkungsgruppe Euregio Maas-Rhein in Crisismanagement, 2012).

With regard to cross border disaster and emergency care, exchange of knowledge,

standards and experiences is vital to develop a joint approach. The international

cooperation between countries in terms of emergency services includes a variety of

obstacles such as the language barrier, differences in the level of training, different

crisis management procedures and sometimes also disparate access or use of

resources (Liga.Fokus 1, 2008). Local authorities in the Euregio Maas-Rhein realized

already 40 years ago that there is a need for cross border emergency collaboration to

tackle these obstacles (Ramakers et al., 2009). Based on that, it can be said that the

comparison between the EU and the EMR can be seen as a parallel development

towards cross border healthcare collaboration. The EU nowadays creates a legal basis

and makes it easier to develop new arrangements between neighbouring countries

(Perkmann, 2010).

48

2) Comparable Personnel in the Ambulance Service:

49

3) CBRN Alarm Keywords:

• CBRN_M1 (Deutschland und Niederlande) Alarmierung eines AGS aus dem Piketdienst oder des diensthabenden FB CBRN aus der FB Bereitschaft.

• CBRN_M2 (in Deutschland)

Alarmierung gem. CBRN-1 Alarmierung von 2 ABC Erkundern und Ümessen P (Führungskräfte und 2ten FB CBRN) Alarmierung von Lotsen

• CBRN_M2 (in den Niederlanden)

Alarmierung gem. CBRN-1 Alarmierung von 2 Messtrupps Alarmierung Messplanleiter (MPL)

• CBRN_M3 (in Deutschland)

Alarmierung CBRN-1 und ÜMessenP (Führungskräfte und 2ten. und/oder 3ten FB CBRN) Alarmierung von bis zu 5 ABC Erkundern (analog Ümessen 2 jedoch ohne ELW-2)

• CBRN_M3 (in den Niederlanden)Alarmierung gem. CBRN-1 Alarmierung von bis zu 5 Messtrupps Alarmierung MPL

50

4) Overview of different functions in the EMR:

51

6) Coding Tree:

The coding trees in this appendix have been established during the data analysis

phase, where a separate coding tree has been created for the different interview

questions

Coding Tree I - measures already taken to implement/evaluate

• Arrangements implemented in dispatch centre

o Information also available for staff (Ambulance driver, fire

fighter)

o Example: Harbour and drug laboratory incident as an good

example of cross-border collaboration

• no evaluation done yet because there is no incident that could have

been evaluated � regarding EUMED and also EMRIC

o Evaluation would make more sense for day-to-day business

o Regarding EMRIC: only a few (3 in Städteregion Aachen) fire

stations are close to the border of the Netherlands

• Features regarding cross-border collaboration part of the initial training

• Information exchange regarding cross-border collaboration also via

Intranet/E-mail

• Heinsberg � no scientific evaluation, however, information exchange

between stakeholder (they know each other quite well)

• Liege� didn’t implement anything due to language barrier

• Maastricht� implemented in dispatch centre and personnel with a

leading position because only those people need to know how cross-

border collaboration works, the rest is working in their own system

• GGD� training for cross-border collaboration, language course,

internships in different countries

Coding Tree: II- requirements of a tool

• Intermediate step needed (e.g training)

o Different evaluation culture in Germany and Netherlands

o Surveillance of professional activities is difficult

o Negative image of evaluation

52

• Other factors:

o Not time consuming, difficult to find a day for an (group)

evaluation, moderator needed, target group?

o Language, especially for Belgium, tool also available in French

to increase response rate.

• Use evaluation tool as training, so people can learn more and improve

their knowledge, furthermore the motivation would be higher because

there is a direct effect for the participant as well.

• Timing of evaluation implementation: In the Netherlands there is

already a surveillance system for evaluation �risk of to many

evaluation tools � decreased response rate and quality

• Focused evaluation � risk of loosing track �difficult to distinguish

between important and less important data

Coding Tree III - pros and cons of tools

• Semi structured interviews

o Qualitative interviews for personal in leading position (more

work/effort� more quality better outcome and response rate)

quantitative rather for other employees (easier to reach many

people HOWEVER: lower response rate expected if not mandatory

o Good tool for evaluating special events but not useful for day-to-

day operations

o GGD� useful tool and flexibility/semi-structures approach as

advantage to gather more information

o Time intensive

• Focus groups

o Would only make sense at the fire station which are close to the

border

o Useful tool to safe time and reach more participants

o Difficult to organize and find a common date (shift work) �50%

of staff is always on call

o Maastricht—> meeting with OVD once in two month

• Project diary

53

o Useful tool, however, a lot of effort, time consuming and according

to Interviewee: they still focus on data exchange because there is

still room for improvement, this tool would be the high-end

solution for both evaluation and improvement of data exchange

o Good tool for evaluation of dispatch centres (necessary to have one

responsible person)

o Increase response rate with mandatory input field

o There are other objectives in the dispatch centres like telephone-

reanimation which have a higher priority at the moment

�Heinsberg

• Post-activity questionnaire

o Waste of effort to distribute it to crowed –> low response rate

expected (no big difference between online or hardcopy)

o On the other hand easy to reach many people

o A tool which is used quite often which could result in low data

quality

o Example Heinsberg: Same tool for trauma evaluation� response

rate less than 30%

o Mediators necessary to support tool/evaluation

o Mixed method approach: first face-to-face interview from

personnel in leading position, then hand out questionnaire for basis

if there is support from mediator

• Stakeholder Interviews

o Saves time, represent a large group

o Important information from bases are missing out

o Not representative because key stakeholder have an information

advantage so it does not necessary represent overall the overall

knowledge.

• Mixed Method approach

o Evaluation in different steps and different tools for different target

group: Questionnaire for the basis with questions about knowledge

and experiences� can be used as foundation for semi-sturctured

interviews with people in leading position

54

Coding Tree IV – support for evaluation

• Interviewee would like to support evaluation also for own interest�

self reflection

• Would be willing to distribute information and to get in touch with key

stakeholders

• Would support evaluation to improve cross-border collaboration and to

furthermore learn from other countries/systems

• GGD� zero measurement